Add baseline feature to track expected failures

[chr-hertel]

Is your feature request related to a problem? Please describe.
When adopting the conformance tests for PHP SDK, we have know issues that we accept to fail for now.
We solve that by an ugly stunt in GitHub Actions, which kinda works, but is optimistic since it only tracks a plain number and not which scenarios are expected to pass or fail.
Another example could be an update of the conformance test app, bringing more scenarios.

Describe the solution you'd like
Baselines are a common feature to pin exactly expected failures. Would be great to have an option to dump current failures as some kind of yaml/json/... (readable) file and reference it while executing the tests.

Expectation would be that if only expected failures occur, the exit code of the test run would be 0.

See PHPStan as example.

Describe alternatives you've considered
We could also have some kind of report, that gets compared in the end. Maybe an abstract of those check.json files.

[pcarleton]

Thanks @chr-hertel ! I spoke with @maxisbey about this yesterday, and we agreed on a file (likely yaml) like you described.

One other option we were considering is how codecov does it in python, but it seems like that requires saving artifacts to an external service, or running on main and the current branch and diffing each time.

We also discussed whether it should exit 1 if the failures decrease (e.g. you've fixed a conformance issue without realizing it). This adds a bit of friction in that you'd have to push, see the CI failure, re-generate your failures file, and then push again, but that seems like it will be rare enough that this flow seems fine.

[chr-hertel]

+1 for failure on decrease, yes - baseline is basically an expectation, if that's not matched devs should be aware - i don't think that's a huge trade off