Changed behaviour when used with `pytest-rerunfailures`

[lukasmasuch]

We are using pytest-github-actions-annotate-failures + pytest-rerunfailures with one additional rerun for our e2e tests in https://github.com/streamlit/streamlit. With pytest-github-actions-annotate-failures==0.3.0, annotations are also added for failed tests in the first run. With pytest-github-actions-annotate-failures==0.4.0, the behaviour seems to have changed to only annotate the final test failure. Is this expected?

[edgarrmondragon]

Not expected nor intended. We don't test integration with other pytest plugins, so it's easy to miss regressions like these.

I've confirmed that the change you're seeing was introduced by #121 to support subtests. In particular, the change to the pytest hook we use from pytest_runtest_makereport to pytest_runtest_logreport.

I can dive a little deeper, but I don't immediately have any good ideas as to how to support both Pytest 9+ subtests and third-party pytest-rerunfailures, so any suggestions are welcome 🙂.

[edgarrmondragon]

For additional context:

https://github.com/pytest-dev/pytest/blob/79392133b63bc79efde860eb94901108b88cfcd6/src/_pytest/subtests.py#L250-L265

[edgarrmondragon]

PR is up:

• fix: Ensure the first failure of a flaky test is annotated when using in tandem with pytest-rerunfailures #144

[edgarrmondragon]

Fixed in https://pypi.org/project/pytest-github-actions-annotate-failures/0.4.1/