Introspection: adds backward compatibility tests

[Tpt]

Introspect crates using old PyO3 versions and check if the output is still correct

Will be useful if we do breaking change to the introspection format

[codspeed-hq]

Merging this PR will improve performance by 12.89%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 104 untouched benchmarks
⏩ 1 skipped benchmark¹

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
⚡	bench_pyclass_create	4.6 µs	4.1 µs	+12.89%

---

_{Comparing Tpt:tpt/back-compat (f831993) with main (f57bda7)}

Footnotes

1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

[davidhewitt]

Interesting, this makes a ton of sense to do. I wonder, how far do you expect we want to test backwards compatibility in this way?

[Tpt]

I wonder, how far do you expect we want to test backwards compatibility in this way?

I see two possible interpretation of your question so I will try to answer both:

• I would hope we can keep backward compatibility as much as possible i.e. be able to introspect builds with quite old versions of pyo3.
• The current test is indeed currently quite basic. Expending it would be great. Topic for a follow up?

[davidhewitt]

Sorry for the very slow reply. Makes sense that we would want to support a wide range of PyO3 versions, given maturin will be baking in the latest pyo3-intropection only.

I worry slightly about the maintenance overhead of having a snapshot of a test with a PyO3 version in-tree. I am not sure what the solution is; having a separate repository for regression tests doesn't seem helpful. I briefly wondered about having a single version of this test and (if needed) patches to make it build with older PyO3 versions. I'm not sure that'd be any less maintenance in practice.

I wonder, any other ideas how to make this easier to maintain? (e.g. we probably want to cut a new snapshot each release, maybe at least with the patch approach it'd only need new patches every breaking release, but extra overhead of maintaining the patch machinery?)

[Tpt]

I'm not sure that'd be any less maintenance in practice.

Agreed. Having a directory per version basically means that we write once and don't have to touch anymore (outside of dropping backward compatibility intentionally). But it becomes quite verbose. I guess we can also only test versions that have made significant changes (and not all of them).

An other idea might be to not save the source code but either built binaries (not nice) or just the introspection fragments in a JSON file. This way we only have static artifacts. But with the JSON fragment we don't test possible changes in the way the introspection data is stored in the binary...

[davidhewitt]

Slightly crazy idea, do you think we could build a tool which takes a JSON file and "links" it to make a fake binary? I'm not sure how much work it'd be, but then we could have as many JSON snapshots as we like and build them into the various formats we want to test. It'd have the upside of making the various test inputs very readable, perhaps?