Support PYO3_BASE_PYTHON to make builds caching-friendly

[alex]

When a project is built by a PEP 517 frontend with build isolation, the frontend creates a randomly-named temporary virtualenv and the build tool points PYO3_PYTHON inside it. Because pyo3-build-config registers rerun-if-env-changed=PYO3_PYTHON, the ephemeral path defeats cargo's build cache and forces recompilation of the pyo3 crates on every build, even when the underlying interpreter is identical.

Build tools can now additionally export PYO3_BASE_PYTHON pointing at a stable interpreter path (e.g. sys._base_executable). When set, it takes precedence over PYO3_PYTHON, and PYO3_PYTHON is not read at all - so changes to the ephemeral path no longer trigger rebuilds.

Fixes #6113

https://claude.ai/code/session_01JhPTZXLDhuxTSXCZNJuuAA

[ngoldbaum]

Can you add tests to the noxfile to cover the uncovered branches?

[alex]

Ok, (Claude) added a test -- first one was silly so ignore it :-)

[alex]

(netlify stuff is unrelated)

[alex]

Hmm, codecov doesn't seem to have budged, but the new nox test should precisely hit these lines.

[ngoldbaum]

It looks like failed builds don't contribute to coverage?

[alex]

Ugh, is this somehow caused by the netlify thing? That'd be mega annoying.

[ngoldbaum]

No, I think it's because you're doing builds with PYO3_PRINT_CONFIG which causes a build failure and ultimately doesn't contribute to coverage. The test-version-limits job does have successful builds, so it contributes to coverage.

Is there any reason why this can't be a unit test in _impl.py?

But also the test exists so I'm not particularly worried if the coverage report doesn't reflect that for dumb reasons.

[alex]

I could do a unit test, but I think it'd be much less actually validating than these nox tests (which were a good idea, thanks!)

[alex]

@ngoldbaum anything else required to merge this do you think?

[ngoldbaum]

I think it's probably fine but I want to give @davidhewitt chance to weigh in on the idea, since we're just about to cut a release.

[alex]

Thanks much! (For obvious reasons, would love it if this could slip into 0.29!)

[davidhewitt]

Doesn't this run into the same concern as PyO3/setuptools-rust#448 (comment) wrt needing the PEP517 build dependencies as part of the build process?

[alex]

@davidhewitt it shouldn't -- the PYO3_PYTHON env var is still available and still points at the right binary (for use by cryptography/pydantic), it's just that pyo3 itself doesn't need it and so it can get something cache-friendlier.

[davidhewitt]

My worry is things like pyo3_build_config::InterpreterConfig::executable, which will currently be PYO3_PYTHON. I think some build scripts might depend on that executable to launch other Python scripts, which require to be run in the actual build venv?

[alex]

Hmm, that's definitely tougher. The way it's documented doesn't really commit us to it being the in-venv python, but it's quite hard to be sure what users are relying on.

I do think we should look for a way to make this change -- it's a significant improvement in build times.

[alex]

I guess a question is do you think docs (+changelog of course) would be sufficient?

[davidhewitt]

I wonder if we could make pyo3_build_config::get() automatically substitute executable with PYO3_PYTHON (if set)? That might avoid breakage but introduce other weirdness.

Another thought, could we make it opt-in with PYO3_USE_BASE_PYTHON, which is user-set, so even if maturin is populating PYO3_BASE_PYTHON, it's ignored except where users opt in. This would avoid breakage, however maybe it's just layering additional complexity when really there's a more elegant general solution to #1708. I don't think rust-analyzer would set PYO3_BASE_PYTHON env var(s) without user input, for example.

I am genuinely sympathetic to this problem (#1708 is a longstanding issue and big user pain) but I'm not confident this is quite the right patch as-is.

Given that 0.29 is already looking like a massive release, I'd much prefer that we explored this space more and got the right solution into 0.30. I really do want to have better user-facing build times, and I hate to block improvements, but we've lived this far with the build time cost and I'm not yet confident in this pathway.

Please accept my apologies on dragging feet a bit here. I really want to get 0.29 shipped to resolve the security issues and 3.15 beta support, then we can figure out the full design here with clear heads. If we can find a fully-backcompat solution here, maybe we can even land this in 0.29.1 to avoid a longer wait to 0.30.

[alex]

I don't think this should be opt-in -- it means the vast majority of users won't benefit from the gains. My mental model here is: pyo3 only needs base python for the correctness of its build scripts, therefore this should be possible :-) We just need to figure out the backwards compatibility impact on other APIs.

And no worries about punting to 0.30. Cheers