gh-150228: Improve the PEP 829 batch processing APIs by warsaw · Pull Request #150542 · python/cpython

warsaw · 2026-05-28T01:35:08Z

As previously discussed with @ncoghlan and approved for 3.15b2 by @hugovk, this exposes the batch processing APIs for addsitedir() and friends. We remove the defer_processing_start_files flag which required some implicit module global state, and promote StartupState to the public documented API. This removes the need for module global implicit state and allows callers to control when accumulated .start and .pth file state is processed if they want.

This also fixes the interleaving regression identified by @ncoghlan in the same issue. Now, .pth file sys.path extensions are added to sys.path after the sitedir that the .pth file is found in, restoring the legacy behavior.

Along the way, I've made a lot of improvements to function docstrings, site.rst documentation, and comments in the code explaining what's going on.

Issue: PEP 829: resolve the design of the site dir batch processing API #150228

@ncoghlan

As previously discussed with @ncoghlan and approved for 3.15b2 by @hugovk, this exposes the batch processing APIs for addsitedir() and friends. We remove the `defer_processing_start_files` flag which required some implicit module global state, and promote StartupState to the public documented API. This removes the need for module global implicit state and allows callers to control when accumulated .start and .pth file state is processed if they want. This also fixes the interleaving regression identified by @ncoghlan in the same issue. Now, .pth file sys.path extensions are added to sys.path after the sitedir that the .pth file is found in, restoring the legacy behavior. Along the way, I've made a lot of improvements to function docstrings, site.rst documentation, and comments in the code explaining what's going on.

read-the-docs-community · 2026-05-28T01:46:05Z

Documentation build overview

📚 cpython-previews | 🛠️ Build #32909971 | 📁 Comparing d54be32 against main (9242700)

🔍 Preview build

10 files changed · ± 10 modified

± Modified

…NPiO-.rst Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

* Add a note that if known_paths is provided to StartupState.__init__(), it will get mutated in place. * Improve some conditional flows. * Improve some comments. * Improve the what's new entry.

gpshead · 2026-05-29T22:08:38Z

+      Apply the accumulated state by first adding the path extensions to
+      :data:`sys.path`, then executing the :file:`.start` file entry points
+      and :file:`.pth` file ``import`` lines (:ref:`deprecated


I think the ordering here does not match the code in site.py's process():

self._extend_syspath() self._exec_imports() self._execute_start_entrypoints()

make sure this and the docstring and the code all agree. (claude /review flagged this)

I worded it this way because a) I want to consistently emphasize the .start files over the .pth file import lines, and b) because it read better with the parenthesized deprecation note. I've tried to be consistent about a) but maybe I can find a better way to phrase it.

gpshead · 2026-05-29T22:10:12Z



-.. function:: addsitedir(sitedir, known_paths=None, *, defer_processing_start_files=False)
+.. class:: StartupState(known_paths=None)


should StartupState.read_pth_file() and StartupState.read_start_file() be documented? they're public by API name. if not, why not make them _private named?

gpshead · 2026-05-29T22:10:58Z

    The internal data is intentionally private; the public methods
    (read_pth_file, read_start_file, process) are the only supported write
    APIs.


... documented as public here but not in the site.rst docs as methods?

gpshead · 2026-05-29T22:13:01Z

+   Instances of this class are used as an accumulator for interpreter startup
+   configuration data, such as ``.pth`` and ``.start`` files, from one or more
+   site directories.  These are used to batch the processing of these startup
+   files.  The optional *known_paths* argument is a set of case-normalized


How does one get "a set of case-normalized paths" if not using the default? As a public API that feels like a footgun. Accepting an iterable or sequence of paths and doing that normalization for the user would be easier to use.

gpshead · 2026-05-29T22:17:32Z

+   (the default), this set is built from the current :data:`sys.path`.
+   :func:`main` implicitly uses an instance of this class.
+
+   .. method:: process()


API question: calling this twice will re-exec imports and entry point code twice. At a minimum, document that this should not be done and is undefined behavior.

Ideally protect against it. Can we have it consume the internal state (draining the internal path entries, import execs, and entrypoints) so that a repeat call is a no-op?

gpshead · 2026-05-29T22:19:36Z

+            # (callers can pass an empty set), and multiple StartupState
+            # instances against the same sys.path don't share state, so always
+            # do a final anti-duplication check.
+            if dir_ in sys.path:


lack of case normalization and case insensitive filesystems could trip this backstop check up?

gpshead · 2026-05-29T22:28:13Z

+    if startup_state is not None:
+        # Explicit batch mode: accumulate startup data in the caller's state.
+        # The caller is responsible for calling startup_state.process().
+        known_paths = startup_state._known_paths


technically an unused assignment given flush_now = False and what gets returned below. maybe del known_paths instead so it becomes an obvious error if that logic changes? or consider if flush_now and known_paths need to co-exist.

gpshead · 2026-05-29T22:30:46Z

+      and :file:`.pth` file ``import`` lines (:ref:`deprecated
+      <site-pth-files>`).
+
+   .. versionadded:: 3.15


this should be moved up to apply to the whole class, not the method.

gpshead · 2026-05-29T22:32:11Z

+        indexes = [
+            sys.path.index(path) for path in (
+            self.sitedir, extdir1, sitedir2, extdir2
+        )]


Suggested change

indexes = [

sys.path.index(path) for path in (

self.sitedir, extdir1, sitedir2, extdir2

)]

indexes = [

sys.path.index(path) for path in (

self.sitedir, extdir1, sitedir2, extdir2)

]

bedevere-app · 2026-05-29T22:33:42Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

ncoghlan

My approval is for the updated API design - much tidier without the implicit global state. Thanks @warsaw!

For the exact implementation and docs details, +1 to @gpshead's comments and questions (I don't have any strong opinions on how the open questions should be resolved, I just agree there are some details still to be tweaked for consistency)

warsaw self-assigned this May 28, 2026

warsaw requested a review from FFY00 as a code owner May 28, 2026 01:35

warsaw added the 3.15 pre-release feature fixes, bugs and security fixes label May 28, 2026

warsaw requested a review from AA-Turner as a code owner May 28, 2026 01:35

warsaw added 3.16 new features, bugs and security fixes needs backport to 3.15 pre-release feature fixes, bugs and security fixes labels May 28, 2026

bedevere-app Bot mentioned this pull request May 28, 2026

PEP 829: resolve the design of the site dir batch processing API #150228

Open

bedevere-app Bot added the awaiting core review label May 28, 2026

warsaw requested review from hugovk and ncoghlan May 28, 2026 01:35

warsaw added 3 commits May 27, 2026 18:36

Elaborate what's new

ab6f2ac

Merge branch 'main' into gh150228

6cc7ccc

Fix markup typo

498f853

hugovk reviewed May 28, 2026

View reviewed changes

Comment thread Misc/NEWS.d/next/Library/2026-05-27-11-18-36.gh-issue-150228.pNPiO-.rst Outdated

Comment thread Misc/NEWS.d/next/Library/2026-05-27-11-18-36.gh-issue-150228.pNPiO-.rst Outdated

warsaw and others added 2 commits May 27, 2026 22:51

Update Misc/NEWS.d/next/Library/2026-05-27-11-18-36.gh-issue-150228.p…

19b8b15

…NPiO-.rst Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

Update Misc/NEWS.d/next/Library/2026-05-27-11-18-36.gh-issue-150228.p…

e2407d2

…NPiO-.rst Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

warsaw requested a review from hugovk May 28, 2026 05:52

warsaw added 4 commits May 29, 2026 12:22

Fix a typo and point the what's new to the issue not the branch

5405f69

Some minor improvements

c15af5b

* Add a note that if known_paths is provided to StartupState.__init__(), it will get mutated in place. * Improve some conditional flows. * Improve some comments. * Improve the what's new entry.

Merge branch 'main' into gh150228

1821c70

Merge branch 'main' into gh150228

d54be32

gpshead requested changes May 29, 2026

View reviewed changes

bedevere-app Bot added awaiting changes and removed awaiting core review labels May 29, 2026

ncoghlan approved these changes May 29, 2026

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting changes labels May 29, 2026



		.. function:: addsitedir(sitedir, known_paths=None, *, defer_processing_start_files=False)
		.. class:: StartupState(known_paths=None)

Uh oh!

Conversation

warsaw commented May 28, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

read-the-docs-community Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-app Bot commented May 29, 2026

Uh oh!

ncoghlan left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

warsaw commented May 28, 2026 •

edited by bedevere-app Bot

Loading

read-the-docs-community Bot commented May 28, 2026 •

edited

Loading