Skip to content

gh-150228: Improve the PEP 829 batch processing APIs #150542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
warsaw wants to merge 10 commits into
base: main
Choose a base branch
from
Open

gh-150228: Improve the PEP 829 batch processing APIs #150542

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
385e132
gh-150228: Improve the PEP 829 batch processing APIs
warsaw May 28, 2026
ab6f2ac
Elaborate what's new
warsaw May 28, 2026
6cc7ccc
Merge branch 'main' into gh150228
warsaw May 28, 2026
498f853
Fix markup typo
warsaw May 28, 2026
19b8b15
Update Misc/NEWS.d/next/Library/2026-05-27-11-18-36.gh-issue-150228.p…
warsaw May 28, 2026
e2407d2
Update Misc/NEWS.d/next/Library/2026-05-27-11-18-36.gh-issue-150228.p…
warsaw May 28, 2026
5405f69
Fix a typo and point the what's new to the issue not the branch
warsaw May 29, 2026
c15af5b
Some minor improvements
warsaw May 29, 2026
1821c70
Merge branch 'main' into gh150228
warsaw May 29, 2026
d54be32
Merge branch 'main' into gh150228
warsaw May 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 51 additions & 10 deletions Doc/library/site.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -356,7 +356,27 @@ Module contents
This function used to be called unconditionally.


.. function:: addsitedir(sitedir, known_paths=None, *, defer_processing_start_files=False)
.. class:: StartupState(known_paths=None)
Comment thread
warsaw marked this conversation as resolved.

Instances of this class are used as an accumulator for interpreter startup
configuration data, such as ``.pth`` and ``.start`` files, from one or more
site directories. These are used to batch the processing of these startup
files. The optional *known_paths* argument is a set of case-normalized
Copy link
Copy Markdown
Member

@gpshead gpshead May 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does one get "a set of case-normalized paths" if not using the default? As a public API that feels like a footgun. Accepting an iterable or sequence of paths and doing that normalization for the user would be easier to use.

paths used to prevent duplicate :data:`sys.path` entries. With ``None``
(the default), this set is built from the current :data:`sys.path`.
:func:`main` implicitly uses an instance of this class.

.. method:: process()
Copy link
Copy Markdown
Member

@gpshead gpshead May 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

API question: calling this twice will re-exec imports and entry point code twice. At a minimum, document that this should not be done and is undefined behavior.

Ideally protect against it. Can we have it consume the internal state (draining the internal path entries, import execs, and entrypoints) so that a repeat call is a no-op?


Apply the accumulated state by first adding the path extensions to
:data:`sys.path`, then executing the :file:`.start` file entry points
and :file:`.pth` file ``import`` lines (:ref:`deprecated
Comment on lines +371 to +373
Copy link
Copy Markdown
Member

@gpshead gpshead May 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the ordering here does not match the code in site.py's process():

    self._extend_syspath()
    self._exec_imports()
    self._execute_start_entrypoints()

make sure this and the docstring and the code all agree. (claude /review flagged this)

Copy link
Copy Markdown
Member Author

@warsaw warsaw May 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worded it this way because a) I want to consistently emphasize the .start files over the .pth file import lines, and b) because it read better with the parenthesized deprecation note. I've tried to be consistent about a) but maybe I can find a better way to phrase it.

<site-pth-files>`).

.. versionadded:: 3.15
Copy link
Copy Markdown
Member

@gpshead gpshead May 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be moved up to apply to the whole class, not the method.



.. function:: addsitedir(sitedir, known_paths=None, *, startup_state=None)

Add a directory to sys.path and parse the :file:`.pth` and :file:`.start`
files found in that directory. Typically used in :mod:`sitecustomize` or
Expand All @@ -366,17 +386,39 @@ Module contents
used to prevent duplicate :data:`sys.path` entries. When ``None`` (the
default), the set is built from the current :data:`sys.path`.

While :file:`.pth` and :file:`.start` files are always parsed, set
*defer_processing_start_files* to ``True`` to prevent processing the
startup data found in those files, so that you can process them explicitly
(this is typically used by the :func:`main` function).
Pass an instance of :class:`StartupState` as *startup_state* to accumulate
startup data from multiple site directories before explicitly processing
with :meth:`StartupState.process`. The *known_paths* and *startup_state*
arguments cannot both be given.

For example:

.. code-block:: python

state = site.StartupState()
for sitedir in site_dirs:
site.addsitedir(sitedir, startup_state=state)
state.process()

Semantics and return values:

* When only *sitedir* is given, startup configuration is processed before
the function returns, and ``None`` is returned.
* When *known_paths* is given, startup configuration is processed before the
function returns, and the updated *known_paths* is returned.
* When *startup_state* is given, startup configuration is **not**
processed, and the state instance is returned. It is up to the caller to
call :meth:`StartupState.process` on this instance.
* It is a :exc:`TypeError` to pass both *known_paths* and *startup_state*.

.. versionchanged:: 3.15

Also processes :file:`.start` files. See :ref:`site-start-files`.
All :file:`.pth` and :file:`.start` files are now read and
accumulated before any path extensions, ``import`` line execution,
or entry point invocations take place.
Also processes :file:`.start` files. See :ref:`site-start-files`. All
:file:`.pth` and :file:`.start` files are now read and accumulated
before any path extensions, entry point invocations, or ``import`` line
execution take place.

The *startup_state* keyword-only argument was added.


.. function:: getsitepackages()
Expand Down Expand Up @@ -447,4 +489,3 @@ value greater than 2 if there is an error.
* :pep:`370` -- Per user site-packages directory
* :pep:`829` -- Startup entry points and the deprecation of import lines in ``.pth`` files
* :ref:`sys-path-init` -- The initialization of :data:`sys.path`.

10 changes: 9 additions & 1 deletion Doc/whatsnew/3.15.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -478,7 +478,15 @@ matching :file:`.start` file is found, ``import`` lines in :file:`.pth` files
are ignored. There is no change to :data:`sys.path` extension lines in
:file:`.pth` files.

(Contributed by Barry Warsaw in :gh:`148641`.)
The :mod:`site` module also provides :class:`site.StartupState` to batch
startup processing for multiple site directories, ensuring all static path
extensions are applied before any startup code is executed. :func:`site.main`
uses an instance of this class implicitly to batch process all startup
configuration files during normal interpreter startup. Callers of
:func:`site.addsitedir` can pass one of these explicitly to control startup
configuration file batch processing.

(Contributed by Barry Warsaw in :gh:`148641` and :gh:`150228`.)


.. _whatsnew315-abi3t:
Expand Down
Loading
Loading