gh-150228: Improve the PEP 829 batch processing APIs

As previously discussed with @ncoghlan and approved for 3.15b2 by @hugovk,
this exposes the batch processing APIs for addsitedir() and friends.  We
remove the `defer_processing_start_files` flag which required some implicit
module global state, and promote StartupState to the public documented API.
This removes the need for module global implicit state and allows callers to
control when accumulated .start and .pth file state is processed if they want.

This also fixes the interleaving regression identified by @ncoghlan in the
same issue.  Now, .pth file sys.path extensions are added to sys.path after
the sitedir that the .pth file is found in, restoring the legacy behavior.

Along the way, I've made a lot of improvements to function docstrings,
site.rst documentation, and comments in the code explaining what's going on.

Author: warsaw

Doc/library/site.rst

@@ -356,7 +356,27 @@ Module contents
       This function used to be called unconditionally.
 
 
-.. function:: addsitedir(sitedir, known_paths=None, *, defer_processing_start_files=False)
+.. class:: StartupState(known_paths=None)
+
+   Instances of this class are used as an accumulator for interpreter startup
+   configuration data, such as ``.pth`` and ``.start`` files, from one or more
+   site directories.  These are used to batch the processing of these startup
+   files.  The optional *known_paths* argument is a set of case-normalized
+   paths used to prevent duplicate :data:`sys.path` entries.  With ``None``
+   (the default), this set is built from the current :data:`sys.path`.
+   :func:`main` implicitly uses an instance of this class.
+
+   .. method:: process()
+
+      Apply the accumulated state by first adding the path extensions to
+      :data:`sys.path`, then executing the :file:`.start` file entry points
+      and :file:`.pth` file ``import`` lines (:ref:`deprecated
+      <site-pth-files>`).
+
+   .. versionadded:: 3.15
+
+
+.. function:: addsitedir(sitedir, known_paths=None, *, startup_state=None)
 
    Add a directory to sys.path and parse the :file:`.pth` and :file:`.start`
    files found in that directory.  Typically used in :mod:`sitecustomize` or
@@ -366,17 +386,39 @@ Module contents
    used to prevent duplicate :data:`sys.path` entries.  When ``None`` (the
    default), the set is built from the current :data:`sys.path`.
 
-   While :file:`.pth` and :file:`.start` files are always parsed, set
-   *defer_processing_start_files* to ``True`` to prevent processing the
-   startup data found in those files, so that you can process them explicitly
-   (this is typically used by the :func:`main` function).
+   Pass an instance of :class:`StartupState` as *startup_state* to accumulate
+   startup data from multiple site directories before explicitly processing
+   with :meth:`StartupState.process`.  The *known_paths* and *startup_state*
+   arguments cannot both be given.
+
+   For example:
+
+   .. code-block:: python
+
+      state = site.StartupState()
+      for sitedir in site_dirs:
+          site.addsitedir(sitedir, startup_state=state)
+      state.process()
+
+   Semantics and return values:
+
+   * When only *sitedir* is given, startup configuration is processed before
+     the function returns, and ``None`` is returned.
+   * When *known_paths* is given, startup configuration is process before the
+     function returns, and the updated *known_paths* is returned.
+   * When *startup_state* is given, startup configuration is **not**
+     processed, and the state instance is returned.  It is up to the caller to
+     call :meth:`StartupState.process` on this instance.
+   * It is a :exc:`TypeError` to pass both *known_paths* and *startup_state*.
 
    .. versionchanged:: 3.15
 
-      Also processes :file:`.start` files.  See :ref:`site-start-files`.
-      All :file:`.pth` and :file:`.start` files are now read and
-      accumulated before any path extensions, ``import`` line execution,
-      or entry point invocations take place.
+      Also processes :file:`.start` files.  See :ref:`site-start-files`.  All
+      :file:`.pth` and :file:`.start` files are now read and accumulated
+      before any path extensions, entry point invocations, or``import`` line
+      execution take place.
+
+      The *startup_state* keyword-only argument was added.
 
 
 .. function:: getsitepackages()
@@ -447,4 +489,3 @@ value greater than 2 if there is an error.
    * :pep:`370` -- Per user site-packages directory
    * :pep:`829` -- Startup entry points and the deprecation of import lines in ``.pth`` files
    * :ref:`sys-path-init` -- The initialization of :data:`sys.path`.
-

Doc/whatsnew/3.15.rst

@@ -478,6 +478,14 @@ matching :file:`.start` file is found, ``import`` lines in :file:`.pth` files
 are ignored.  There is no change to :data:`sys.path` extension lines in
 :file:`.pth` files.
 
+The :mod:`site` module also provides :class:`site.StartupState` to batch
+startup processing for multiple site directories, ensuring all static path
+extensions are applied before any startup code is executed.  :func:`site.main`
+uses an instance of this class implicitly to batch process all startup
+configuration files during normal interpreter startup.  Callers of
+:func:`site.addsitedir` can pass one of these explicitly to control startup
+configuration file batch processing.
+
 (Contributed by Barry Warsaw in :gh:`148641`.)