Skip to content

Experimental _pytest.ensemble API: nested configs + in-memory collection (PoC) - #14809

Draft
RonnyPfannschmidt wants to merge 15 commits into
pytest-dev:mainfrom
RonnyPfannschmidt:scenario-api-poc
Draft

Experimental _pytest.ensemble API: nested configs + in-memory collection (PoC)#14809
RonnyPfannschmidt wants to merge 15 commits into
pytest-dev:mainfrom
RonnyPfannschmidt:scenario-api-poc

Conversation

@RonnyPfannschmidt

@RonnyPfannschmidt RonnyPfannschmidt commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Important

Substantially reworked since the first review round (@bluetech, thanks). The
package was named _pytest.scenario; it is now _pytest.ensemble, and the
single squashed commit is now a 7-commit stack on top of current main, so the
precursor changes can be reviewed — and landed — on their own. See
What changed since the first round at the
bottom for the full delta.

Context

pytest's own testsuite is heavily integration-based: of ~3200 real test functions in testing/, ~61% use pytester, ~91% of those write files to disk, and roughly 1400 assertion sites glob-match rendered terminal output (vs ~270 structured ones). Tests need a full session because stdout is the only observable most of them have — there is no API to collect an item from an in-memory object, or to build a Config without a real rootdir/cwd. testing/conftest.py even reorders tests by decompiling __code__.co_names because "uses pytester" is the only available proxy for "slow".

This PR is a proof of concept for fixing that at the API level. Draft — the API shape is up for discussion.

What an ensemble is

A pytest ensemble is a deliberately small pytest assembled from parts handed to it, rather than a full session discovered from a filesystem. The chamber-ensemble-vs-orchestra reading is the intended one: the reduced plugin set is a design property, not a limitation to apologise for.

def test_static_method(tmp_path):
    class Test:
        @staticmethod
        def test_something(): ...
        @pytest.fixture
        def fix(self): return 1
        @staticmethod
        def test_fix(fix): assert fix == 1

    record = run_tests(Test, rootpath=tmp_path)
    record.assert_outcomes(passed=2)

Three layers, all internal and not exported from pytest:

  • Config — ConfigSpec is plain frozen data describing a nested configuration; configured() turns it into a parsed and configured Config with paired teardown via _ensure_unconfigure(). Built through the real parse phases, but without rootdir discovery, config files, conftests, plugin autoload, or env consultation.
  • Collection — build_module(name, *members) groups loose functions and classes into an in-memory module with an explicit name. collect_tests() / run_tests() / the stepwise Ensemble context manager collect through the standard pytest_collection flow — so -k/-m, parametrize, fixture scoping and pytest_collection_modifyitems all work — via an EnsembleModule whose _getobj serves the preset object. Nodeids derive from synthetic rootdir-relative paths that are never touched on disk.
  • Results — RunRecord / ItemRecord, derived from real report objects via pytest_report_teststatus, with assert_outcomes() signature-compatible with pytester and per-test lookup (record["test_x"].setup.longreprtext).

The commit stack

Reviewable and landable bottom-up; every commit is independently green (full suite, mypy, and the whole pre-commit hook set):

commit notes
1 config: split Config.parse into phase methods behavior-preserving; only observable change is the frame a conftest-load warning is attributed to
2 config: let get_config() take an explicit invocation dir 3 lines
3 python: make Class.from_parent honor a passed obj it was accepted and dropped
4 debugging: trace via pytest_runtest_call instead of pytest_pyfunc_call net deletion, see below
5 ensemble: build a hermetic nested Config from declarative data ConfigSpec/configured() + ArgsSource.SPEC
6 ensemble: collect and run in-memory sources, record typed results rest of the package
7 testing: convert pytester-based tests to _pytest.ensemble seven conversions
8 bench: compare pytester and ensemble run costs outside testpaths, never runs in CI

Commits 1–4 are independent of the ensemble work and I'm happy to peel any of them into their own PR on request. They are here only so the stack builds.

Commit 4 deserves a note, because it turned out to be a real bug rather than a workaround. TestCaseFunction.runtest never calls pytest_pyfunc_call, so PdbTrace's wrapper never reached unittest test cases; unittest.py compensated by importing _pytest.debugging and re-deriving "is --trace active" from the raw option. Wrapping pytest_runtest_call instead covers both kinds of item, and maybe_wrap_pytest_function_for_tracing plus the cross-plugin import are deleted outright. The wrapper must be trylast so it nests inside CaptureManager's — otherwise _init_pdb()'s capture suspend is immediately undone when the capture manager starts the call phase, which test_pdbcls_via_local_module catches.

Validation

43 self-tests in testing/test_ensemble.py run in ~0.8s with hermeticity asserted (no sys.path/sys.modules/cwd/environ mutation, no files created). Seven existing tests in testing/python/{collect,fixtures,metafunc}.py are converted, including the hand-rolled make_function harness — which existed precisely because there was no API to collect an item from an in-memory object.

What the harness costs

bench/ensemble_vs_pytester.py contrasts the arms at 1, 10 and 100 test functions per run. Splitting fixed cost from marginal cost is what makes the comparison legible:

arm fixed cost per run marginal per test files written
makepyfile + runpytest_subprocess 537 ms 2.00 ms 1
makepyfile + inline_run 300 ms 1.98 ms 1
makepyfile + runpytest_inprocess 258 ms 2.77 ms 1
run_tests (ensemble) 10 ms 1.00 ms 0
makepyfile alone 0.10 ms 0.002 ms 1

The fixed column is the point. At one test per run — which is what most pytester-based tests are — an ensemble is ~25x cheaper than inline_run and ~52x cheaper than runpytest_subprocess. By 100 tests the ratio falls to ~13x, as real test execution starts to dominate. With roughly 1900 pytester-based tests in our own suite, that bootstrap is paid ~1900 separate times.

Note also what the makepyfile arm shows: writing the module costs 0.10 ms. "pytester touches the disk" is a hermeticity argument, not a performance one — the performance argument is config construction, plugin loading, conftest discovery, capture and terminal setup.

Two caveats. Ensembles skip assertion rewriting entirely (a documented limitation below, and part of why they are cheaper), and the fixed/marginal split is a two-point estimate rather than a regression fit. The absolute numbers are from one machine; the ratios are the transferable part.

Deliberate scope cuts (documented follow-ups)

  • The default plugin set excludes capture, terminal, assertion, cacheprovider and the process-global-state plugins. capsys/capfd inside an ensemble are unavailable until CaptureManager becomes stack-aware (suspend/resume is currently absolute, restoring values memoised at __init__ — nested managers work only by LIFO luck today). That is the next pillar, and it is the same work the free-threading effort needs.
  • Conftest loading (load_conftests=True) raises NotImplementedError; plugin objects via ConfigSpec.extra_plugins are the replacement.
  • Host process warning filters (e.g. our own filterwarnings = error) are inherited; an ensemble's inicfg={"filterwarnings": [...]} takes precedence.
  • RunRecord currently has three lookup paths (by_test, _by_name, __getitem__); that surface wants tightening before anything else builds on it.
  • Longer term: rebase pytester's inline_run onto this API, migrate stdout outcome-assertions to RunRecord, and consider graduating the package to public.

What changed since the first round

Addressing #14809 (comment) point by point:

"scenario sounds too generic / like an end-user feature." Agreed — renamed to ensemble. Considered and rejected: subpytest, rig, embed, compose, kernel, machinery, and setups (which collides with pytest's four existing meanings of "setup", including determine_setup — the very function configured() exists to bypass). Member names stay deliberately literal so the metaphor lives only in the package name: Ensemble, build_module(), collect_sources(), RunRecorder, RunRecord/ItemRecord.

"ConfigSpec/configured() would be nice in a separate commit or PR." Done — commit 5, on a clean boundary above the parse split.

"The three drive-by fixes look like nice independent improvements; consider separate PRs." They are now commits 1–4 and can be peeled on request. Commit 4 changed shape entirely in the process: instead of defensively defaulting the trace option read, it removes the read (and the cross-plugin call) by fixing the hook bypass that caused it. Only the usepdb read remains, as an explicit default — postponing tearDown is genuine unittest logic that has to know pdb is live.

"What's the reason to use separate types than the TestReport etc.?" They're exploratory, and the goal is the opposite of a parallel hierarchy: I want to find out whether pytester's result handling can become a thin wrapper over these. Keeping them distinct from TestReport/RunResult for now is what makes that experiment legible — RunRecord is an aggregate over real report objects, not a replacement for them, and nothing is scraped from rendered output.

"Ideally we can have this nice stuff in pytester from the start." Long-term agreed. pytester is very intertwined with the problematic way to run things and completely dependent on capture; I wanted to start without that as a limit in order to explore the details first. Rebasing inline_run onto this is on the follow-up list above.

"These sound like they can share the benefits with the threading work." Yes — stack-aware CaptureManager is exactly the same prerequisite.

Per our AI contribution policy: this was researched, designed and implemented with Claude Code under my direction and review.

@psf-chronographer psf-chronographer Bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Jul 31, 2026
@bluetech

bluetech commented Aug 3, 2026

Copy link
Copy Markdown
Member

Nice initiative. Anything that brings more isolation, hermeticity and control of side-effects seems like a good direction to me.

I haven't dug into the code or details yet, so just some small comments:

The name "scenario" sounds possibly too generic for this, it doesn't sound like sometime that is intended for testing pytest itself, but like an end-user feature, like pytest-bdd has @scenario for example.

ConfigSpec / configured() — a parsed+configured nested Config from declarative data, built through the real parse phases but without rootdir discovery, config files, conftests, plugin autoload, or env consultation. Hermetic by default; teardown is paired via _ensure_unconfigure().

This sounds possibly nice but didn't look at the details. In due time, it would be nice to have this in a separate commit or PR to be considered separately.

fake_module(name, *members) — group loose functions/classes into an in-memory module with an explicit name; collect_tests() / run_tests() / stepwise scenario() collect through the standard pytest_collection flow (so -k/-m, parametrize, fixture scoping and pytest_collection_modifyitems all work) via a ScenarioModule whose _getobj serves the preset object. Nodeids derive from synthetic rootdir-relative paths that are never touched on disk.

👍 on the general idea.

RunRecord / ItemRunRecord — typed results derived from real report objects via pytest_report_teststatus, with assert_outcomes() signature-compatible with pytester and per-test lookup (record["test_x"].setup.longreprtext).

What's the reason to use separate types than the TestReport etc. themselves? I do believe pytester has some helpers to grab those, although we don't use them as much as we should.

Regarding these:

  • Class.from_parent() no longer silently discards a passed obj (it was accepted and dropped).
  • get_config() gained an explicit invocation dir= (was hardcoded Path.cwd()).
  • debugging/unittest guard their cross-plugin option reads (trace/usepdb) so they work when the debugging plugin is not registered.

These all sound (just based on the description) look nice independent improvements. Consider submitting as separate PRs, to reduce the changes here.

The default scenario plugin set excludes capture, terminal, assertion, cacheprovider and the process-global-state plugins. capsys/capfd inside scenarios are unavailable until CaptureManager becomes stack-aware (suspend/resume is currently absolute, restoring values memoised at init — nested managers work only by LIFO luck today). That is the next pillar.

These sound like they can share the benefits with the threading work.

Longer term: rebase pytester's inline_run onto this API, migrate stdout outcome-assertions to RunRecord, and consider pytest.scenario graduation.

Ideally we can have this nice stuff in pytester from the start, instead of a separate API, just to keep things simple for users (mostly plugin authors). But maybe there's good reason to have a separate API, I don't know yet.

@RonnyPfannschmidt

RonnyPfannschmidt commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

This is a initial poc

It includes a number of precursor enhancements that need to be extracted and landed first

As to why new types/apis - pytester is very intertwined with the problematic way to run things and completely dependent on capture- i wanted to start without that as limit to explore the details first

@RonnyPfannschmidt
config: split Config.parse into phase methods
c030bd0 
Extract the body of ``Config.parse`` into named phase methods
(``_preparse_addopts``, ``_apply_rootdir``, ``_register_core_ini_options``,
``_load_plugins_phase``, ``_load_initial_conftests_phase``,
``_finalize_parse``) so the individual steps can be driven separately by
callers that build a configuration programmatically instead of from a
command line.

Behavior preserving: ``parse()`` calls the phases in the same order and
with the same arguments as before. The only observable change is the
function name a conftest load warning is attributed to, which
test_warnings asserts on.

``known_args_namespace`` gains an explicit class-level annotation because
the phases that read it are now defined before the one that assigns it.
@RonnyPfannschmidt
config: let get_config() take an explicit invocation dir
809826d 
``get_config()`` hardcoded ``Path.cwd()`` as the invocation directory,
which forces any programmatic caller to either chdir or accept whatever
the ambient working directory happens to be. Accept an explicit ``dir=``
instead, defaulting to the previous behavior.
@RonnyPfannschmidt
python: make Class.from_parent honor a passed obj
a4205eb 
``Class.from_parent`` accepted an ``obj`` keyword and dropped it on the
floor, so the collector always resolved its object by looking ``name`` up
on the parent's object. Keep the passed object and serve it from
``_getobj``, which makes it possible to collect a class that is not an
attribute of any importable module.
@RonnyPfannschmidt
debugging: trace via pytest_runtest_call instead of pytest_pyfunc_call
f215b4c 
``TestCaseFunction.runtest`` never calls ``pytest_pyfunc_call``, so
``PdbTrace``'s wrapper did not reach unittest test cases and
``unittest.py`` compensated by reaching into ``_pytest.debugging`` and
re-deriving "is --trace active" from the raw option value.

Wrap ``pytest_runtest_call`` instead, which covers both kinds of item, and
drop ``maybe_wrap_pytest_function_for_tracing`` along with the
cross-plugin import. The wrapper is ``trylast`` so it nests inside
``CaptureManager``'s wrapper: ``_init_pdb()`` suspends capturing, and an
outer wrapper would have that immediately undone when the capture manager
starts the call phase.

The remaining ``usepdb`` read gets an explicit default, so that reading it
does not require the debugging plugin to be registered.
@RonnyPfannschmidt
ensemble: build a hermetic nested Config from declarative data
4d15b35 
Add the experimental, internal ``_pytest.ensemble`` package, starting with
its configuration layer: ``ConfigSpec`` describes a nested pytest
configuration as plain frozen data, and ``configured()`` turns it into a
parsed *and* configured ``Config`` with paired teardown.

The config is built through the same parse phases a command line
invocation uses, but from the spec's explicit values only - no rootdir
discovery, config file reading, conftest loading, plugin autoloading or
environment variable consultation. The default plugin set is the essential
core plus the plugins that give tests their usual semantics, deliberately
excluding everything that renders output, captures io or installs
process-global state.

``Config`` gains ``ArgsSource.SPEC`` and ``_finalize_parse(decide_args=False)``
so a programmatic caller can take positional args verbatim instead of
falling back to testpaths or the invocation directory.
@RonnyPfannschmidt
ensemble: collect and run in-memory sources, record typed results
684eb36 
Add the collection and results layers of ``_pytest.ensemble``.

``build_module(name, *members)`` groups loose functions and classes into
an in-memory module with an explicit name. ``EnsembleModule`` is a
``Module`` collector whose synthetic path is rootdir-relative - giving
well-formed nodeids - but never touched on disk, serving the preset object
from ``_getobj`` instead of going through the import chokepoint.

``collect_sources()`` feeds those collectors to the standard
``pytest_collection`` flow, so ``-k``/``-m`` deselection, parametrization,
fixture scoping and ``pytest_collection_modifyitems`` all apply as usual.
``run_items()`` drives the normal runtest protocol.

``RunRecorder`` observes reports, warnings and deselections; ``RunRecord``
and ``ItemRecord`` derive typed results from the real report objects via
``pytest_report_teststatus``, with per-test lookup and an
``assert_outcomes()`` that is signature compatible with pytester's.

``Ensemble`` ties the two together as a context manager for stepwise use;
``run_tests()`` and ``collect_tests()`` are the one-shot forms.
@RonnyPfannschmidt
testing: convert pytester-based tests to _pytest.ensemble
4fc76a7 
Convert seven existing tests in testing/python/ to build their sources as
real python objects and assert on typed records, instead of writing files
to disk and glob-matching rendered terminal output. This includes the
hand-rolled ``make_function`` harness in test_metafunc, which existed
precisely because there was no API to collect an item from an in-memory
object.
@RonnyPfannschmidt RonnyPfannschmidt changed the title Experimental _pytest.scenario API: nested configs + in-memory collection (PoC) Experimental _pytest.ensemble API: nested configs + in-memory collection (PoC) Aug 13, 2026
@RonnyPfannschmidt
bench: compare pytester and ensemble run costs
d87783e 
Add a benchmark contrasting the pytester harness with _pytest.ensemble
across five arms - runpytest_subprocess, runpytest_inprocess, inline_run,
run_tests, and a bare makepyfile so the file materialization cost can be
subtracted - at 1, 10 and 100 test functions per run.

Reporting fixed cost per run separately from marginal cost per test is
what makes the comparison useful: pytester's cost is dominated by session
bootstrap, which every one of the ~1900 pytester-based tests in our own
suite pays individually.

Lives in bench/, which is outside testpaths, so it does not run in CI.
@RonnyPfannschmidt
ensemble: apply addopts and ini overrides when building a config
5714234 
configured() replicated the ini setup of Config.parse but skipped the step
that prepends `addopts` and folds `--override-ini` values back into the
inifile config. Every OverrideIniAction option - --strict, --strict-markers,
--strict-config - and every `-o name=value` was therefore parsed into the
namespace and then dropped on the floor.

The failure mode is the bad one: a spec asking for --strict-markers produced
a config that silently ignored it, so a test written against that spec would
pass while asserting nothing.

Note that a `@pytest.mark.unregistered` decorator in the enclosing test body
cannot be used to check this, because MarkGenerator resolves it against the
host config at decoration time; `-m` expression validation is the reachable
enforcement point, and the test says so.
@RonnyPfannschmidt
config, debugging: render without a terminal reporter
0fa25e3 
Three places assumed the terminal plugin is registered, which a
programmatically constructed config need not load:

- Config.get_terminal_writer() asserted a terminalreporter exists. It now
  falls back to create_terminal_writer(self), so code that needs to render
  something does not have to care whether anything is reporting.
- create_terminal_writer() read option.color / option.code_highlight
  directly, but those are registered by the terminal plugin, so the
  fallback above would have crashed on the very configs it exists for.
  Read defensively; a writer is still useful without them.
- debugging._enter_pdb() reached into terminalreporter._tw and read
  option.showcapture. Entering the debugger on a failing test in a
  terminal-less config raised AttributeError instead of debugging.

--pdb now works end to end in such a config.
@RonnyPfannschmidt
setuponly, setupplan: normalize options at configure time
be41349 
Both plugins turned their own flag into the related ones from
pytest_cmdline_main: --setup-plan implies --setup-only and --setup-show,
--setup-only implies --setup-show. That hook only runs for command line
invocations, so a programmatically constructed config accepted the flags
and then behaved as if they had not been passed - the runner reads
setuponly/setupshow and saw False.

Move the normalization to pytest_configure. Every consumer reads these
options at runtest or report time, well after configure, so the behavior
for a command line run is unchanged; configs that are configured without
going through the entry point now get it too.
@RonnyPfannschmidt
ensemble: do not let a config take ownership of a spec's ini values
855fc16 
configured() put the caller's own list objects into the config's ini
cache. Config.addinivalue_line appends to that cached list, so a
ConfigSpec - a frozen dataclass, and therefore reasonably expected to be
reusable - grew every time it was configured: three uses of one spec took
its `markers` list from 9 entries to 17 to 25.

Copy mutable values on the way in, for both plain lists and preconstructed
ConfigValues.
@RonnyPfannschmidt
ensemble: stop swallowing collection failures
154d416 
collect_tests() returned an empty list when collection failed, with no
exception and no way to reach the failed CollectReport. "Collected
nothing" and "collection blew up" were therefore indistinguishable, so a
test asserting the former would pass for entirely the wrong reason - and
that assertion shape is common in the suite this API is meant to serve.

Report the failure through whichever channel each entry point has:

- collect_tests() has none, so it raises CollectError.
- Ensemble.collect() stays permissive - it is the stepwise API - but the
  failures are now reachable as Ensemble.collect_errors.
- run_tests() keeps recording them; a collection error is a legitimate
  outcome there. RunRecord.collect_errors filters the failed reports.

Collecting nothing because nothing matched remains perfectly fine.
@RonnyPfannschmidt
ensemble: make Ensemble.collect() idempotent
7959c87 
Calling collect() twice registered the ensemble's sources a second time,
so the session ended up with every test collected twice - a stepwise use
that collects, inspects, then collects again silently doubled its items
and its outcome counts.

A repeat call without new sources now returns what was already collected.
Passing new sources still adds them, under a suffixed module name so that
loose sources in a later round do not land on a module path already in
the tree.
@RonnyPfannschmidt
ensemble: give an ensemble its own output sink
151f512 
Adds `capture_output=True`, which loads the terminal plugin and gives it a
buffer to render into: RunRecord.output is the text, RunRecord.stdout a
LineMatcher, so fnmatch_lines assertions can be kept verbatim rather than
rewritten against the structured reports.

The stream is bound when the terminal reporter is constructed, not
redirected around it. terminal.pytest_configure hardcoded
TerminalReporter(config, sys.stdout), so a nested run would have taken the
stdout of whatever was running it - and would have computed isatty() from
it too. It now reads terminal_file_key from the config stash, which
ConfigSpec.output populates before the config is configured; a plain
command line run is unaffected.

An ensemble therefore renders into its own buffer and never writes to the
outer stdout at all, which a tripwire test pins down.
@RonnyPfannschmidt RonnyPfannschmidt mentioned this pull request Aug 13, 2026
Draft
@RonnyPfannschmidt

RonnyPfannschmidt commented Aug 13, 2026

Copy link
Copy Markdown
Member Author

RonnyPfannschmidt#140 tests effects of testsuite porting - its promising

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nicoddemus nicoddemus Awaiting requested review from nicoddemus
@bluetech bluetech Awaiting requested review from bluetech

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

bot:chronographer:provided (automation) changelog entry is part of PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@RonnyPfannschmidt @bluetech