Defer pygments and pdb imports (9% speedup)

[giampaolo]

Hello,
pytest currently imports pygments and pdb on every run, even though they are only needed in specific cases:

• pygments is only used when there is a failure, for the summary shown at the end, and only when terminal colors are enabled.
• pdb is only used with the --pdb CLI option, and only if a test fails.

Note: pdb became noticeably more expensive to import in Python 3.14, as it now pulls in socket, asyncio, and selectors modules, which together bring in roughly 40 additional modules.

Using this patched pytest version against the psutil test suite (839 tests) with --plugins-disabled and --collect-only, I see about a 9% speedup:

~/svn/pytest {speedup}$ python3 bench_real.py
upstream/main   378.8ms
working tree    343.8ms
-> +35.0ms (+9.2%)

bench_real.py

[Pierre-Sassoulas]

Can't we just import pdb in a conditional branch ? The new class feel unnecessary.

[giampaolo]

Unfortunately no. With a pure import the following tests fail:

FAILED test_pdb_set_trace_interception
FAILED TestDebuggingBreakpoints::test_sys_breakpoint_interception
FAILED test_pdbcls_via_local_module
FAILED test_pdb_wrapper_class_is_reused
FAILED test_quit_with_swallowed_SystemExit

test_pdb_set_trace_interception runs pytest with no options at all and expects import pdb; pdb.set_trace() inside a test to be intercepted. Checking "pdb" in sys.modules at configure time doesn't work either, because test modules are imported during collection, which happens after pytest_configure.

So the only thing left to hook is the import itself, and a sys.meta_path entry has to be an object exposing find_spec.

[bluetech]

The pygments change LGTM, but maybe @nicoddemus should ACK.

The pdb seems to add some complexity. It seems to me like it would perhaps be better to improve this in cpython itself, maybe the imports can be changed to lazy there? I see cpython has an issue Improve import time of various stdlib modules so at least they seem open to it.

[nicoddemus]

I also think the pygments changes are OK, but I agree with @bluetech that the pdb changes are a bit too complex to our liking. After all, we will need to maintain that code moving forward, and who knows what bugs might happen because of it?

I'm -0 on accepting this, while 9% speed up is not negligible, the necessary code to achieve that is not ideal (and this is not saying the code per-se is bad, seems like it is the only solution possible here).

[nicoddemus]

We could leave a note at the global level mentioning that the local imports for pygments are optional, and related to a speed up.

[giampaolo]

@nicoddemus sure, I see where you're coming from. I would probably reject the pdb complexity as well. Happy to remove it and just ship the pygments change if you think it's worth it. That alone should be around a 5% speedup.

[nicoddemus]

Thanks.

Lets give it a day for others to weigh in but if nobody manifests otherwise, go ahead with just keeping the pygments changes.

Btw, how much speedup do the pygments changes (in isolation) provide?

[giampaolo]

Done. The speedup is very modest, so not sure if it's worth it.

~/svn/pytest {speedup}$ python3 bench_real.py 
upstream/main   371.7ms
working tree    364.8ms
-> +6.9ms (+1.9%)