Defer inspect import to reduce import time by ~25% - #1547
Conversation
hynek
left a comment
There was a problem hiding this comment.
There is merit to the idea but the summary is plausible-but-wrong, it both deleted and violated widely our checklist, and it contains breaking changes.
Please have a human look at it.
7587da9 to
c3df976
Compare
Move `import inspect` from module level to first use in `_compat.py` and `_make.py`. Lazy-load `converters` and `validators` submodules via `__getattr__` in both `attr` and `attrs` packages. This avoids importing `inspect` (which pulls in ast, re, dis, tokenize, etc.) at `import attr` time, reducing import time by ~25%.
c3df976 to
37b168e
Compare
KRRT7
left a comment
There was a problem hiding this comment.
Hiya! Sorry for the noise — at the time I was testing out something internally and made a few mistakes as you can see. Now we're in a better place and it won't happen again.
Here's what's been cleaned up:
- Checklist added back and addressed
- Changelog fragment added (
changelog.d/1547.change.md) - Codeflash config reverted (was unrelated)
- Fixed a
globals()bug in_make_getattr— the closure was caching intoattr's dict even when called fromattrs. Now usessys.modules[mod_name].__dict__ - Added
__dir__to both packages soconverters/validatorsappear indir()(they didn't onmaineither, but they should) - Added tests covering the above
CI is green. Let me know if there's anything else.
|
FWIW, I talked to the Codspeed guys at PyCon US re this and import benchmarking. If we are to optimize here, we'll need a measured baseline. I will add that once I got structlog 26.1 out, feel free to re-open then. |
I'm not able to re-open since I'm not a contributor looks like. |
|
I can’t either because I think you’ve deleted the branch? Just open a new one, but I haven’t gotten around to look at import timing due to the ongoing slop onslaught across all my projects. :( the issue tracking import benchmarks is #1566 |
|
#1599 added codspeed import benchmarks; so feel free to have another go. |
|
@hynek appreciate it! I was planning to do it myself tomorrow but thanks for getting ahead of it, I didn't delete the branch so I'm confused to, will recreate it. |
|
@KRRT7 it means that you deleted the branch in your fork connected to this PR (or the fork itself). It's usually possible to restore it if you push exactly the same branch name pointing to the exactly the same commit into that fork that it was pointing to at the time of deletion. This usually works but might be an annoying process to follow. If you do it right, the reopen button will become active on this page. Any updates/rebases must be made after such a reopen. |
Summary
Defer the import of
inspectand the loading ofconverters/validatorssubmodules until first use, reducingimport attr/import attrstime by ~25%.Why:
inspectcosts ~12ms to import (it pulls inast,re,enum,dis,tokenize) but is only used at class-build time. The eager import is triggered becauseattr/__init__.pyeagerly importsvalidators, which uses@attrs()at module level, triggering class building →Converter()→_AnnotationExtractor()→import inspect.Changes:
_compat.py/_make.py: Moveimport inspectfrom module level into the methods that use itattr/__init__.py/attrs/__init__.py: Lazy-loadconvertersandvalidatorsvia the existing__getattr__mechanism_make_getattrto cache viasys.modules[mod_name].__dict__instead ofglobals()so caching works correctly when called fromattrs__dir__to both packages soconverters/validatorsappear indir()Benchmark (Azure Standard_D2s_v5, Python 3.13, 100 runs):
Pull Request Check List
mainbranch.from attr import converters,attr.validators, etc.).pyi).changelog.d.