Fix memory leak: clear managed dict in pybind11_object_dealloc on Python 3.13+ #5999
Open
yamedvedya wants to merge 1 commit intopybind:masterfrom
Open
Fix memory leak: clear managed dict in pybind11_object_dealloc on Python 3.13+ #5999yamedvedya wants to merge 1 commit intopybind:masterfrom
yamedvedya wants to merge 1 commit intopybind:masterfrom
Conversation
On Python 3.14, PyObject_GC_Del (tp_free) no longer implicitly clears the managed dict of objects with Py_TPFLAGS_MANAGED_DICT. Without an explicit PyObject_ClearManagedDict() call before tp_free(), objects stored in the __dict__ of py::dynamic_attr() instances have their refcounts permanently abandoned, causing memory leaks — capsule destructors for numpy arrays (and other objects) never run. Adds a regression test: stores a py::capsule in the __dict__ of a DynamicClass instance and asserts the capsule destructor is called when the instance is deleted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #5998
Problem
On Python 3.14, objects stored in the
__dict__ofpy::dynamic_attr()instances are permanently leaked — their refcounts are abandoned when the pybind11 object is freed, so destructors (including capsule destructors) never run.Root cause:
pybind11_object_dealloc()callstype->tp_free(self)without first callingPyObject_ClearManagedDict(self). On Python ≤ 3.13,PyObject_GC_Del(whattp_freeresolves to for GC-tracked objects) cleared the managed dict as a side effect. On Python 3.14 this implicit clearing was removed, requiring an explicit call intp_dealloc.The existing
pybind11_clear()(which correctly callsPyObject_ClearManagedDict) is only invoked by the cyclic GC viatp_clear— it is never called during normal reference-count-driven deallocation.Fix
// pybind11/detail/class.h — pybind11_object_dealloc()
PyObject_ClearManagedDictis idempotent, so calling it beforetp_freeis safe on all Python 3.13+ versions.Test
Added test_dynamic_attr_dealloc_frees_dict_contents to test_methods_and_attributes:
DynamicClass (py::dynamic_attr())instancepy::capsulein its__dict__; the capsule destructor sets a global flagdel instance+gc_collect()Impact
Any
py::dynamic_attr()class that stores Python objects in instance__dict__from C++ (viaobj.attr("x") = value) is affected. A prominent real-world case is https://gitlab.com/tango-controls/pytango, whereTango::DeviceAttributeusespy::dynamic_attr()to store zero-copy capsule-backed numpy arrays — these arrays leaked completely under Python 3.14 (https://gitlab.com/tango-controls/pytango/-/issues/744).