Cursor-generated: Add leak_detach path for internals after shutdown

[rwgk]

Description

Py_IsInitialized/Py_IsFinalizing are runtime-wide, not per-interpreter, and Py_IsFinalizing stays set after Py_Finalize until the next init. Using those checks to decide whether to DECREF internals can skip cleanup or touch the C-API after a specific interpreter is gone.

Add internals::leak_detach() and local_internals::leak_detach() to clear the owned PyType/PyObject pointers without calling into Python, and invoke that from internals_pp_manager::destroy() before deleting the pp in the post- Py_Finalize/Py_EndInterpreter paths. The destructors now always Py_CLEAR when invoked during state-dict teardown, preserving cleanup of the pybind11 heap types while avoiding UB in late-destroy scenarios.

---

Cursor GPT-5.2 Codex Extra High

 ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
 │ → Could you please read this file for context: /wrk/logs/gh_pr_info_pybind_pybind11_5958_2026-01-19+100747.md                                                                    │
 │                                                                                                                                                                                  │
 │   In my last comment there I wrote: I'll point Cursor to this, the pybind11 sources, and the CPython 3.14 sources, and ask it to look very thoroughly to figure out what is the  │
 │   best achievable solution in terms of avoiding UB but leaking as little as possible.                                                                                            │
 │                                                                                                                                                                                  │
 │   The up-to-date pybind11 sources are right here (master branch).                                                                                                                │
 │                                                                                                                                                                                  │
 │   The CPython sources are here:                                                                                                                                                  │
 │                                                                                                                                                                                  │
 │   smc120-0009.ipp2a2.colossus.nvidia.com:/wrk/clone/cpython $ git branch                                                                                                         │
 │   * 3.14                                                                                                                                                                         │
 │     main                                                                                                                                                                         │
 │     v3.14.2_release                                                                                                                                                              │
 │                                                                                                                                                                                  │
 │   JIC we need this for your analysis:                                                                                                                                            │
 │                                                                                                                                                                                  │
 │   I last git pulled the CPython 3.14 branch yesterday and built from sources using this script:                                                                                  │
 │                                                                                                                                                                                  │
 │   /home/rgrossekunst/rwgk_config/bin/build_cpython_from_git_twice.sh                                                                                                             │
 │                                                                                                                                                                                  │
 │   The installations are here:                                                                                                                                                    │
 │                                                                                                                                                                                  │
 │   /wrk/cpython_installs/3.14_branch_23e3c0ae867_default                                                                                                                          │
 │   /wrk/cpython_installs/3.14_branch_23e3c0ae867_freethreaded                                                                                                                     │
 │                                                                                                                                                                                  │
 │   Could you please "look very thoroughly to figure out what is the best achievable solution in terms of avoiding UB but leaking as little as possible.", also taking Joshua's "I │
 │    think the correct solution is a internals::leak_detach() method" comment into account?                                                                                        │
 │                                                                                                                                                                                  │
 │   Please use the cursor_workspace/ subdirectly here for all intermediate/temporary files. Do NOT use /tmp.                                                                       │
 │                                                                                                                                                                                  │
 │   If you want to rebuild the pybind11 unit tests for your analysis, please do NOT use cmake. Use this command instead:                                                           │
 │                                                                                                                                                                                  │
 │   cd /wrk/bld/pybind11_gcc_v3.14.2_df793163d58_default/ && scons -j $(nproc) && TestVenv/bin/python3 ../../clone/pybind11_scons/run_tests.py ../../forked/pybind11 24            │
 │                                                                                                                                                                                  │
 │   Currently I want to focus on the "default" cpython build. We can leave any "freethreaded" considerations for later.                                                            │
 └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Suggested changelog entry:

• Placeholder.

[rwgk]

@b-pass @oremanj

I'm dumping here what Cursor showed on the screen:

  ⬢ To-do All done
    ☒ Inspect pybind11 internals shutdown paths
    ☒ Check subinterpreter lifecycle helpers
    ☒ Review CPython finalization semantics
    ☒ Derive safest leak-minimizing approach
    ☒ Propose or implement pybind11 changes

  ⬢ Read 7 files
    … 4 earlier items hidden
    Read /wrk/clone/cpython/Python/pylifecycle.c lines 2069-2098
    Read include/pybind11/embed.h lines 237-296
    Read include/pybind11/subinterpreter.h lines 161-210

  Implemented a leak_detach() path so internals can be destroyed after interpreter teardown without touching the Python C-API, while still letting the normal state‑dict teardown
  decref the three PyTypes.
  Why this is needed
  • Py_IsFinalizing() is runtime‑wide and stays set until the next init, so it cannot tell you whether a specific interpreter is still safe to
    touch.`144:147:/wrk/clone/cpython/Python/pylifecycle.c

  int
  Py_IsFinalizing(void)
  {
      return _PyRuntimeState_GetFinalizing(&_PyRuntime) != NULL;
  }

   ```494:503:/wrk/clone/cpython/Python/pylifecycle.c

     1 │/* Py_Finalize leaves _Py_Finalizing set in order to help daemon
     2 │ * threads behave a little more gracefully at interpreter shutdown.
     3 │ * We clobber it here so the new interpreter can start with a clean
     4 │ * slate.
     5 │ */
     6 │_PyRuntimeState_SetFinalizing(runtime, NULL);

  • pybind11 explicitly destroys internals after Py_Finalize/Py_EndInterpreter, so if internals are re‑created during shutdown their destructors can run after the interpreter is
    gone.`243:266:/wrk/forked/pybind11/include/pybind11/embed.h

  inline void finalize_interpreter() {
      // ...
      Py_Finalize();
      detail::get_internals_pp_manager().destroy();
      // Local internals contains data managed by the current interpreter, so we must clear them to
      // avoid undefined behaviors when initializing another interpreter
      detail::get_local_internals_pp_manager().destroy();

   ```176:190:/wrk/forked/pybind11/include/pybind11/subinterpreter.h

     1 │        // End it
     2 │        Py_EndInterpreter(destroy_tstate);
     3 │
     4 │        // It's possible for the  internals to be created during endinterpreter (e.g. if a
     5 │        // py::capsule calls `get_internals()` during destruction), so we destroy afterward.
     6 │        detail::get_internals_pp_manager().destroy();
     7 │        detail::get_local_internals_pp_manager().destroy();

  What changed
  • Added leak_detach() to internals and local_internals, and made their destructors always Py_CLEAR(...). In unsafe post‑shutdown paths, leak_detach() zeroes the pointers so the
     destructor won’t touch the C‑API.`311:353:/wrk/forked/pybind11/include/pybind11/detail/internals.h

  void leak_detach() noexcept {
      // Used when internals must be destroyed after interpreter teardown.
      // Avoid touching the Python C-API by clearing pointers only.
      instance_base = nullptr;
      default_metaclass = nullptr;
      static_property_type = nullptr;
  }
  ~internals() {
      // Destruction is expected to run while the interpreter is still intact
      // (e.g., during state-dict teardown). If we must destroy after shutdown,
      // leak_detach() must have been called first.
      Py_CLEAR(instance_base);
      Py_CLEAR(default_metaclass);
      Py_CLEAR(static_property_type);
  }
  ...
  void leak_detach() noexcept {
      // Used when local internals must be destroyed after interpreter teardown.
      function_record_py_type = nullptr;
  }

     1 │- `internals_pp_manager::destroy()` now calls `leak_detach()` before deleting the pointer‑to‑pointer.```711:730:/wrk/forked/pybind11/include/pybind11/detail/internals.h
     2 │void destroy() {
     3 │#ifdef PYBIND11_HAS_SUBINTERPRETER_SUPPORT
     4 │    if (has_seen_non_main_interpreter()) {
     5 │        auto *tstate = get_thread_state_unchecked();
     6 │        // this could be called without an active interpreter, just use what was cached
     7 │        if (!tstate || tstate->interp == last_istate_tls()) {
     8 │            auto tpp = internals_p_tls();
     9 │            if (tpp && tpp->get()) {
    10 │                tpp->get()->leak_detach();
    11 │            }
    12 │            delete tpp;
    13 │        }
    14 │        unref();
    15 │        return;
    16 │    }
    17 │#endif
    18 │    if (internals_singleton_pp_ && internals_singleton_pp_->get()) {
    19 │        internals_singleton_pp_->get()->leak_detach();
    20 │    }
    21 │    delete internals_singleton_pp_;
    22 │    unref();
    23 │}

  Best achievable trade‑off
  • This avoids UB in the post‑shutdown destroy paths, while still freeing the three PyTypes during normal interpreter teardown.
  • The existing 8‑byte unique_ptr shell leak for non‑pybind11‑owned interpreters remains; eliminating that fully still needs a cross‑module manager registry or an ABI bump (as
    in Joshua’s intrusive‑list idea).

[rwgk]

Closing in favor of #5965