fix: align py::print with Python semantics

[rwgk]

Description

This is a comprehensive cleanup of py::print, prompted by the issue reported in #6012.

Original problem

Windows GUI applications built with /SUBSYSTEM:WINDOWS may have sys.stdout set to None. py::print selected that object as its default stream and then attempted to access its write attribute, so printing could fail in this otherwise valid interpreter state.

A narrowly placed None check addresses that immediate failure, but the surrounding implementation had several related differences from Python's print behavior. In particular, an explicit file=None was not treated like an omitted file, stream selection happened after argument formatting, unknown keyword arguments were silently ignored, and output was assembled and sent through a cached write method instead of following Python's incremental file protocol.

Resulting behavior

This change makes py::print follow Python's relevant semantics more closely:

• Omitting file, or passing file=None, resolves the current sys.stdout.
• If the resolved sys.stdout is None, printing is a no-op. This decision is made before converting values with str, matching Python and avoiding side effects during shutdown or in GUI applications without a console stream.
• If sys.stdout is missing during normal interpreter operation, py::print raises RuntimeError("lost sys.stdout") instead of silently discarding output.
• Explicit custom streams remain valid even when they are falsey.
• sep=None and end=None select their defaults, while invalid sep and end values are rejected before any output is written.
• Only sep, end, file, and flush are accepted as keyword arguments. Unknown keywords, including names containing embedded NULs or unpaired surrogates, produce a safe and informative error.
• flush is truth-tested after successful output. This deliberately preserves py::print's historical portable ordering rather than interpreter-specific argument-parsing details; truth-testing and flushing errors propagate normally.
• Output is written incrementally through the public PyFile_WriteObject and PyFile_WriteString APIs. This preserves consequential incremental stream behavior, including repeated write lookup, partial output when a later conversion or write fails, and correct zero-argument behavior.
• Exceptions from value conversion, stream writes, truth testing, and flushing propagate without being replaced.

The implementation deliberately does not delegate to builtins.print: doing so would make py::print sensitive to monkey-patching (and possible recursion) and would complicate its shutdown behavior.

Interpreter lifetime and free-threading

The default-stream lookup distinguishes ordinary execution from interpreter teardown. During normal execution a missing sys.stdout remains an error; during finalization, printing safely becomes a no-op when the interpreter's sys state is no longer available.

CPython's built-in print obtains a strong reference directly from the current interpreter's sys dictionary through the private _PySys_GetRequiredAttr(), which is unavailable to extension modules. On conventional GIL builds, pybind11 can safely promote the borrowed result of PySys_GetObject() while retaining the canonical lookup. On free-threaded builds, concurrent replacement makes that unsafe, so pybind11 obtains a strong reference through the imported sys module instead.

Because the free-threaded fallback necessarily consults sys.modules, the regression that replaces sys.modules["sys"] is skipped on those builds; this records the narrow limitation without complicating production code.

Regression coverage exercises shutdown callbacks in both the main interpreter and subinterpreters. A separate sys.stdout is None regression verifies that values are not converted with str once no output is possible.

Test coverage

The expanded tests exercise py::print's portable contract across normal output and failure paths, including:

• omitted, None, missing, falsey, and custom streams;
• sep and end defaults and validation, plus file and flush handling;
• unknown keyword names, including embedded-NUL and unpaired-surrogate names;
• one and multiple positional arguments;
• failures in __str__, write, flush truth testing, and flush itself;
• partial-output behavior and operation ordering;
• main-interpreter and subinterpreter shutdown.

No ABI-visible data structures or layouts are changed.

Validation completed locally:

• GCC / CPython 3.14 GIL build: the full Catch2 and pytest suites passed.
• GCC / CPython 3.14 free-threaded build: the full Catch2 and pytest suites passed.
• pre-commit run --all-files passed.

This supersedes the implementation proposed in #6012 while preserving its intended behavior when sys.stdout is None.

Suggested changelog entry:

• Fix py::print to more closely follow Python's stream and keyword semantics, propagate stream errors, and safely handle sys.stdout is None, including during interpreter shutdown.

---

📚 Documentation preview 📚: https://pybind11--6120.org.readthedocs.build/

---

📚 Documentation preview 📚: https://pybind11--6120.org.readthedocs.build/

[henryiii]

TL;DR: No zero args test, REQUIRE(state.stdout_was_none) is an interpreter detail -- probably fine, but probably avoidable if you want to avoid it.

I'm also running a simplification review, I'll push that and post it next.

🤖 AI text below 🤖

Verified: kwargs keys reaching detail::print are always str — Python-side calls enforce string keywords, and the C++ collector converts names with str() (cast.h:2291, cast.h:2135 area). The dict_getitemstringref helper throws on real errors and returns nullptr only for a missing key, so the "lost sys.stdout" logic is clean. Here's the review.

Review of PR #6120 — fix: align py::print with Python semantics

Overview

Rewrites detail::print (include/pybind11/pybind11.h:3766) to follow CPython's builtin_print semantics: kwargs are parsed once with unknown names rejected, file=None resolves sys.stdout, a None stdout makes printing a no-op before any str() conversion (the #6012 fix), a missing stdout raises RuntimeError("lost sys.stdout") outside shutdown, sep/end are validated up front, and output goes incrementally through PyFile_WriteObject/PyFile_WriteString instead of join-then-write. Free-threaded builds get a separate strong-reference stdout lookup. Extensive pytest coverage plus embedded-interpreter and subinterpreter shutdown tests.

Correctness — verified sound

• Ordering matches CPython: kwarg rejection → file resolution → early return on None stdout → sep/end type checks → writes → flush truth-test after output. I checked this against builtin_print; the early return before sep/end validation is what CPython does too.
• PyUnicode_CompareWithASCIIString on kwargs keys is safe: both entry paths guarantee str keys (CPython enforces string keywords; the C++ **dict collector converts keys via str()). Embedded-NUL keys compare correctly since the compare is length-aware — and the test covers it.
• Reference handling is correct: dict_getitemstringref returns a strong ref or throws on real errors (pytypes.h:991); the GIL-build borrow-then-promote of PySys_GetObject is safe as commented; PyModule_GetDict failure is handled.
• Error paths: every C-API result is checked; exceptions propagate without replacement, confirmed by the pybind_print_trace protocol tests.

Behavior changes worth flagging

• Unknown kwargs now raise TypeError where they were silently ignored. This is the right fix but is user-visible; the changelog entry covers it only implicitly ("keyword semantics"). Consider naming it explicitly in the changelog line since previously-working (buggy) callers will now throw.
• Output is no longer atomic-ish: the old code did two write calls; the new code does one per value/separator. Under threading, py::print output can interleave more — same as Python's print, so acceptable, but it's a change in observable behavior.

Minor issues / suggestions

• Zero-argument coverage gap: the PR description claims coverage of "correct zero-argument behavior," but I don't see a m.print_args() (no positionals) test in the diff. A one-liner asserting only "\n" is written (single write call, not the old write("") + write("\n")) would back the claim.
• Shutdown tests assert an interpreter implementation detail: REQUIRE(state.stdout_was_none) in both Catch2 tests checks when CPython nulls out sys.stdout during finalization, not pybind11 behavior. If any supported interpreter/version runs the capsule destructor before stdout is cleared, the test fails even though py::print behaved correctly (it would take the also-safe RuntimeError→caught path... actually no, it would raise and print_threw would trip too). Since the property you actually care about is "does not crash and does not throw", consider dropping or softening the stdout_was_none assertion, or at least be prepared for matrix surprises (PyPy/GraalPy embed variants, older CPython).
• Nit: in the FT path the local object sys; shadows the conceptual module name fine, but sys_module would read slightly clearer next to sys_dict.
• Nit: detail::print is technically callable directly with a hand-built dict containing non-str keys (UB in PyUnicode_CompareWithASCIIString / %U). It's a detail:: internal so I wouldn't add a check, but a one-line comment stating the str-key precondition would be cheap insurance.

Conventions, tests, performance, security

• Conventions: matches project style — public C API only, no ABI-visible changes, comments explain non-obvious constraints (the GIL/FT split, the flush-ordering decision), docs updated. The FT test skip uses env.PY_GIL_DISABLED, and env is already imported in test_pytypes.py — verified.
• Tests: excellent breadth — the event-trace protocol test (pybind_print_trace) pinning partial-output and failure ordering is particularly good, and the PyPy-Windows surrogate caveat is handled with a portable assertion.
• Performance: more Python-level calls per print than before (per-value writes, per-key compares), but py::print is not hot-path and this mirrors CPython itself. PYBIND11_NOINLINE keeps the code-size impact to one instantiation.
• Security: improves robustness — no cached write bound method, validated kwarg names with %U (safe for NUL/surrogates on CPython), %.200s truncation on type names matches CPython practice.

Verdict

High-quality change; the implementation is correct and carefully matched against CPython's actual builtin_print behavior. The only substantive asks are the missing zero-arg test and a second look at how strict the stdout_was_none shutdown assertions should be across the CI matrix — everything else is nits.

[henryiii]

TL;DR: 31 lines shorter pybind11.h, 43 lines shorter overall, 4KB smaller, uses our set_error instead of raw C-API.

🤖 AI text below 🤖

All 14 print tests pass. The cleanup is complete — here's the summary.

Applied

include/pybind11/pybind11.h — detail::print body (net −31 lines):

• Replaced the two copy-paste sep/end validation blocks with one as_string_or_default lambda that also normalizes an unset/None value to its default str up front. This removes the four scattered x && !x.is_none() re-derivations (including the one re-evaluated every loop iteration) and the PyFile_WriteString fallback ternaries — PyFile_WriteString is literally PyUnicode_FromString + PyFile_WriteObject, so behavior is identical, and CPython interns 1-char strings so the up-front str(" ") is free.
• Collapsed the five separate throw error_already_set() write checks into one write lambda; the output section is now if (!first) write(sep); write(arg); ... write(end);.
• Switched the argument loop to for (handle arg : args), which uses pybind11's fast tuple iterator (PyTuple_GET_ITEM) — eliminating the per-item accessor object, its incref/decref churn, and the per-iteration PyTuple_Size call.
• Replaced the 8-line manual PyObject_IsTrue flush block with if (flush && bool_(flush)) — I verified bool_'s converting constructor runs PyObject_IsTrue and throws error_already_set on failure, and its operator bool tests the value (pytypes.h:1885), so semantics are unchanged.
• Used the existing set_error() wrapper at the two "lost sys.stdout" raise sites.

Tests:

• Extracted the byte-for-byte duplicated ~26-line shutdown-capsule harness from the two Catch2 files into a new shared tests/test_with_catch/print_shutdown_probe.h (both files compile into the same test_with_catch binary; catch_skip.h is the precedent). Each TEST_CASE is now ~6 lines.
• Consolidated the three marker-exception classes in test_pytypes.py into the one module-level PrintMarkerError.

Skipped (with reasons)

• Altitude: a general detail::sys_getattr helper in pytypes.h (using PySys_GetOptionalAttrString on 3.14+, unifying GIL/free-threaded semantics): a real design idea, but it changes user-visible behavior across builds and reaches well outside this diff — it belongs as discussion on the PR, not a silent cleanup.
• Dropping the PyModule_GetDict null check — false positive: on free-threaded builds sys.modules["sys"] can hold a non-module object, module_::import returns it, and PyModule_GetDict then returns null with SystemError. The check is load-bearing.
• Replacing the strong-ref dict lookup with py::getattr(sys, "stdout", handle()) — the default-taking getattr overload swallows non-AttributeError failures, and the dict lookup is the PR's documented deliberate mechanism.
• type_error + obj_class_name() for the sep/end messages — would lose the %.200s truncation that matches CPython; the duplication it targeted is gone via the lambda anyway.

Verification

prek is clean, the test module builds with PYBIND11_WERROR=ON (and shrank ~4 KB), and all 121 test_pytypes.py tests pass including the 14 print tests. The Catch2 shutdown tests can't be run locally — your build/ config skips the embed tests ("no embed libs" for the uv Python) — so I compile-checked both edited translation units instead; CI exercises them. Net effect: −43 lines on the branch with no behavior change.

[rwgk]

Quick update (I can only work on it more tomorrow in my timezone):

In the meantime I had this idea:

It just randomly crossed my mind: why are we emulating Python's native print ... why are we not just calling it?

codex:

• Honestly, mostly because py::print has been an emulator since it was introduced in 2016, and we followed that precedent. I found no
  historical discussion showing that delegation was considered and deliberately rejected.

  There is one API wrinkle: CPython does not export a public Py_Print() function. Its actual implementation, builtin_print_impl, is internal
  and static. CPython source (https://github.com/python/cpython/blob/main/Python/bltinmodule.c#L2163-L2259)

  We can nevertheless retrieve the current builtins.print callable and invoke it with PyObject_Call, which is precisely equivalent to
  callable(*args, **kwargs). Python C API (https://docs.python.org/3/c-api/call.html#c.PyObject_Call)

  That is very attractive. It would eliminate almost all of include/pybind11/pybind11.h:3766, including:

  - keyword parsing and diagnostics;
  - sep, end, file, and flush handling;
  - the GIL/free-threaded sys.stdout split;
  - stream-operation ordering;
  - interpreter-specific differences;
  - the Windows PyPy %U problem;
  - the risk of future Python behavior drifting away from our replica.

  The real tradeoffs are:

  - builtins.print is mutable. After delegation, monkey-patching it would affect py::print. There is no public way to request an immutable
    “original print”; safely caching one across subinterpreters and interpreter restarts would reintroduce considerable complexity.

  - A replacement that calls back into py::print could recurse. I now think the PR description overstates this concern—it is normal self-
    recursion caused by such a patch, not an inherent pybind11 hazard.

  - We would reverse one deliberate decision from this PR: py::print currently truth-tests flush after output, preserving its historical
    ordering. Native implementations may truth-test it during argument parsing, before output. Delegation means intentionally accepting each
    interpreter’s behavior.

  - Shutdown still needs verification. Native print correctly handles sys.stdout is None, but we must ensure retrieving builtins.print remains
    safe during the main- and subinterpreter shutdown cases. The existing tests are unusually well suited to proving this.

  - Obtaining the callable requires a strong-reference lookup. PyEval_GetBuiltins() provides the current frame/interpreter builtins, although
    Python 3.13 recommends the newer strong-reference PyEval_GetFrameBuiltins(). Python reflection API
    (https://docs.python.org/3/c-api/reflection.html#c.PyEval_GetBuiltins)

  My present view: we should seriously explore delegation. The weeds we have encountered—especially free-threaded stdout access and the PyPy
  formatting discrepancy—are concrete evidence that maintaining a replica has substantial ongoing cost.

  I would make the semantic choice explicit: “py::print calls the current Python print callable and therefore follows that interpreter’s
  behavior, including monkey-patching.” If that is acceptable, delegation looks like the cleaner and more permanent solution. No changes made
  yet.

[henryiii]

Did a quick check for performance and potential downsides:

(and PyEval_GetFrameBuiltins is the same, and we need that one for free-threading).

🤖 AI text below 🤖

Both questions answered — here are the results.

Performance

Benchmarked on CPython 3.14 (macOS, -O2, output to StringIO/redirected stdout so I/O cost doesn't drown the comparison):

Scenario	custom detail::print	builtins.print via import	via PyEval_GetBuiltins
file=StringIO	269 ns	433 ns	293 ns
default stdout	246 ns	369 ns	233 ns
sep/end kwargs	351 ns	489 ns	—
end-to-end py::print("Hello", 42, "world")	331 ns	427 ns	—

So the custom code is not buying meaningful speed. The naive delegation (module_::import("builtins").attr("print") each call) costs ~100–160 ns extra per call, all of it in the module/attr lookup. Fetching print from PyEval_GetBuiltins() (borrowed, per-interpreter-correct, so no caching problem with subinterpreters) is at parity with the custom code. And any real print that touches an actual stream is microseconds, so even the naive version disappears in practice.

Behavioral differences (the actual downsides)

I diffed the edge cases directly; most things match exactly — stream protocol call sequence (write per item/sep/end, then flush), sep/end type errors, sep=None/end=None, sys.stdout = None (no-op), deleted sys.stdout (RuntimeError: lost sys.stdout), and the sys-module-unavailable path (native print also reads the interpreter's sys directly). The differences:

1. flush evaluation ordering. CPython now converts flush to bool during argument parsing, before writing anything; py::print historically evaluates it after successful output — exactly what the comment at include/pybind11/pybind11.h:3872 and test_print_flush_truthiness_error_after_output pin down. With delegation, a flush whose __bool__ raises produces the error with nothing written. Almost certainly nobody depends on this, but it's a documented-by-test behavior change.

2. Error message drift. CPython 3.14 says print() got an unexpected keyword argument 'unknown', while pybind11 emits the older-style 'unknown' is an invalid keyword argument for print(). Messages would also become PyPy/GraalPy-native on those interpreters. The exact-match tests (test_print_rejects_unknown_keyword etc.) would need loosening — arguably that's a feature, since users get their interpreter's real messages.

3. Monkeypatched builtins.print gets honored. mock.patch("builtins.print") currently doesn't intercept py::print (it writes to sys.stdout directly); after delegation it would. Probably desirable, but it's observable.

4. Shutdown behavior needs re-verifying. The current code silently no-ops when sys can't be imported during finalization; a delegating version must handle builtins lookup failing the same way. Your shutdown harness from this branch can pin that.

Bottom line

There's no performance argument for keeping the emulation — with a PyEval_GetBuiltins() lookup it's a wash, and even the naive import costs ~150 ns on a sub-µs operation. The trade is really: delete ~100 lines of subtle code (including the whole Py_GIL_DISABLED stdout-lookup branch, since native print handles that internally) and get exactly-native behavior on every interpreter forever, in exchange for two small compat breaks (flush ordering, error-message wording) and loosening a handful of the tests this branch just added. One caveat on PyEval_GetBuiltins: it's soft-deprecated as of 3.13 in favor of PyEval_GetFrameBuiltins, though it still works and remains the only spelling on older Pythons.

Benchmark and diff scripts are in the scratchpad (bench_print.cpp, bench.py) if you want to rerun on free-threaded or PyPy builds.

[Skylion007]

Why use PyObject instead of handles or py::object here?

[Skylion007]

Hold on can this it just be bound as py::print? Or does the overloading system not like that?

[rwgk]

I pointed codex to this question. I then added this to the PR #6121 Description:

The lambda is intentional: py::print is a variadic function template rather than a single function that m.def can bind directly, and a specialization taking py::args and py::kwargs would pass those containers as ordinary arguments instead of unpacking them.

[rwgk]

Converted to Draft, in favor of PR #6121.

(I will close this PR after 6121 is merged.)