gh-153419: Fix several issues around bytearray __init__ by stestagg · Pull Request #153498 · python/cpython

stestagg · 2026-07-10T10:04:36Z

Introduce a bytearray_new function to ensure that
ob_bytes_object is always set on a bytearray.

Resizing a bytearray to 0 length explicitly sets
the ob_bytes_object to the empty constant immortal.

Add a check in the 'bytearray init from string'
fast path to ensure there are no active exports.

This fixes asserts/crashes on the following:

bytearray(1).__init__()
bytearray().__new__(bytearray).append(1)
a = bytearray(); b = memoryview(a); a.__init__('x', 'ascii')

I realise that the explicit setting of bytes_object to Py_CONSTANT_EMPTY_BYTES is not strictly necessary, as this is also done by the _PyBytes_Resize too, but this seemed to mark the intent of what's happening more clearly, given we rely and assert on that exact behaviour.

Issue: debug abort/ref leak calling __init__(<str>, <enc>) on a one-byte bytearray #153419

Introduce a bytearray_new function to ensure that ob_bytes_object is always set on a bytearray. Resizing a bytearray to 0 length explicitly sets the ob_bytes_object to the empty constant immortal. Add a check in the 'bytearray init from string' fast path to ensure there are no active exports. This fixes asserts/crashes on the following: - bytearray(1).__init__() - bytearray().__new__(bytearray).append(1) - a = bytearray(); b = memoryview(a); a.__init__('x', 'ascii')

This reverts commit c961d61.

There are several ways that bytearray_resize_lock_held gets called and for any of them, ob_bytes_object may be NULL. This allows us to remove the extra initialization in __init__ as this can't be guaranteed to be called anyway. Handle a related issue in take_bytes where in the extreme case of the bytes resize allocation failing (on a size reduction) then the bytearray other state fileds could be left inconsistent with a null ob_bytes_object. unconditionally call _canresize in __init__ as on initial creation, this will always be true, and if there are any exported views, then we just always fail, even if technically an empty bytearray doesn't get modified in this case, it's so rare, it's not worth it.

stestagg · 2026-07-11T00:51:41Z

OK, rewritten based on @cmaloney's comments on the issue.

Some of the changes are a bit non-obvious, for example initializing ob_bytes_object at the top of bytearray_resize_lock_held, but not doing so makes the following obj->ob_start - obj->ob_bytes logic technically UB, so I figured better safe than sorry.

As it now intentionally tolerates uninitialized ob_bytes_object field, (And there was another place that set ob_bytes_object to null, so this is reasonable) I've added a bunch of tests to cover the code paths that used to crash individually, even though the actual fix is in the resize method, this seemed more robust going forward.

We have to check _canresize explicitly in init as the bytearray_resize_lock_held call doesn't always do this (if the size doesn't change).

There may be some other __init__ free-threading issues still lurking, but if there are, that's out of scope for this change really.

Format blurb correctly, and reduce the chatter (There is a user-visible behaviour change, but I guess it's so minor it's not worth mentioning)

stestagg · 2026-07-17T10:32:36Z

@cmaloney: are you able to take another look? Sorry for bugging you!

cmaloney · 2026-07-18T23:05:12Z

+    /* If ob_bytes_object has not been initialized yet, eagerly initialize
+       it here so the following code can reason about state more easily,
+       and things like pointer comparisons are valid. */
+    if (obj->ob_bytes_object == NULL) {


Reading through https://docs.python.org/3/extending/newtypes_tutorial.html a bit it feels like should make it so tp_new does the "ob_bytes_object" is always NULL, then other places either get an allocation or assign to the empty bytes.

I think that will reduce the amount of code a bit and make it easier to get right all the corner cases.

… not NULL

cmaloney

Generally looking good, like the shape. Some comments and suggestions.

cmaloney · 2026-07-19T23:53:23Z

@@ -0,0 +1 @@
+Fix several :class:`bytearray` crashes caused by calling :meth:`~object.__init__`/:meth:`~object.__new__` in unusual ways.


Suggested change

Fix several :class:`bytearray` crashes caused by calling :meth:`~object.__init__`/:meth:`~object.__new__` in unusual ways.

Fix multiple :class:`bytearray` crashes and reference leaks caused by skipping :meth:`~object.__init__` and broken state setup code.

cmaloney · 2026-07-20T05:34:20Z

        self.assertRaises(BufferError, ba.hex, S(b':'))

+    def test_uninitialized_instance(self):
+        # A bytearray created with __new__ so that __init__ is never called


This is fairly verbose at the moment, now that there is a new I think can condense a bit

stestagg · 2026-07-20T11:26:34Z

@cmaloney I've updated the PR with the review comments, the ob_exports is technically a no-op because PyType_GenericNew calls PyType_GenericAlloc which zeroes the fields, but it's trivial safe and simple to add, so I did.

Thanks!

cmaloney

Two small comment nits, think this is ready for other core review.

cmaloney · 2026-07-21T05:11:44Z

-        bytearray_reinit_from_bytes(self, 0, 0);
-        self->ob_exports = 0;
+    /* Disallow any __init__ call if the object is not resizable (has exports)
+       to make the handling of non-null `source` init values simpler. */


I don't think this quite matches the current state? The non-null is always handled in the tp_new; the resize is just preventing a reallocation which would leave the existing exports and pointers into unowned memory

This was probably a badly worded comment, as per your previous, the _canresize() isn't strictly needed here, because PyByteArray_Resize checks this before doing an actual resize.

The benefit of putting it here is that the processing of the first argument (source in the docs) is a bit easier if we eagerly check _canresize(). The non-null in the comment is referring to that (because when source is null, we don't do anything more than resize()).

I'm open to suggestions here :)

Fix capitalization in comment Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>

vstinner

Overall, the change looks good to me. Here is a first review :-)

vstinner · 2026-07-24T09:56:37Z

+    def test_no_init_called(self):
+        # A bytearray created without calling bytearray.__init__
+        # should not crash the interpreter (see gh-153419).
+        def uninitialized():


Hum, "uninitialized" name is misleading. Maybe before the fix, the object could be "uninitialized". But with this fix, the object is now well defined. I suggest renaming the function to "bytearray_new()".

vstinner · 2026-07-24T10:03:36Z

    _Py_CRITICAL_SECTION_ASSERT_OBJECT_LOCKED(self);
    PyByteArrayObject *obj = ((PyByteArrayObject *)self);
+
+    assert(obj->ob_bytes_object != NULL);


Would you mind to apply this sanity check to _PyBytes_Resize()? The empty bytes string singleton must never be resized.

diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c index f63185e1428..264ce0d9a10 100644 --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -3381,6 +3381,7 @@ _PyBytes_Resize(PyObject **pv, Py_ssize_t newsize) return (*pv == NULL) ? -1 : 0; } + assert(v != bytes_get_empty()); #ifdef Py_TRACE_REFS _Py_ForgetReference(v); #endif

vstinner · 2026-07-24T10:05:51Z

+    if (self == NULL) {
+        return NULL;
+    }
+    PyByteArrayObject *obj = _PyByteArray_CAST(self);


I would prefer to rename self to op and rename obj to self.

vstinner · 2026-07-24T10:07:15Z

+    if (PyByteArray_Resize((PyObject *)self, 0) < 0) {
+        return -1;
    }



nitpick: you can remove this empty line, since these 2 assertions are related to PyByteArray_Resize() :-)

vstinner · 2026-07-24T10:10:14Z

    }

    if (_PyBytes_Resize(&self->ob_bytes_object, to_take) == -1) {
+        self->ob_bytes_object = Py_GetConstant(Py_CONSTANT_EMPTY_BYTES);


I'm not 100% sure that _PyBytes_Resize() always set self->ob_bytes_object to NULL on error. Also, this private function can change in the future. So I would prefer to add this assertion to make sure that there is no refleak:

Suggested change

self->ob_bytes_object = Py_GetConstant(Py_CONSTANT_EMPTY_BYTES);

assert(self->ob_bytes_object == NULL);

self->ob_bytes_object = Py_GetConstant(Py_CONSTANT_EMPTY_BYTES);

- Add assert to bytes object resize (ensure we're not resizing the empty constant) - Test helper rename - Flip obj<->self usage in bytearray_new - Assert PyBytes_Resize failure sets ob_bytes_object to NULL

…vstinner's suggestion.

vstinner

LGTM. Thanks for the updates.

vstinner · 2026-07-24T15:12:20Z

It would be interesting to backport these changes to 3.13 and 3.14, but I expect that it should be done manually to solve merge conflicts.

miss-islington-app · 2026-07-24T15:25:16Z

Thanks @stestagg for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

bedevere-app · 2026-07-24T15:25:29Z

GH-154622 is a backport of this pull request to the 3.15 branch.

stestagg · 2026-07-24T15:26:02Z

LGTM. Thanks for the updates.

@vstinner - Thanks for the review, and finding the issues!

…3498) (#154622) gh-153419: Fix several issues around bytearray __init__ (GH-153498) Introduce a bytearray_new() function to ensure that ob_bytes_object is always set on a bytearray. Resizing a bytearray to 0 length now explicitly sets the ob_bytes_object to the empty constant immortal. Add a check in the 'bytearray init from string' fast path to ensure there are no active exports. This fixes asserts/crashes on the following: - bytearray(1).__init__() - bytearray().__new__(bytearray).append(1) - a = bytearray(); b = memoryview(a); a.__init__('x', 'ascii') (cherry picked from commit d5c1b29) Co-authored-by: Steve Stagg <stestagg@gmail.com> Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>

bedevere-app Bot mentioned this pull request Jul 10, 2026

debug abort/ref leak calling __init__(<str>, <enc>) on a one-byte bytearray #153419

Open

bedevere-app Bot added the awaiting review label Jul 10, 2026

stestagg added 3 commits July 11, 2026 01:33

Revert initial version

b54ca0f

This reverts commit c961d61.

Add blurb

98888e0

Comments/blurb spelling and grammar

f5ea967

cmaloney reviewed Jul 11, 2026

View reviewed changes

Comment thread Lib/test/test_bytes.py Outdated

Comment thread Lib/test/test_bytes.py Outdated

Comment thread Misc/NEWS.d/next/Core_and_Builtins/2026-07-11-01-17-57.gh-issue-153419.U8HJOJ.rst Outdated

Comment thread Objects/bytearrayobject.c

StanFromIreland reviewed Jul 11, 2026

View reviewed changes

Comment thread Lib/test/test_bytes.py Outdated

Update tests to just run inline, and fold many of the tests into one.

c57fe6b

Format blurb correctly, and reduce the chatter (There is a user-visible behaviour change, but I guess it's so minor it's not worth mentioning)

cmaloney reviewed Jul 18, 2026

View reviewed changes

stestagg added 3 commits July 19, 2026 23:18

Revert to tp_new approach

b03da26

Tidy up some comments

689a52d

Reset to Py_SETREF on resize path now we guarantee ob_bytes_object is…

9ee14d4

… not NULL

cmaloney reviewed Jul 20, 2026

View reviewed changes

Review comments

6f73d57

cmaloney approved these changes Jul 21, 2026

View reviewed changes

bedevere-app Bot added awaiting core review and removed awaiting review labels Jul 21, 2026

Update Objects/bytearrayobject.c

0d64e2b

Fix capitalization in comment Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>

vstinner reviewed Jul 24, 2026

View reviewed changes

stestagg added 2 commits July 24, 2026 15:27

Review feedback:

ecd5941

- Add assert to bytes object resize (ensure we're not resizing the empty constant) - Test helper rename - Flip obj<->self usage in bytearray_new - Assert PyBytes_Resize failure sets ob_bytes_object to NULL

Fix some sloppy errors, using correct '->' operator, and mis-reading …

072bc04

…vstinner's suggestion.

vstinner approved these changes Jul 24, 2026

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Jul 24, 2026

vstinner enabled auto-merge (squash) July 24, 2026 15:11

vstinner added the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Jul 24, 2026

vstinner merged commit d5c1b29 into python:main Jul 24, 2026
59 checks passed

bedevere-app Bot removed the awaiting merge label Jul 24, 2026

bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Jul 24, 2026

		@@ -0,0 +1 @@
		Fix several :class:`bytearray` crashes caused by calling :meth:`~object.__init__`/:meth:`~object.__new__` in unusual ways.

	Fix several :class:`bytearray` crashes caused by calling :meth:`~object.__init__`/:meth:`~object.__new__` in unusual ways.
	Fix multiple :class:`bytearray` crashes and reference leaks caused by skipping :meth:`~object.__init__` and broken state setup code.

	self->ob_bytes_object = Py_GetConstant(Py_CONSTANT_EMPTY_BYTES);
	assert(self->ob_bytes_object == NULL);
	self->ob_bytes_object = Py_GetConstant(Py_CONSTANT_EMPTY_BYTES);

Uh oh!

Uh oh!

Conversation

stestagg commented Jul 10, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stestagg commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stestagg commented Jul 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmaloney left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stestagg commented Jul 20, 2026

Uh oh!

cmaloney left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

vstinner commented Jul 24, 2026

Uh oh!

Uh oh!

miss-islington-app Bot commented Jul 24, 2026

Uh oh!

bedevere-app Bot commented Jul 24, 2026

Uh oh!

stestagg commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

stestagg commented Jul 10, 2026 •

edited by bedevere-app Bot

Loading

stestagg commented Jul 11, 2026 •

edited

Loading