Guard against f_lasti == -1 when detecting exception returns#266
Conversation
The tracer reads co_code[frame.f_lasti] to decide whether a 'return' event was caused by an exception. f_lasti can be -1 (e.g. a frame that hasn't executed an instruction), and co_code[-1] reads the last byte -- the wrong opcode -- which can misreport a normal return as "Call ended by exception". Extract the check into call_ended_by_exception() and return False when f_lasti < 0.
cool-RR
left a comment
There was a problem hiding this comment.
Thanks @koriyoshi2041, good catch and a clean fix. I went and verified it carefully and it holds up.
The bug is real on CPython 3.8 to 3.10 (and latent on 3.11+).
The repro in #260 is mock-only, so I checked whether a real return event ever actually carries f_lasti == -1. Across normal returns, exceptions, and a pile of generator scenarios:
- 3.11 / 3.12 / 3.13: never happens. Purely latent.
- 3.8 / 3.9 / 3.10: it does. Closing (or
throw()-ing into) an unstarted generator fires areturnevent whose frame hasf_lasti == -1, which is exactly the case whereco_code[-1]reads an unrelated byte.
And it surfaces through our own depth>=2 path. On 3.8, master vs this PR for a plain g = gen(5); g.close():
master this PR
exception 12 def gen(n): exception 12 def gen(n):
Call ended by exception --> return 12 def gen(n):
Return value:.. None
master's "exception" verdict there is just luck of the last bytecode byte, not detection, so I'm glad to see it go.
Verification
- Full suite passes on 3.8, 3.12, 3.13 (plus the 4 new unit tests).
- Behavior is identical to master except the
f_lasti < 0case, which is the fix. It also can't hide a genuine error: with nothing executed in the frame the only way out is an injected exception, which still gets its ownexceptionevent. - Bonus: moving the opcode read behind the
event == 'return'check skips it on every line event, which is the hot path.
How others handle it, for reference
python-hunterguards the same way before indexing:len(co_code) > f_lasti >= 0. Same instinct as this PR.snoophas the identical unguardedco_code[f_lasti]and nobody's hit it, which matches the "latent on modern CPython" finding.- stdlib
bdb/pdbsidestep it entirely by routing on the event string and never reading opcodes.
Two small things
- The new tests are unit-only (mocked frames). Fair, since the bug isn't reachable on CI Python, but I'd like a real integration test alongside them (below). It passes on all versions and would have failed on master for 3.8 to 3.10.
tests/test_ended_by_exception.pyimportsunittest.mock, which is Py3-only, whilesetup.pystill lists 2.7. That 2.7 classifier is vestigial and I'll drop it; just flagging so the test file isn't what trips over it.
Suggested integration test (verified: passes on this PR for 3.8/3.10/3.12/3.13, fails on master for 3.8/3.10):
def test_unstarted_generator_close_is_not_an_exception():
# Regression for #260. On CPython 3.8-3.10, closing an unstarted generator
# fires a 'return' event whose frame has f_lasti == -1; the old code read
# co_code[-1] (an unrelated opcode) and mislabeled this normal close as
# "Call ended by exception". Passes on all versions; only catches the bug
# on 3.8-3.10, where it is reachable.
string_io = io.StringIO()
def gen(n):
for i in range(n):
yield i
@pysnooper.snoop(string_io, depth=2, color=False)
def main():
g = gen(5)
g.close()
return 42
assert main() == 42
output = string_io.getvalue()
assert 'Call ended by exception' not in output
assert 'Return value:.. 42' in output(needs import io and import pysnooper at the top of the file.)
Add that and I'm happy to merge. Thanks again!
|
Added the integration regression test you suggested and pushed it as Checked locally with: |
|
Good job, thank you. |
Fixes #260.
To distinguish a normal
returnevent from one caused by an exception (both arrive asevent == 'return'witharg is None), the tracer inspects the last executed opcode:frame.f_lastican be-1(for example a frame that hasn't executed an instruction yet).co_code[-1]then reads the last byte of the bytecode — an unrelated opcode — and the result can flipended_by_exceptiontoTrue, so a normal return is printed asCall ended by exception.This extracts the check into
call_ended_by_exception(frame, event, arg)and returnsFalsewhenf_lasti < 0, so the wrong opcode is never read.Added
tests/test_ended_by_exception.pywith mock frames:f_lasti == -1is treated as a normal return; a return opcode → normal return; a non-return opcode → ended by exception; non-returnevents are never flagged. Thef_lasti == -1case fails without the guard and passes with it; existing tracer tests still pass.