Skip to content

fix(extract): truncate comments/docstrings at UTF-8 character boundaries#1107

Closed
DeusData wants to merge 1 commit into
mainfrom
fix/1017-utf8-docstring
Closed

fix(extract): truncate comments/docstrings at UTF-8 character boundaries#1107
DeusData wants to merge 1 commit into
mainfrom
fix/1017-utf8-docstring

Conversation

@DeusData

Copy link
Copy Markdown
Owner

Closes #1017

Bug

extract_comment_text (extract_defs.c) capped over-long comments/docstrings with a raw byte cut — text[MAX_COMMENT_LEN] = '\\0' — which splits a multi-byte UTF-8 character straddling byte 500 and leaves an incomplete sequence. That value lands verbatim in nodes.properties: syntactically valid JSON but invalid UTF-8, so any consumer decoding the SQLite TEXT column strictly (Python's sqlite3, the default for anyone building tooling on the exported DB) throws on read. Never surfaced on English-only code (single-byte chars cut safely anywhere); reproduces on Japanese/Chinese/etc. comments — reporter hit it on EC-CUBE.

Fix

Snap the cut back over UTF-8 continuation bytes (10xxxxxx) to the lead byte of the straddling character and terminate there, dropping the partial character — result is always valid UTF-8. The reporter also noted the append_json_string escape loop as a possible defense-in-depth site; I confirmed the primary corruption is here (the atomic-precheck prevents the escape loop from firing on a value that passed it), so this is the single root-cause fix.

Reproduce-first

python_docstring_utf8_boundary_truncation_issue1017 builds a docstring with 3-byte CJK chars (E3 81 82 = U+3042 あ) straddling the 500-byte cap and asserts the captured docstring decodes as valid UTF-8 via an inline validator. RED before (mid-char split → invalid), GREEN after. End-to-end verified: the stored nodes.properties for the fixture goes from Python-sqlite3-unreadable to valid UTF-8.

Verification

Full local suite 6334 passed, 1 skipped; lint-ci clean.

extract_comment_text capped over-long comments with a raw byte cut
(text[MAX_COMMENT_LEN] = 0), which splits a multi-byte UTF-8 character
straddling byte 500 and leaves an incomplete sequence. That corrupted
value lands verbatim in nodes.properties — syntactically valid JSON but
invalid UTF-8 — so any consumer decoding the SQLite TEXT column strictly
(Python's sqlite3, the default for tooling on the exported DB) throws on
read. It never surfaced on English-only code because a single-byte-per-
char comment can be cut anywhere safely; it reproduces on Japanese/
Chinese/etc. comments (reporter: EC-CUBE).

Snap the cut back over UTF-8 continuation bytes (10xxxxxx) to the lead
byte of the straddling character and terminate there, dropping the
partial character. Result is always valid UTF-8.

Reproduce-first: python_docstring_utf8_boundary_truncation_issue1017
builds a docstring with 3-byte CJK chars (E3 81 82) straddling the cap
and asserts the captured docstring decodes as valid UTF-8 (via an
inline validator). RED before (mid-char split), GREEN after.

Closes #1017

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData DeusData added this to the 0.9.1-rc milestone Jul 15, 2026
@DeusData DeusData enabled auto-merge July 15, 2026 22:53
@DeusData

Copy link
Copy Markdown
Owner Author

Superseded by #1094, which landed the same UTF-8 codepoint-boundary snap-back for docstring/comment truncation first — thank you @harshitaajoshi for the clean fix and the thorough boundary test. Closing this in favour of yours (merged as e2d41d9).

@DeusData DeusData closed this Jul 16, 2026
auto-merge was automatically disabled July 16, 2026 00:16

Pull request was closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docstring truncation splits multi-byte UTF-8 characters, producing invalid UTF-8 in nodes.properties (non-ASCII comments, e.g. Japanese)

1 participant