fix(extract): truncate comments/docstrings at UTF-8 character boundaries#1107
Closed
DeusData wants to merge 1 commit into
Closed
fix(extract): truncate comments/docstrings at UTF-8 character boundaries#1107DeusData wants to merge 1 commit into
DeusData wants to merge 1 commit into
Conversation
extract_comment_text capped over-long comments with a raw byte cut (text[MAX_COMMENT_LEN] = 0), which splits a multi-byte UTF-8 character straddling byte 500 and leaves an incomplete sequence. That corrupted value lands verbatim in nodes.properties — syntactically valid JSON but invalid UTF-8 — so any consumer decoding the SQLite TEXT column strictly (Python's sqlite3, the default for tooling on the exported DB) throws on read. It never surfaced on English-only code because a single-byte-per- char comment can be cut anywhere safely; it reproduces on Japanese/ Chinese/etc. comments (reporter: EC-CUBE). Snap the cut back over UTF-8 continuation bytes (10xxxxxx) to the lead byte of the straddling character and terminate there, dropping the partial character. Result is always valid UTF-8. Reproduce-first: python_docstring_utf8_boundary_truncation_issue1017 builds a docstring with 3-byte CJK chars (E3 81 82) straddling the cap and asserts the captured docstring decodes as valid UTF-8 (via an inline validator). RED before (mid-char split), GREEN after. Closes #1017 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Owner
Author
|
Superseded by #1094, which landed the same UTF-8 codepoint-boundary snap-back for docstring/comment truncation first — thank you @harshitaajoshi for the clean fix and the thorough boundary test. Closing this in favour of yours (merged as e2d41d9). |
auto-merge was automatically disabled
July 16, 2026 00:16
Pull request was closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1017
Bug
extract_comment_text(extract_defs.c) capped over-long comments/docstrings with a raw byte cut —text[MAX_COMMENT_LEN] = '\\0'— which splits a multi-byte UTF-8 character straddling byte 500 and leaves an incomplete sequence. That value lands verbatim innodes.properties: syntactically valid JSON but invalid UTF-8, so any consumer decoding the SQLite TEXT column strictly (Python'ssqlite3, the default for anyone building tooling on the exported DB) throws on read. Never surfaced on English-only code (single-byte chars cut safely anywhere); reproduces on Japanese/Chinese/etc. comments — reporter hit it on EC-CUBE.Fix
Snap the cut back over UTF-8 continuation bytes (
10xxxxxx) to the lead byte of the straddling character and terminate there, dropping the partial character — result is always valid UTF-8. The reporter also noted theappend_json_stringescape loop as a possible defense-in-depth site; I confirmed the primary corruption is here (the atomic-precheck prevents the escape loop from firing on a value that passed it), so this is the single root-cause fix.Reproduce-first
python_docstring_utf8_boundary_truncation_issue1017builds a docstring with 3-byte CJK chars (E3 81 82= U+3042 あ) straddling the 500-byte cap and asserts the captured docstring decodes as valid UTF-8 via an inline validator. RED before (mid-char split → invalid), GREEN after. End-to-end verified: the storednodes.propertiesfor the fixture goes from Python-sqlite3-unreadable to valid UTF-8.Verification
Full local suite 6334 passed, 1 skipped; lint-ci clean.