extract_comment_text capped over-long comments with a raw byte cut
(text[MAX_COMMENT_LEN] = 0), which splits a multi-byte UTF-8 character
straddling byte 500 and leaves an incomplete sequence. That corrupted
value lands verbatim in nodes.properties — syntactically valid JSON but
invalid UTF-8 — so any consumer decoding the SQLite TEXT column strictly
(Python's sqlite3, the default for tooling on the exported DB) throws on
read. It never surfaced on English-only code because a single-byte-per-
char comment can be cut anywhere safely; it reproduces on Japanese/
Chinese/etc. comments (reporter: EC-CUBE).
Snap the cut back over UTF-8 continuation bytes (10xxxxxx) to the lead
byte of the straddling character and terminate there, dropping the
partial character. Result is always valid UTF-8.
Reproduce-first: python_docstring_utf8_boundary_truncation_issue1017
builds a docstring with 3-byte CJK chars (E3 81 82) straddling the cap
and asserts the captured docstring decodes as valid UTF-8 (via an
inline validator). RED before (mid-char split), GREEN after.
Closes#1017
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
18e4ceb490