The benchmarks and the reference-implementation equivalence harness
existed to justify the sanitizer rewrite. They have served that purpose,
so remove them along with the verbatim copy of the old pipeline they
carried.
Five checks move into sanitize_test.go rather than going away, because
none of them reference the old implementation and all of them guard
behaviour the rewrite introduced:
- isHTMLInert must be a fixed point of the live bluemonday policy,
checked byte by byte and as whole strings, with the accepted byte set
pinned explicitly. Nothing else fails if that set is widened, and
widening it changes sanitizer output.
- Both filters are fixed points on their own output, which is what
licenses Sanitize to skip its second pass.
- Clean ASCII sanitizes with zero allocations.
- Invalid UTF-8 is re-encoded to U+FFFD.
- Known payloads still lose content.
Net -560 lines.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sanitizing user-authored response fields ran multiple allocating passes
over every string regardless of content: FilterInvisibleCharacters
converted the whole input to []rune and back, FilterCodeFenceMetadata
split and rejoined every line, and bluemonday ran unconditionally. On
comment- and issue-heavy responses this dominated conversion CPU and
allocation.
Three changes, none of which alter output or widen what the policy
allows:
- FilterInvisibleCharacters scans first and copies only from the first
filtered rune, skipping ASCII runs without decoding them. Invalid
UTF-8 is still re-encoded to U+FFFD, matching the []rune round trip it
replaces.
- FilterCodeFenceMetadata walks lines in place and returns the input
when no line changes.
- FilterHTMLTags skips bluemonday for input that is provably a fixed
point of the policy: printable ASCII, TAB and LF, with none of the
five characters html.EscapeString rewrites. Sanitize also skips the
second invisible/code-fence pass when HTML normalization returned its
input unchanged, since both filters are fixed points there.
Equivalence is pinned by a verbatim copy of the previous pipeline: the
new code is diffed against it over a corpus of ~22k deterministic cases
plus two fuzz targets, and the fast path is checked byte by byte against
the live bluemonday policy.
Benchmarks (Intel Ultra 9 185H, n=6):
Sanitize/TitleASCII 5.35µs -> 114ns 1 -100% allocs
Sanitize/Comment1KiB 45.3µs -> 1.27µs 1 -100% allocs
Sanitize/Body64KiB 2.47ms -> 85.9µs 1 -100% allocs
30 issues x 2KiB body 3.00ms -> 88.6µs 1.55MiB -> 1.9KiB
100 comments x 1KiB 5.04ms -> 169µs 2.19MiB -> 6.3KiB
Content that genuinely needs rewriting still pays for it, and non-ASCII
text still goes through bluemonday by design.
Fixes#3117
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Address review feedback on the post-HTML-entity sanitization pass.
Entity decoding could still smuggle code-fence metadata past the
sanitizer. A first line such as "`​``steal secrets" is not a fence
in the raw input, so FilterCodeFenceMetadata left it alone; decoding the
entity and stripping the zero width space then produced a real fence with
its info string intact. Sanitize now re-runs the fence filter after the
input is fully normalized.
Filtering every variation selector also corrupted legitimate text: VS15
and VS16 select text or emoji presentation, so "✈️" was reduced to "✈",
and the Variation Selectors Supplement encodes registered CJK ideographic
variation sequences. Selectors are now filtered contextually. A selector
is kept when it can apply to the character it follows, and dropped when it
is orphaned, follows a removed or non-graphic character, or continues a
run of selectors. Supplement selectors additionally require a CJK
ideograph base, matching the Ideographic Variation Database. That keeps
the anti-smuggling property, since hidden payloads rely on selector runs,
without rewriting valid Unicode.
Also corrects a lowercase-hex test case that claimed uppercase digits.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
FilterInvisibleCharacters previously ran only before FilterHTMLTags,
so numeric HTML entities (e.g. ​ or ​) that bluemonday
decodes into invisible or bidirectional control characters could
survive sanitization untouched. Sanitize now applies the
invisible-character filter both before HTML processing (so raw
invisible characters don't interfere with code-fence parsing) and
again after, so entity-decoded characters cannot escape the policy.
Also expands the removal set to include:
- ARABIC LETTER MARK (U+061C), a directional format character in the
same family as the already-covered LRM/RLM marks.
- Variation selectors (U+FE00-U+FE0F) and the variation selectors
supplement (U+E0100-U+E01EF), which can be used to hide payloads
after emoji or other base characters.
Fixes#3101
Sam Morrow