deps: bump tokenizers from 0.22.2 to 0.23.1 (#3149)

Bumps [tokenizers](https://github.com/huggingface/tokenizers) from
0.22.2 to 0.23.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/huggingface/tokenizers/releases">tokenizers's
releases</a>.</em></p>
<blockquote>
<h2>Release v0.23.1</h2>
<h2>TL;DR</h2>
<p><code>tokenizers 0.23.1</code> is the first proper stable release in
the <code>0.23</code> line — <code>0.23.0</code> only ever shipped as
<code>rc0</code> because the release pipeline itself was broken (Node
side hadn't shipped multi-platform binaries since 2023, Python side was
on <code>pyo3 0.27</code> without free-threaded support).
<code>0.23.1</code> is the version where everything actually goes out
the door together: full Node multi-platform wheels for the first time in
years, Python 3.14 (regular <strong>and</strong> free-threaded
<code>3.14t</code>), full type hints for every Python class, and a stack
of measurable perf wins on the BPE / added-vocab hot paths.</p>
<p>There is no functional <code>0.23.0</code> published — we tag
<code>0.23.1</code> directly so users don't accidentally pull a
never-shipped version.</p>
<hr />
<h2>🚨 Breaking changes</h2>
<ul>
<li><strong>Drop Python 3.9</strong> (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/1952">#1952</a>)
— <code>requires-python = &quot;&gt;=3.10&quot;</code>; 3.9 users stay
on <code>0.22.x</code>.</li>
<li><strong><code>add_tokens</code> normalizes <code>content</code> at
insertion</strong> (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/1995">#1995</a>)
— re-saved <code>tokenizer.json</code> may differ in the
<code>added_tokens</code> block. Existing files load unchanged.</li>
<li><strong>Type stubs are precise</strong> (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/1928">#1928</a>,
<a
href="https://redirect.github.com/huggingface/tokenizers/issues/1997">#1997</a>)
— methods that returned <code>Any</code> now return real types;
<code>mypy --strict</code> may surface previously-hidden errors. Stub
layout also moved from <code>tokenizers/&lt;sub&gt;/__init__.pyi</code>
to <code>tokenizers/&lt;sub&gt;.pyi</code>. This breaks the surface of
some of the processors like <code>RobertaProcessign</code>'s
<code>__init__</code> .</li>
<li><strong>3.14t-only</strong>: setters/getters return
<code>PyResult&lt;T&gt;</code> because of
<code>Arc&lt;RwLock&lt;Tokenizer&gt;&gt;</code>; a poisoned lock
surfaces as <code>PyException</code> instead of a panic.</li>
</ul>
<hr />
<h2>⚡ Performance — measured locally on this Mac, not lifted from
PRs</h2>
<p>Run with <code>cargo bench --bench &lt;name&gt; -- --save-baseline
v0_22_2</code> on <code>v0.22.2</code>, then <code>--baseline
v0_22_2</code> on <code>v0.23.1</code>. Numbers are point-in-time wall
clock on a single laptop; relative deltas are what matters, absolute
numbers will differ on CI hardware.</p>
<h3>Added-vocabulary deserialize — the headline win (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/1995">#1995</a>,
<a
href="https://redirect.github.com/huggingface/tokenizers/issues/1999">#1999</a>)</h3>
<p><code>bench: improve added_vocab_deserialize to reflect real-world
workloads</code> (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/2000">#2000</a>)
is now representative of how transformers actually loads tokenizer.json
files. The combined effect of <code>daachorse</code> for the matching
automaton plus the normalize-on-insert refactor is enormous on this
workload:</p>
<table>
<thead>
<tr>
<th>benchmark</th>
<th align="right">v0.22.2</th>
<th align="right">v0.23.1</th>
<th align="right">change</th>
</tr>
</thead>
<tbody>
<tr>
<td>100k tokens, special, no norm</td>
<td align="right">~410 ms</td>
<td align="right">248 ms</td>
<td align="right"><strong>−40%</strong></td>
</tr>
<tr>
<td>100k tokens, non-special, no norm</td>
<td align="right">~7.1 s</td>
<td align="right">273 ms</td>
<td align="right"><strong>−96%</strong></td>
</tr>
<tr>
<td>100k tokens, special, NFKC</td>
<td align="right">~395 ms</td>
<td align="right">235 ms</td>
<td align="right"><strong>−40%</strong></td>
</tr>
<tr>
<td>100k tokens, non-special, NFKC</td>
<td align="right">~7.4 s</td>
<td align="right">290 ms</td>
<td align="right"><strong>−96%</strong></td>
</tr>
<tr>
<td>400k tokens, special, no norm</td>
<td align="right">~15 s</td>
<td align="right">980 ms</td>
<td align="right"><strong>−94%</strong></td>
</tr>
</tbody>
</table>
<p>Real-world impact: loading a Llama-3-style tokenizer with a large set
of added tokens dropped from &quot;noticeable pause&quot; to
&quot;instant&quot;.</p>
<h3>BPE encode</h3>
<table>
<thead>
<tr>
<th>benchmark</th>
<th align="right">v0.22.2</th>
<th align="right">v0.23.1</th>
<th align="right">change</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>BPE GPT2 encode batch, no cache</code></td>
<td align="right">530 ms</td>
<td align="right">446 ms</td>
<td align="right"><strong>−16%</strong></td>
</tr>
<tr>
<td><code>BPE GPT2 encode batch</code> (cached)</td>
<td align="right">690 ms</td>
<td align="right">685 ms</td>
<td align="right">noise</td>
</tr>
<tr>
<td><code>BPE GPT2 encode</code> (single)</td>
<td align="right">1.95 s</td>
<td align="right">1.94 s</td>
<td align="right">noise</td>
</tr>
<tr>
<td><code>BPE Train (small)</code></td>
<td align="right">32.6 ms</td>
<td align="right">31.5 ms</td>
<td align="right">−3%</td>
</tr>
<tr>
<td><code>BPE Train (big)</code></td>
<td align="right">1.01 s</td>
<td align="right">988 ms</td>
<td align="right">−2%</td>
</tr>
</tbody>
</table>
<p>The BPE per-thread cache PR (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/2028">#2028</a>)
shows much larger wins on highly-parallel workloads (+47–62% at 88+
threads on a server box, per the PR's own measurements on Vera).
Single-thread batch numbers above are flat or slightly improved because
cache-hit overhead was already low without contention.</p>
<h3>Llama-3 encode</h3>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/huggingface/tokenizers/commit/7f1623b90b5adfb9bc327d4c3468d2f70bbce262"><code>7f1623b</code></a>
Bump version to 0.23.1</li>
<li><a
href="https://github.com/huggingface/tokenizers/commit/bbe43ad73d8fc8932b9d0e657ddee3cd70c649a4"><code>bbe43ad</code></a>
ci: release workflow fixes (node + python) (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/2043">#2043</a>)</li>
<li><a
href="https://github.com/huggingface/tokenizers/commit/ab0c5d8fc13eb1c5001d9c06806635e2b5a42e9f"><code>ab0c5d8</code></a>
Fix node release (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/2034">#2034</a>)</li>
<li><a
href="https://github.com/huggingface/tokenizers/commit/decd8e07dad15f296c0adc2bc3a560f62d3de2eb"><code>decd8e0</code></a>
bindings/python: free-threaded Python (3.14t) support (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/2041">#2041</a>)</li>
<li><a
href="https://github.com/huggingface/tokenizers/commit/3992692d483bf3177219b52cb101b1bb055c18e6"><code>3992692</code></a>
update for release (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/2033">#2033</a>)</li>
<li><a
href="https://github.com/huggingface/tokenizers/commit/bcdd25b97fcd78549903082ecf3ddd87d42c456b"><code>bcdd25b</code></a>
BPE cache: per-thread read-through cache to avoid RwLock atomics on hits
(<a
href="https://redirect.github.com/huggingface/tokenizers/issues/2028">#2028</a>)</li>
<li><a
href="https://github.com/huggingface/tokenizers/commit/618eb383f43e207139eb5cdb9bca17796b5e9bd7"><code>618eb38</code></a>
Bump follow-redirects in /tokenizers/examples/unstable_wasm/www (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/2024">#2024</a>)</li>
<li><a
href="https://github.com/huggingface/tokenizers/commit/b6b1688bef2e87efc91af18edf7ac38b4d2dfbe6"><code>b6b1688</code></a>
chore: bump doc-builder SHA for PR upload workflow (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/2025">#2025</a>)</li>
<li><a
href="https://github.com/huggingface/tokenizers/commit/19015d6b44aa3896626de5092e4171aed1b56d5b"><code>19015d6</code></a>
fix: use uvx --with cairosvg instead of uv pip install --system (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/2021">#2021</a>)</li>
<li><a
href="https://github.com/huggingface/tokenizers/commit/efbcc68e321c364c8f9541f1c93a158df54d7da4"><code>efbcc68</code></a>
Ci benchmarks (<a
href="https://redirect.github.com/huggingface/tokenizers/issues/2019">#2019</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/huggingface/tokenizers/compare/v0.22.2...v0.23.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=tokenizers&package-manager=cargo&previous-version=0.22.2&new-version=0.23.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Cargo.lock

@@ -1256,6 +1256,12 @@ dependencies = [
  "cmov",
 ]
 
+[[package]]
+name = "daachorse"
+version = "1.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6f55d7153ba3b507595872a3874803f07a8a81d1e888abed8e5db7da0597d6e2"
+
 [[package]]
 name = "darling"
 version = "0.20.11"
@@ -1562,7 +1568,7 @@ dependencies = [
  "safetensors",
  "serde",
  "serde_json",
- "tokenizers",
+ "tokenizers 0.22.2",
 ]
 
 [[package]]
@@ -1900,7 +1906,7 @@ dependencies = [
  "tempfile",
  "thiserror 2.0.20",
  "tiktoken-rs",
- "tokenizers",
+ "tokenizers 0.23.1",
  "toml",
  "tracing",
  "tree-sitter",
@@ -4318,6 +4324,39 @@ dependencies = [
  "derive_builder",
  "esaxx-rs",
  "getrandom 0.3.4",
+ "itertools 0.14.0",
+ "log",
+ "macro_rules_attribute",
+ "monostate",
+ "onig",
+ "paste",
+ "rand 0.9.4",
+ "rayon",
+ "rayon-cond",
+ "regex",
+ "regex-syntax",
+ "serde",
+ "serde_json",
+ "spm_precompiled",
+ "thiserror 2.0.20",
+ "unicode-normalization-alignments",
+ "unicode-segmentation",
+ "unicode_categories",
+]
+
+[[package]]
+name = "tokenizers"
+version = "0.23.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "44e5bea67576e04b6ff8564c5d9e09c2ef0cf476502245f2f120e497769d3112"
+dependencies = [
+ "ahash",
+ "compact_str",
+ "daachorse",
+ "dary_heap",
+ "derive_builder",
+ "esaxx-rs",
+ "getrandom 0.3.4",
  "indicatif",
  "itertools 0.14.0",
  "log",

crates/headroom-core/Cargo.toml

@@ -17,7 +17,7 @@ tiktoken-rs = "0.11"
 # `tokenizers` is the HuggingFace pure-Rust tokenizer crate. Default features
 # pull in `onig` for the BPE pre-tokenizer regex; that vendors oniguruma so it
 # builds without a system dep on macOS/Linux.
-tokenizers = "0.22"
+tokenizers = "0.23"
 # `hf-hub` is the HuggingFace Hub client. We use the blocking `ureq` transport
 # with `rustls` (no system OpenSSL dep — keeps the binary static-linkable for
 # AWS deploys). `from_pretrained` is called once at startup, so blocking is