github-featured / ollama--ollama

发布

0 发布 616 Git 标签

RSS 订阅源
  • v0.20.0 de9673ac3f
    比较

    tokenizer: add byte fallback for SentencePiece BPE encoding (#15232)

    frostbyte_neo 发布于 2026-04-02 20:04:45 +00:00

    • tokenizer: add byte fallback for SentencePiece BPE encoding

    When BPE merging produces tokens not in the vocabulary, fall back to
    encoding each UTF-8 byte as <0xHH> byte tokens instead of silently
    dropping the character. Also teach Decode to convert <0xHH> tokens
    back to raw bytes.

    Fixes #15229, fixes #15231

    • tokenizer fixes
    下载附件