github-featured / lfnovo--open-notebook

发布

0 发布 44 Git 标签

RSS 订阅源
  • v1.8.5 6aabacfca6
    比较

    feat: use token-based sizing for embedding chunking (#749)

    frostbyte_neo 发布于 2026-04-19 16:49:09 +00:00 | 277 次提交 在此版本后已推送到 main

    • feat: make chunk sizing token-based with 512-token default

    • fix: defer embedding debug token metrics

    • chore: lower default chunk size to 400 tokens and document rationale

    The previous 512-token default matched exactly the context window of
    BERT-family embedders like mxbai-embed-large, leaving no margin for:

    • tokenizer mismatch between our o200k_base measurement and the
      embedder's own WordPiece tokenizer
    • occasional splitter overshoot (RecursiveCharacterTextSplitter can
      emit chunks slightly above chunk_size when separators are sparse)
    • special tokens ([CLS], [SEP]) that consume context-window budget

    400 tokens keeps ~20% headroom below 512 while still being a large
    improvement over the old character-based default for most content.
    Users with larger-context embedders can raise OPEN_NOTEBOOK_CHUNK_SIZE
    via env var. Also adds a CHANGELOG entry for the full PR behavior
    change.

    • chore: move chunking changelog entry under 1.8.5

    Target release is 1.8.5 — moving the Changed section out of Unreleased.

    Co-authored-by: Luis Novo lfnovo@gmail.com

    下载附件