github-featured / mudler--localai

发布

0 发布 177 Git 标签

RSS 订阅源
  • v4.5.6 02b007a31e
    比较

    feat(config): default swa_full:true for sliding-window-attention models (#10611)

    frostbyte_neo 发布于 2026-06-30 15:58:17 +00:00 | 1054 次提交 在此版本后已推送到 master

    LocalAI enables a cross-request prompt-prefix cache (cache_reuse, see
    core/config/serving_defaults.go) so repeated prefixes — system prompts,
    RAG context, agent scaffolds, multi-turn chat — are not reprocessed every
    turn. For sliding-window-attention (SWA) models (Gemma 2/3, Cohere2,
    Llama 4, ...) this silently does nothing: llama.cpp defaults to a reduced
    SWA KV cache sized to the sliding window, and that reduced cache cannot
    preserve a prompt prefix across requests, so every turn reprocesses the
    whole prompt anyway.

    llama.cpp's --swa-full (params.swa_full, already wired through the
    LocalAI llama.cpp backend's swa_full option) keeps the full KV cache so
    the shared prefix is reused. Enable it automatically, but only for models
    that are actually SWA: detection reads the gguf-parser-normalized
    <arch>.attention.sliding_window metadata (which also applies llama.cpp's
    family rules, e.g. Phi-3 → not SWA), right where the GGUF is already
    parsed for defaults. It is never applied to dense models (pure memory
    waste) and never overrides an explicit user swa_full/n_swa choice.

    Tradeoff: the full SWA cache scales with context_size, so it costs more
    memory at large contexts — hence the SWA gating and the documented
    swa_full:false opt-out.

    Assisted-by: Claude:claude-opus-4-8 [Claude Code] golangci-lint

    Co-authored-by: Ettore Di Giacinto mudler@localai.io

    下载附件