github-featured / lmcache--lmcache

发布

0 发布 86 Git 标签

RSS 订阅源
  • v0.4.2 9d413181c3
    比较

    feat(kv_cache): enable asymmetric store/retrieve storages in PD backend (#2509)

    frostbyte_neo 发布于 2026-03-16 06:48:55 +00:00

    • feat(kv_cache): enable asymmetric save/remote storage in PD backend

    Remove the restriction that prevented using save_decode_cache and
    remote_backend simultaneously in Prefill-Decode (PD) separation scenarios.

    This change introduces pd_retrieve_locations and pd_store_location
    parameters to decouple the KV cache retrieval and storage logic. This
    enables an asymmetric cache flow:

    1. Prefill nodes transmit KV cache to Decode nodes via the PDBackend.
    2. Decode nodes write back their generated KV cache to a remote backend
      for subsequent prefill reuse.
    3. In multi-turn dialogue scenarios, subsequent
      prefill requests retrieve historical KV cache from the remote backend,
      significantly increasing Prefix Cache hit rates and reducing TTFT

    This decoupling provides greater flexibility for cross-instance cache
    management and improves overall pipeline efficiency in distributed
    inference.

    [ Compute Layer ]
    +----------------------+ +------------------+
    | Prefill Node | ===============>| Decode Node |
    | (Hit-Remote & GenKV) | (1) PDBackend | (Hit-PD & GenKV) |
    +-------^--------------+ +-------+----------+
    | |

    : :
    [ Storage Layer ]
    (2) pd_store_location
    (3) pd_retrieve_locations (Decode -> Pool)
    (Pool -> Prefill)
    v
    +-------+--------------------------------------------+
    Distributed Storage Pool
    [Node A] [Node B] [Node C] [Node D]
    <======= (Object Storage / NFS / DFS) =======>
    +----------------------------------------------------+

    Workflow:

    1. Prefill -> Decode (PDBackend): Initial KV transfer for the current turn.
    2. Decode -> Remote (Store): Decode saves updated KV to NFS for persistence.
    3. Remote -> Prefill (Retrieve): Next-turn prefill pulls from Remote,
      drastically increasing Prefix Cache hit rate for multi-turn dialogues.

    Signed-off-by: Tony Lin tony.lin@intel.com

    • small refactor

    Signed-off-by: Tony Lin tony.lin@intel.com

    • config examples for pd + remote backends

    Signed-off-by: Tony Lin tony.lin@intel.com

    • refactor: rename pd_retrieve_locations/pd_store_location to retrieve_locations/store_location

    Remove the PD-specific prefix to make the retrieve/store locations
    generic instead of being limited to PD only.

    This breaks the PD-only feature restriction and allows the mechanism
    to be reused by other roles/components.

    Signed-off-by: Tony Lin tony.lin@intel.com

    • move retrieve & store locations from storage manger to cache engine

    Signed-off-by: Tony Lin tony.lin@intel.com

    • add para validation check

    Signed-off-by: Tony Lin tony.lin@intel.com

    • config: replace hardcoded IP with placeholder in decoder remote configs

    Signed-off-by: Tony Lin tony.lin@intel.com

    • resolve conflicts and rebase to the latest

    Signed-off-by: Tony Lin tony.lin@intel.com

    • address review comments

    Signed-off-by: Tony Lin tony.lin@intel.com

    • add description in configurations.rst

    Signed-off-by: Tony Lin tony.lin@intel.com

    Signed-off-by: Tony Lin tony.lin@intel.com
    Co-authored-by: deng451e 57919305+deng451e@users.noreply.github.com

    下载附件