github-featured / infiniflow--ragflow

发布

0 发布 98 Git 标签

RSS 订阅源
  • dev-20260804 b59d6e8ba1
    比较

    refactor(ingestion/task): extract index-doc mapping into task/indexdoc package (#17749)

    frostbyte_neo 发布于 2026-08-04 02:05:27 +00:00

    Summary

    Extract the pipeline-output → search-engine index document mapping
    helpers out of the task package into a dedicated, dependency-light
    leaf package internal/ingestion/task/indexdoc.

    These functions are pure transforms (they only depend on
    common/utility) and are not task-orchestration concerns:

    • NormalizeChunks, DeepCopyChunks (was unexported deepCopyChunks),
      toChunkMaps → indexdoc/normalize.go
    • ProcessChunksForPipeline, RenameTextToContentWithWeight,
      GetEmbeddingTokenConsumption, cleanupConsumedChunkFields,
      mergeChunkMetadata, processChunkPositions,
      AggregateTableDocMetadata, resolveTableColumnConfig →
      indexdoc/process.go
    • AddPositions → indexdoc/position.go
    • EmbeddingTokenConsumptionKey constant → indexdoc/constants.go
      (task/constants.go keeps only GRAPH_RAPTOR_FAKE_DOC_ID)

    Call sites in pipeline_executor.go and golden_compare.go now
    reference the indexdoc package; package-task tests qualify the moved
    symbols.

    Why

    The task package had grown into a "orchestration + pure mapping +
    debug" mix. Splitting the pure mapping helpers into a leaf package
    sharpens package boundaries, removes a misleading top-level
    ingestion/chunk candidate (there are already parser/chunk and
    service/chunk), and lets the golden tool / future reuse pull in the
    mapping logic without dragging in task's dao/engine/service
    dependency graph (Go subpackage import does not pull in the parent).

    Test plan

    • build.sh --test ./internal/ingestion/task/... — green (task
      4.7s, indexdoc 0.007s), matching the pre-change baseline.
    • gofmt clean; build.sh builds both ragflow-cli and
      ragflow_server successfully.
    • Integration/E2E tiers are delegated to CI (need real MySQL/MinIO/ES
      services).

    Note: pipeline_e2e_test.go has a pre-existing compile error
    (server.ElasticsearchConfig / server.InfinityConfig are now defined
    under internal/server/config/, not re-exported by internal/server).
    This is unrelated to this change — the diff to that file is only the
    added indexdoc import and the qualified EmbeddingTokenConsumptionKey
    reference.

    下载附件