* Fix JSONL loader for blank and invalid rows
* Sync docs for schema and metadata fields
* Revert "Sync docs for schema and metadata fields"
This reverts commit 55f170f7d25f58ba4494f2dee8c5690fd282c778.
* feat: native CosmosTableProvider with namespace partitioning
Replace the parquet-decomposition approach in AzureCosmosStorage with a
native CosmosTableProvider that implements TableProvider directly:
- CosmosTableProvider: stores DataFrame rows as Cosmos documents with
/namespace partition key. All queries are single-partition (no fan-out).
- CosmosTable: streaming Table impl with async SDK and server-side pagination.
- AzureCosmosStorage: simplified to key-value only (context.json, stats.json,
cache). child() now works via ':'-separated namespace prefixes.
- TableProvider.child(): new non-abstract method for namespace isolation.
ParquetTableProvider/CSVTableProvider delegate to Storage.child().
- Pipeline wiring: run_pipeline.py and utils.py use table_provider.child()
for update-run delta/previous isolation.
- Legacy fallback: CosmosTableProvider reads from old containers when
legacy_container is configured, enabling transparent migration.
Tested against Cosmos DB Linux emulator (vNext, ARM64).
302 unit tests + 15 verb tests pass (no regressions).
* fix: remove enable_cross_partition_query from async SDK calls
The async azure-cosmos SDK (v4.9) leaks this kwarg through to
aiohttp.ClientSession, causing TypeError. Omitting partition_key
achieves the same cross-partition behavior automatically.
Also documents the caveat in the design doc.
Verified: migration test passes all 5 phases against Cosmos emulator.
* feat: transactional batch writes with configurable batch_size
Add batch_size parameter (default 50, max 100) to CosmosTableProvider
and CosmosTable. Documents are written using Cosmos transactional
batch (execute_item_batch) for ~50× fewer network round-trips.
If a batch fails (e.g. payload too large), falls back to individual
upserts for that chunk so partial progress is never lost.
Config: table_provider.batch_size in settings.yaml
Propagates through child() and open() to streaming writes.
Tested: 120 rows at batch_size=50, 25 rows at batch_size=10,
75 streamed rows, clamping to max 100, child inheritance.
* chore: lint cleanup and dead code removal
- Remove unused _INTERNAL_FIELDS constant (duplicated _COSMOS_SYSTEM_KEYS)
- Fix TRY300: move returns to else blocks in AzureCosmosStorage
- Fix SIM105: use contextlib.suppress for CosmosResourceNotFoundError
- Fix SLF001: replace __new__ + private attr copy with __init__ in child()
- Fix RUF002: replace en-dash with hyphen in docstrings
- Fix D105: add __aiter__ docstring
- Add noqa: PERF401 for async iteration (false positive: no async listcomp)
- All ruff checks pass, pyright 0 errors, 317 tests pass
* fix: address code review findings
Critical fixes:
- Fix ID round-trip corruption: _strip_cosmos_metadata now restores
original id from row_id field. Previously, read_dataframe returned
'{table_name}:{key}' instead of the pipeline's original id value.
- Always store row_id on write (consistent between provider and table).
- has() now catches CosmosResourceNotFoundError specifically instead of
bare Exception — auth/network errors propagate correctly.
Medium fixes:
- Add asyncio.Lock to _ensure_container() for concurrent-task safety.
- _batch_upsert catches only CosmosBatchOperationError for fallback;
other exceptions (auth, network) now propagate instead of silently
falling back to individual upserts.
Verified: ID round-trip, streaming write, no-id tables all pass
against Cosmos emulator. 317 unit/verb tests pass.
* chore: fix spellcheck and add semversioner change
- Add dictionary words: aiohttp, aiter, colls, serde, upserts, vnext
- Fix British spellings: serialisation→serialization, initialisation→initialization, behaviour→behavior
- Replace 'Unparameterized' with 'Non-parameterized'
- Add semversioner minor change file
* fix: update test_clear assertion for new clear() behavior
clear() now drops and recreates the container instead of deleting the
entire database. The container and database clients remain valid after
clear() — only the data is removed.
* refactor: extract Cosmos connection from Storage, not TableProviderConfig
Connection fields (connection_string, account_url, database_name) removed
from TableProviderConfig. The factory extracts them from the affiliated
AzureCosmosStorage instance when table_provider.type is cosmosdb.
This eliminates config duplication — credentials are defined once on
output_storage, and table_provider only carries table-specific fields
(container_name, batch_size, legacy_container).
Config example:
output_storage:
type: cosmosdb
account_url: https://...
database_name: graphrag
container_name: graphrag-kv
table_provider:
type: cosmosdb
container_name: graphrag-tables
batch_size: 50
* perf: batch deletes in _delete_table to match write batching
Use transactional batches for delete operations instead of
one-at-a-time delete_item calls, mirroring the _batch_upsert pattern.
Falls back to individual deletes on CosmosBatchOperationError.
* feat(graphrag-vectors): add filtering, timestamps, and CRUD operations
Implement the vector store enhancements from the graphrag-vectors-design spec:
New modules:
- filtering.py: Pydantic-based filter expression system with F builder,
operator overloads, JSON serialization, client-side evaluate(), and
per-backend compilation (SQL for LanceDB/CosmosDB, OData for Azure AI Search)
- timestamp.py: ISO 8601 timestamp explosion into filterable component fields
Enhanced VectorStoreDocument:
- data: dict for user-defined metadata fields
- create_date / update_date: automatic ISO 8601 timestamps
Enhanced VectorStore base class:
- fields config for typed metadata columns
- insert / count / remove / update CRUD methods
- select, filters, include_vectors params on search methods
- Automatic timestamp explosion on insert/update
- User-defined date field explosion
Backend implementations (LanceDB, Azure AI Search, CosmosDB):
- Full filter compilation to native query languages
- Typed schema creation with user-defined fields
- All new CRUD operations
Breaking changes:
- search_by_id raises IndexError when document not found
- Updated indexer_adapters.py caller to handle the new exception
Tests:
- 54 unit tests for filtering and timestamp modules
- 28 LanceDB integration tests covering CRUD, filters, timestamps, select,
include_vectors, and user-defined date field explosion
* fix: resolve CI build failures (formatting, lint, pyright, test mocks)
- Fix ruff formatting and lint errors across all changed files
- Refactor filtering.py: move operator overloads from monkey-patching to
direct class methods for pyright visibility
- Use validation_alias/serialization_alias with populate_by_name for
Pydantic AND/OR/NOT models (pyright + runtime compatible)
- Use Operator enum members instead of string literals in FieldRef
- Add missing abstract methods (insert, count, remove, update) to test
mock VectorStore classes
- Update mock method signatures to match base class (select, filters,
include_vectors params)
- Add docstrings to FieldRef magic methods (ruff D105)
- Fix noqa:S608 placement in cosmosdb.py
* feat: add top-level vector_size to VectorStoreConfig
Add a vector_size field (default 3072) to VectorStoreConfig so users
can set it once instead of on every individual index schema. The value
is propagated to new IndexSchema entries during validation.
* chore: add semversioner patch entry
* chore: add ismatch and ftype to spellcheck dictionary
* Add example notebooks for LanceDB, Azure AI Search, and CosmosDB vector stores
- Three notebooks demonstrating: document loading, similarity search, metadata
filtering with F builder, timestamp filtering, document update/removal
- Sample data files (text_units.parquet, embeddings.text_unit_text.parquet)
- Add CPY001, SLF001, DTZ005 to notebook lint ignores in pyproject.toml
* refactor: extract model/tokenizer creation from generate_text_embeddings into callers
* add smoke tests for output csv
* change text fixture test
* change test
* add semver
* run format
---------
Co-authored-by: Nathan Evans <github@talkswithnumbers.com>
* Replace NX-based compute_degree with DataFrame-only implementation
- Add graphrag.graphs package with compute_degree operating directly on
relationships DataFrames instead of building NetworkX graphs
- Update finalize_entities and finalize_relationships to use the new
utility, eliminating NX graph construction in those paths
- Remove the old compute_degree operation from index/operations
- Add side-by-side tests validating parity with NetworkX degree output
* Add DataFrame-based connected components and LCC utilities
- Add connected_components and largest_connected_component to
graphrag.graphs using union-find on edge list DataFrames
- Fix compute_degree to normalise edge direction so (A,B) and (B,A)
are treated as the same undirected edge
- Replace NX largest_connected_component in prune_graph operation with
the new DataFrame utility via graph_to_dataframes
- Add realistic A Christmas Carol graph fixture (529 nodes, 978 edges)
converted from verb test parquet data
- Add side-by-side tests for connected components and fixture-based
test for compute_degree, all validated against NetworkX
* Add DataFrame-based stable LCC utility with side-by-side tests
* Wire stable_lcc into cluster_graph, replacing NX stable_largest_connected_component
* Remove NetworkX from clustering pipeline
- cluster_graph now accepts a DataFrame instead of nx.Graph
- hierarchical_leiden now accepts list[tuple[str, str, float]] edge list
- create_communities passes relationships DataFrame directly, removing
create_graph dependency
- Edge direction normalization and deduplication (keep='last') replaces
implicit NX dedup behavior
- Modularity helper callers convert to edge list via _nx_to_edge_list
* Remove NetworkX from prune_graph
- prune_graph operation now accepts (entities, relationships) DataFrames
instead of nx.Graph, returns pruned DataFrames directly
- Uses compute_degree for degree calculation, largest_connected_component
for LCC filtering — no NetworkX
- Workflow no longer round-trips through create_graph/graph_to_dataframes
- Reset index on returned DataFrames to avoid downstream alignment errors
* Move old NX utilities out of production code
- Move stable_lcc (NX version) to tests/unit/graphs/nx_stable_lcc.py
for side-by-side comparison tests only
- Delete graph_to_dataframes.py (dead code, zero imports)
- Update test imports to use the new test helper location
* Delete create_graph, inline into snapshot_graphml
- snapshot_graphml now accepts edges DataFrame directly and calls
nx.from_pandas_edgelist internally
- finalize_graph workflow passes relationships DataFrame to snapshot
- Removed create_graph.py (no remaining callers)
* Move graph utilities from index/utils/graphs.py to graphrag/graphs/ modules
- hierarchical_leiden, first/final_level_hierarchical_clustering → graphs/hierarchical_leiden.py
- calculate_pmi/rrf_edge_weights → graphs/edge_weights.py
- calculate_* modularity functions, _df_to_edge_list → graphs/modularity.py
- NX-based modularity/LCC/edge-list helpers removed (replaced by DF-based equivalents)
- Delete index/utils/graphs.py (no remaining callers)
- Update cluster_graph.py and build_noun_graph.py to import from new locations
- Inline NX largest_connected_component into test helper nx_stable_lcc.py
- Add side-by-side modularity tests (9 tests comparing DF vs NX)
* Add semversioner patch for NetworkX removal
* Spelling
* Spelling config
* Fix British English spellings to American English
* Add async iterator support to InputReader and use in load workflows
InputReader now implements __aiter__ so it can be used as `async for doc in reader`. The core iteration logic is in _iterate_files(), and read_files() delegates to the iterator for batch loading. Both load_input_documents and load_update_documents workflows now use the async iterator with dataclasses.asdict for DataFrame construction.
* Format
* Move document ID, human_readable_id, and raw_data setup from create_final_documents into load workflows
Consolidates core document field initialization (id string cast, human_readable_id index, raw_data default) into load_input_documents and load_update_documents so that create_final_documents only handles the text unit join. Also applies the same setup in the run_pipeline input_documents bypass paths.
* Remove overzealous input document assignment
* Semver
* Format
* add notebook example support for each package
* add notebook example support for each package
* semversioner change
* feedback implemented for notebooks
* feedback implemented for notebooks
* feedback implemented for notebooks
---------
Co-authored-by: Gaudy Blanco <gaudy-microsoft@MacBook-Pro-m4-Gaudy-For-Work.local>
* Add DataReader class for typed dataframe loading
Introduce DataReader that wraps TableProvider and applies type coercion
functions when loading dataframes from weakly-typed formats (e.g. CSV).
- Add DataReader class with methods for each table type: entities,
relationships, communities, community_reports, covariates, text_units,
and documents
- Add typed loading functions in dfs.py for community_reports, covariates,
text_units, and documents (entities, relationships, communities already
existed)
- Integrate DataReader into all 17 indexing workflows replacing raw
read_dataframe calls
- Integrate DataReader into CLI query's _resolve_output_files for typed
loading across all search types (global, local, drift, basic)
- Export DataReader from data_model package __init__
* Fix column check
Derek Worthen