Index migration to v3

In [1]:

# Copyright (c) 2024 Microsoft Corporation.
# Licensed under the MIT License.

# Copyright (c) 2024 Microsoft Corporation. # Licensed under the MIT License.

Index Migration (v2 to v3)¶

This notebook is used to maintain data model parity with older indexes for version 3.0 of GraphRAG. If you have a pre-3.0 index and need to migrate without re-running the entire pipeline, you can use this notebook to only update the pieces necessary for alignment. If you have a pre-2.0 index, please run the v2 migration notebook first!

NOTE: we recommend regenerating your settings.yml with the latest version of GraphRAG using graphrag init. Copy your LLM settings into it before running this notebook. This ensures your config is aligned with the latest version for the migration. The config changes from v2 to v3 are significant in places!

WARNING: This will overwrite your parquet files, you may want to make a backup!

In [2]:

# This is the directory that has your settings.yaml
PROJECT_DIRECTORY = "<your project directory>"

# This is the directory that has your settings.yaml PROJECT_DIRECTORY = "<your project directory>"

In [3]:

from pathlib import Path

from graphrag.config.models.graph_rag_config import GraphRagConfig
from graphrag_common.config import load_config
from graphrag_storage.storage_factory import create_storage

config = load_config(GraphRagConfig, config_path=Path(PROJECT_DIRECTORY))
storage = create_storage(config.output_storage)

from pathlib import Path from graphrag.config.models.graph_rag_config import GraphRagConfig from graphrag_common.config import load_config from graphrag_storage.storage_factory import create_storage config = load_config(GraphRagConfig, config_path=Path(PROJECT_DIRECTORY)) storage = create_storage(config.output_storage)

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[3], line 7
      3 from graphrag.config.models.graph_rag_config import GraphRagConfig
      4 from graphrag_common.config import load_config
      5 from graphrag_storage.storage_factory import create_storage
      6 
----> 7 config = load_config(GraphRagConfig, config_path=Path(PROJECT_DIRECTORY))
      8 storage = create_storage(config.output_storage)

File ~/work/graphrag/graphrag/packages/graphrag-common/graphrag_common/config/load_config.py:172, in load_config(config_initializer, config_path, overrides, set_cwd, parse_env_vars, load_dot_env_file, dot_env_path, config_parser, file_encoding)
    117 """Load configuration from a file.
    118 
    119 Parameters
   (...)    169     - If the parser fails to parse the configuration text.
    170 """
    171 config_path = Path(config_path).resolve() if config_path else Path.cwd()
--> 172 config_path = _get_config_file_path(config_path)
    174 file_contents = config_path.read_text(encoding=file_encoding)
    176 if parse_env_vars:

File ~/work/graphrag/graphrag/packages/graphrag-common/graphrag_common/config/load_config.py:38, in _get_config_file_path(config_dir_or_file)
     36 if not config_dir_or_file.is_dir():
     37     msg = f"Invalid config path: {config_dir_or_file} is not a directory"
---> 38     raise FileNotFoundError(msg)
     40 for file in _default_config_files:
     41     if (config_dir_or_file / file).is_file():

FileNotFoundError: Invalid config path: /home/runner/work/graphrag/graphrag/docs/examples_notebooks/<your project directory> is not a directory

In [4]:

def remove_columns(df, columns):
    """Remove columns from a DataFrame, suppressing errors."""
    df.drop(labels=columns, axis=1, errors="ignore", inplace=True)

def remove_columns(df, columns): """Remove columns from a DataFrame, suppressing errors.""" df.drop(labels=columns, axis=1, errors="ignore", inplace=True)

In [5]:

from graphrag_storage.tables.parquet_table_provider import ParquetTableProvider

# Create table provider from storage
table_provider = ParquetTableProvider(storage)

text_units = await table_provider.read_dataframe("text_units")

text_units["document_id"] = text_units["document_ids"].apply(lambda ids: ids[0])
remove_columns(text_units, ["document_ids"])

await table_provider.write_dataframe("text_units", text_units)

from graphrag_storage.tables.parquet_table_provider import ParquetTableProvider # Create table provider from storage table_provider = ParquetTableProvider(storage) text_units = await table_provider.read_dataframe("text_units") text_units["document_id"] = text_units["document_ids"].apply(lambda ids: ids[0]) remove_columns(text_units, ["document_ids"]) await table_provider.write_dataframe("text_units", text_units)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[5], line 4
      1 from graphrag_storage.tables.parquet_table_provider import ParquetTableProvider
      2 
      3 # Create table provider from storage
----> 4 table_provider = ParquetTableProvider(storage)
      5 
      6 text_units = await table_provider.read_dataframe("text_units")
      7 

NameError: name 'storage' is not defined

Update settings.yaml¶

If you have left the default settings for your vector store schema, you may need to set explicit values that map each embedding type to a vector schema name. If you have already customized your vector store schema it may not be necessary.

Old default index names:

• default-text_unit-text
• default-entity-description
• default-community-full_content

(if you left all of the defaults, check your output/lancedb folder to confirm the above)

v3 versions are:

• text_unit_text
• entity_description
• community_full_content

Therefore, with a v2 index need to explicitly set the old index names so it connects correctly. We no longer support the "prefix" - you can just set an explicit index_name for each embedding.

NOTE: we are also setting the default vector_size for each index below, under the assumption that you are using a prior default with 1536 dimensions. Our new default of text-embedding-3-large has 3072 dimensions, which will be populated as the default if unset. Again, if you have a more complicated situation you may want to manually configure this.

Here is an example of the new vector store config block that you may need in your settings.yaml:

vector_store:
  type: lancedb
  db_uri: output/lancedb
  index_schema:
    text_unit_text:
      index_name: default-text_unit-text
      vector_size: 1536
    entity_description:
      index_name: default-entity-description
      vector_size: 1536
    community_full_content:
      index_name: default-community-full_content
      vector_size: 1536