Global Search

In [1]:

# Copyright (c) 2024 Microsoft Corporation.
# Licensed under the MIT License.

# Copyright (c) 2024 Microsoft Corporation. # Licensed under the MIT License.

In [2]:

import os

import pandas as pd
from graphrag.query.indexer_adapters import (
    read_indexer_communities,
    read_indexer_entities,
    read_indexer_reports,
)
from graphrag.query.structured_search.global_search.community_context import (
    GlobalCommunityContext,
)
from graphrag.query.structured_search.global_search.search import GlobalSearch
from graphrag.tokenizer.get_tokenizer import get_tokenizer
from graphrag_llm.completion import create_completion
from graphrag_llm.config import ModelConfig

import os import pandas as pd from graphrag.query.indexer_adapters import ( read_indexer_communities, read_indexer_entities, read_indexer_reports, ) from graphrag.query.structured_search.global_search.community_context import ( GlobalCommunityContext, ) from graphrag.query.structured_search.global_search.search import GlobalSearch from graphrag.tokenizer.get_tokenizer import get_tokenizer from graphrag_llm.completion import create_completion from graphrag_llm.config import ModelConfig

Global Search example¶

Global search method generates answers by searching over all AI-generated community reports in a map-reduce fashion. This is a resource-intensive method, but often gives good responses for questions that require an understanding of the dataset as a whole (e.g. What are the most significant values of the herbs mentioned in this notebook?).

LLM setup¶

In [3]:

api_key = os.environ["GRAPHRAG_API_KEY"]

config = ModelConfig(
    type="litellm",
    model_provider="openai",
    model="gpt-4.1",
    api_key=api_key,
)
model = create_completion(config)

tokenizer = get_tokenizer(config)

api_key = os.environ["GRAPHRAG_API_KEY"] config = ModelConfig( type="litellm", model_provider="openai", model="gpt-4.1", api_key=api_key, ) model = create_completion(config) tokenizer = get_tokenizer(config)

Load community reports as context for global search¶

• Load all community reports in the community_reports table from GraphRAG, to be used as context data for global search.
• Load entities from the entities tables from GraphRAG, to be used for calculating community weights for context ranking. Note that this is optional (if no entities are provided, we will not calculate community weights and only use the rank attribute in the community reports table for context ranking)
• Load all communities in the communities table from the GraphRAG, to be used to reconstruct the community graph hierarchy for dynamic community selection.

In [4]:

# parquet files generated from indexing pipeline
INPUT_DIR = "./inputs/operation dulce"
COMMUNITY_TABLE = "communities"
COMMUNITY_REPORT_TABLE = "community_reports"
ENTITY_TABLE = "entities"

# community level in the Leiden community hierarchy from which we will load the community reports
# higher value means we use reports from more fine-grained communities (at the cost of higher computation cost)
COMMUNITY_LEVEL = 2

# parquet files generated from indexing pipeline INPUT_DIR = "./inputs/operation dulce" COMMUNITY_TABLE = "communities" COMMUNITY_REPORT_TABLE = "community_reports" ENTITY_TABLE = "entities" # community level in the Leiden community hierarchy from which we will load the community reports # higher value means we use reports from more fine-grained communities (at the cost of higher computation cost) COMMUNITY_LEVEL = 2

In [5]:

community_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet")
entity_df = pd.read_parquet(f"{INPUT_DIR}/{ENTITY_TABLE}.parquet")
report_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet")

communities = read_indexer_communities(community_df, report_df)
reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL)
entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL)

print(f"Total report count: {len(report_df)}")
print(
    f"Report count after filtering by community level {COMMUNITY_LEVEL}: {len(reports)}"
)

report_df.head()

community_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet") entity_df = pd.read_parquet(f"{INPUT_DIR}/{ENTITY_TABLE}.parquet") report_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet") communities = read_indexer_communities(community_df, report_df) reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL) entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL) print(f"Total report count: {len(report_df)}") print( f"Report count after filtering by community level {COMMUNITY_LEVEL}: {len(reports)}" ) report_df.head()

Total report count: 10
Report count after filtering by community level 2: 10

Out[5]:

	id	human_readable_id	community	level	parent	children	title	summary	full_content	rank	rating_explanation	findings	full_content_json	period	size
0	9785e64b429dacd70cce8238051f3438b387ebcd6920ef...	7	7	1	0	[]	Paranormal Military Squad and Operation Dulce	This community centers on the Paranormal Milit...	# Paranormal Military Squad and Operation Dulc...	8.5	The impact severity rating is high due to the ...	[{'explanation': 'The Paranormal Military Squa...	{\n    "title": "Paranormal Military Squad and...	2026-01-13	4
1	212c8ff1796bff731abe3d54f3216511ab15d5fb0d97eb...	8	8	1	0	[]	Paranormal Military Squad: Leadership and Scie...	This community centers on the Paranormal Milit...	# Paranormal Military Squad: Leadership and Sc...	8.0	The community poses a high impact due to its i...	[{'explanation': 'Taylor Cruz, often referred ...	{\n    "title": "Paranormal Military Squad: Le...	2026-01-13	3
2	dbff99eaacade8ac4337a9bde0f533313445f0643ab3d0...	9	9	1	0	[]	Paranormal Military Squad and Operation: Dulce...	This community centers around the Paranormal M...	# Paranormal Military Squad and Operation: Dul...	8.5	The community poses a high impact severity due...	[{'explanation': 'The military complex is the ...	{\n    "title": "Paranormal Military Squad and...	2026-01-13	3
3	b6e8ab1ac8ecc605bbec09ad78819b95ac08d62c5a9991...	0	0	0	-1	[7, 8, 9]	Paranormal Military Squad and Operation: Dulce	This community centers on the Paranormal Milit...	# Paranormal Military Squad and Operation: Dul...	8.5	The community poses a high impact due to its e...	[{'explanation': 'Taylor Cruz is the central a...	{\n    "title": "Paranormal Military Squad and...	2026-01-13	10
4	97038bee58306d13b709b615fb2b816cae0c11273a9e73...	1	1	0	-1	[]	Team of Agents Investigating Dulce Base	This community centers on a specialized team o...	# Team of Agents Investigating Dulce Base\n\nT...	7.5	The impact severity rating is high due to the ...	[{'explanation': 'The core of this community i...	{\n    "title": "Team of Agents Investigating ...	2026-01-13	5

Build global context based on community reports¶

In [6]:

context_builder = GlobalCommunityContext(
    community_reports=reports,
    communities=communities,
    entities=entities,  # default to None if you don't want to use community weights for ranking
    tokenizer=tokenizer,
)

context_builder = GlobalCommunityContext( community_reports=reports, communities=communities, entities=entities, # default to None if you don't want to use community weights for ranking tokenizer=tokenizer, )

Perform global search¶

In [7]:

context_builder_params = {
    "use_community_summary": False,  # False means using full community reports. True means using community short summaries.
    "shuffle_data": True,
    "include_community_rank": True,
    "min_community_rank": 0,
    "community_rank_name": "rank",
    "include_community_weight": True,
    "community_weight_name": "occurrence weight",
    "normalize_community_weight": True,
    "max_tokens": 12_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)
    "context_name": "Reports",
}

map_llm_params = {
    "max_tokens": 1000,
    "temperature": 0.0,
}

reduce_llm_params = {
    "max_tokens": 2000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 1000-1500)
    "temperature": 0.0,
}

context_builder_params = { "use_community_summary": False, # False means using full community reports. True means using community short summaries. "shuffle_data": True, "include_community_rank": True, "min_community_rank": 0, "community_rank_name": "rank", "include_community_weight": True, "community_weight_name": "occurrence weight", "normalize_community_weight": True, "max_tokens": 12_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000) "context_name": "Reports", } map_llm_params = { "max_tokens": 1000, "temperature": 0.0, } reduce_llm_params = { "max_tokens": 2000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 1000-1500) "temperature": 0.0, }

In [8]:

search_engine = GlobalSearch(
    model=model,
    context_builder=context_builder,
    tokenizer=tokenizer,
    max_data_tokens=12_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)
    map_llm_params=dict(map_llm_params),
    reduce_llm_params=dict(reduce_llm_params),
    allow_general_knowledge=False,  # set this to True will add instruction to encourage the LLM to incorporate general knowledge in the response, which may increase hallucinations, but could be useful in some use cases.
    json_mode=False,
    context_builder_params=context_builder_params,
    concurrent_coroutines=32,
    response_type="multiple paragraphs",  # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report
)

search_engine = GlobalSearch( model=model, context_builder=context_builder, tokenizer=tokenizer, max_data_tokens=12_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000) map_llm_params=dict(map_llm_params), reduce_llm_params=dict(reduce_llm_params), allow_general_knowledge=False, # set this to True will add instruction to encourage the LLM to incorporate general knowledge in the response, which may increase hallucinations, but could be useful in some use cases. json_mode=False, context_builder_params=context_builder_params, concurrent_coroutines=32, response_type="multiple paragraphs", # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report )

In [9]:

result = await search_engine.search("What is operation dulce?")

print(result.response)

result = await search_engine.search("What is operation dulce?") print(result.response)

## Overview of Operation Dulce

Operation Dulce is a classified, high-stakes mission led by the Paranormal Military Squad, with its primary objective centered on investigating the enigmatic Dulce Base—a secretive and technologically advanced underground facility. The operation is designed to uncover hidden technologies, existential threats, and anomalous phenomena within the base. Its scope and objectives are considered to have profound implications for security, intelligence, and potentially the future of humanity [Data: Reports (4, 0, 3, 8, +more)].

## Mission Focus and Objectives

The core focus of Operation Dulce is the investigation and management of anomalous phenomena, particularly those associated with the Dulce base. This includes the retrieval and analysis of alien technology and the handling of existential threats that may arise from encounters with advanced alien technology or other unexplained phenomena. The mission is regarded as a critical asset in safeguarding national security against unconventional threats [Data: Reports (7, 2)].

## Operational Structure and Personnel

Operation Dulce is conducted by the Paranormal Military Squad, an elite unit specializing in addressing existential threats and handling alien technology. The squad operates from a secure military complex and relies on specialized agents, robust technical infrastructure, and advanced communications systems to carry out its mission. Key personnel involved in the operation include Sam Rivera, Alex Mercer, Taylor Cruz, and Dr. Jordan Hayes, making the mission a convergence point for significant figures and a career-defining moment for those involved [Data: Reports (4, 0, 1, 8, +more)].

## Facilities and Security

The operational hub for Operation Dulce is the high-security briefing room within the Dulce base. This facility supports mission planning and execution, emphasizing security, technological capability, and operational readiness. The base itself is a secure, complex facility equipped to support the launch and execution of such a sensitive mission [Data: Reports (5)].

## Technologies and Infrastructure

Operation Dulce is supported by advanced communication infrastructure, secure military facilities, and specialized equipment, such as encrypted radio transmitters. These resources ensure operational discipline and security during the investigation of the Dulce Base. The mission-centric structure is hierarchical, with the Paranormal Military Squad at its core, supported by specialized agents and mission technologies such as projectors and monitors for data analysis and planning [Data: Reports (0, 9, 1, 4, 7, 5, +more)].

## Nature and Significance of Threats

The operation is characterized by its focus on confronting both conventional and anomalous threats, including advanced alien technology, unexplained phenomena, and potential existential risks. This makes Operation Dulce a mission of significant national and possibly global importance [Data: Reports (4, 0, 3, 8, +more)].

## Implications and Consequences

Given the level of preparation, security measures, and the sensitive nature of the mission objectives, the success or failure of Operation Dulce may have far-reaching consequences. The operation’s outcome could impact not only national security but also broader questions about humanity’s future and its relationship with advanced or alien technologies [Data: Reports (4, 9, 0)].

---

**Summary:**  
Operation Dulce is a top-secret mission led by the Paranormal Military Squad to investigate and manage anomalous phenomena and advanced alien technology at the Dulce Base. The operation is marked by high security, advanced technology, and the involvement of key personnel, with the potential for significant national and global implications depending on its outcome [Data: Reports (4, 0, 3, 8, 7, 2, 1, 5, 9, +more)].

In [10]:

# inspect the data used to build the context for the LLM responses
result.context_data["reports"]

# inspect the data used to build the context for the LLM responses result.context_data["reports"]

Out[10]:

	id	title	occurrence weight	content	rank
0	4	Dulce Base and Operation: Dulce Community	1.0	# Dulce Base and Operation: Dulce Community\n\...	9.0
1	0	Paranormal Military Squad and Operation: Dulce	1.0	# Paranormal Military Squad and Operation: Dul...	8.5
2	3	Paranormal Military Squad and Operation: Dulce	1.0	# Paranormal Military Squad and Operation: Dul...	8.5
3	8	Paranormal Military Squad: Leadership and Scie...	1.0	# Paranormal Military Squad: Leadership and Sc...	8.0
4	9	Paranormal Military Squad and Operation: Dulce...	0.2	# Paranormal Military Squad and Operation: Dul...	8.5
5	1	Team of Agents Investigating Dulce Base	0.2	# Team of Agents Investigating Dulce Base\n\nT...	7.5
6	6	Dulce Base Mainframe Room Technology Network	0.2	# Dulce Base Mainframe Room Technology Network...	7.5
7	7	Paranormal Military Squad and Operation Dulce	0.4	# Paranormal Military Squad and Operation Dulc...	8.5
8	5	Dulce Base Operational Command: Briefing Room ...	0.4	# Dulce Base Operational Command: Briefing Roo...	7.5
9	2	Alien Technology Retrieval and Analysis at Dul...	0.2	# Alien Technology Retrieval and Analysis at D...	9.0

In [11]:

# inspect number of LLM calls and tokens
print(
    f"LLM calls: {result.llm_calls}. Prompt tokens: {result.prompt_tokens}. Output tokens: {result.output_tokens}."
)

# inspect number of LLM calls and tokens print( f"LLM calls: {result.llm_calls}. Prompt tokens: {result.prompt_tokens}. Output tokens: {result.output_tokens}." )

LLM calls: 3. Prompt tokens: 12905. Output tokens: 1535.