db36524a18
Timing
2025-09-10 14:21:23 -07:00
e1662fb2e1
Merge branch 'main' into replace-node2vec-with-gee
2025-09-10 14:17:11 -07:00
a398cc38bb
Update command to use no-discover-entity-types ( #2038 )
...
"no-entity-types" is an incorrect configuration parameter.
Co-authored-by: Alonso Guevara <alonsog@microsoft.com >
2025-09-09 16:46:06 -06:00
d779346369
Replace n2v with gee
2025-09-08 12:13:51 -07:00
ac95c917d3
Update fnllm. ( #2043 )
2025-09-05 08:52:05 -07:00
e1ae9bde00
Print runtime errors to console
2025-09-03 17:53:51 -07:00
1cb20b66f5
Input docs API parameter ( #2034 )
...
* Add optional input_documents to index API
* Semver
* Add input dataframe example notebook
* Format
* Fix docs and notebook
2025-09-02 16:15:50 -07:00
2030f94eb4
Refactor CacheFactory, StorageFactory, and VectorStoreFactory to use consistent registration patterns and add custom vector store documentation ( #2006 )
...
* Initial plan
* Refactor VectorStoreFactory to use registration functionality like StorageFactory
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Fix linting issues in VectorStoreFactory refactoring
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Remove backward compatibility support from VectorStoreFactory and StorageFactory
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Run ruff check --fix and ruff format, add semversioner file
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* ruff formatting fixes
* Fix pytest errors in storage factory tests by updating PipelineStorage interface implementation
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* ruff formatting fixes
* update storage factory design
* Refactor CacheFactory to use registration functionality like StorageFactory
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* revert copilot changes
* fix copilot changes
* update comments
* Fix failing pytest compatibility for factory tests
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* update class instantiation issue
* ruff fixes
* fix pytest
* add default value
* ruff formatting changes
* ruff fixes
* revert minor changes
* cleanup cache factory
* Update CacheFactory tests to match consistent factory pattern
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* update pytest thresholds
* adjust threshold levels
* Add custom vector store implementation notebook
Create comprehensive notebook demonstrating how to implement and register custom vector stores with GraphRAG as a plug-and-play framework. Includes:
- Complete implementation of SimpleInMemoryVectorStore
- Registration with VectorStoreFactory
- Testing and validation examples
- Configuration examples for GraphRAG settings
- Advanced features and best practices
- Production considerations checklist
The notebook provides a complete walkthrough for developers to understand and implement their own vector store backends.
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* remove sample notebook for now
* update tests
* fix cache pytests
* add pandas-stub to dev dependencies
* disable warning check for well known key
* skip tests when running on ubuntu
* add documentation for custom vector store implementations
* ignore ruff findings in notebooks
* fix merge breakages
* speedup CLI import statements
* remove unnecessary import statements in init file
* Add str type option on storage/cache type
* Fix store name
* Add LoggerFactory
* Fix up logging setup across CLI/API
* Add LoggerFactory test
* Fix err message
* Semver
* Remove enums from factory methods
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
Co-authored-by: Josh Bradley <joshbradley@microsoft.com >
Co-authored-by: Nathan Evans <github@talkswithnumbers.com >
2025-08-28 13:53:07 -07:00
69ad36e735
Fix id baseline ( #2036 )
...
* Fix all human_readable_id columns to start at 0
* Semver
2025-08-27 11:15:21 -07:00
30bdb35cc8
Selective embeddings loading ( #2035 )
...
* Invert embedding table loading logic
* Semver
2025-08-27 11:12:01 -07:00
77fb7d9d7d
Logging improvements ( #2030 )
...
* Turn down blob/cosmos exception reporting to match file storage
* Restore indexing-engine.log
* Restore some basic console logging and progress for index CLI
* Semver
* Ignore small ruff complaints
* Fix CLI console printing
2025-08-25 14:56:43 -07:00
469ee8568f
Release v2.5.0 ( #2028 )
2025-08-14 08:06:52 -06:00
7c28c70d5c
Switch from Poetry to uv for package management ( #2008 )
...
* Initial plan
* Switch from Poetry to uv for package management
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Clean up build artifacts and update gitignore
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* remove build artifacts
* remove hardcoded version string
* fix calls to pip in cicd
* Update gh-pages.yml workflow to use uv instead of Poetry
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* ruff formatting fixes
* update cicd workflow with latest uv action
* fix command to retrieve package version
* update development instructions
* remove Poetry references
* Replace deprecated azuright action with npm-based Azurite installation
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* skip api version check for azurite
* add semversioner file
* update more changes from switching to UV
* Migrate unified-search-app from Poetry to uv package management
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* minor typo update
* minor Dockerfile update
* update cicd thresholds
* update pytest thresholds
* ruff fixes
* ruff fixes
* remove legacy npm settings that no longer apply
* Update Unified Search App Readme
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
Co-authored-by: Josh Bradley <joshbradley@microsoft.com >
Co-authored-by: Alonso Guevara <alonsog@microsoft.com >
2025-08-13 18:57:25 -06:00
5713205210
Feat/additional context ( #2021 )
...
* Users/snehitgajjar/add optional api param for pipeline state (#2019 )
* Add support for additional context for PipelineState
* Clean up
* Clean up
* Clean up
* Nit
---------
Co-authored-by: Snehit Gajjar <snehitgajjar@microsoft.com >
* Semver
* Update graphrag/api/index.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Remove additional_context from serialization
---------
Co-authored-by: Snehit Gajjar <snehit.gajjar@gmail.com >
Co-authored-by: Snehit Gajjar <snehitgajjar@microsoft.com >
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2025-08-08 16:59:24 -06:00
1da1380615
Release v2.4.0 ( #1994 )
...
* Release v2.4.0
* Update changelog
2025-07-14 18:54:27 -06:00
dce02563eb
Fix/fnllm embedding limiter defaults ( #1993 )
...
* default embeddings tpm/rpm to null
* Semver
2025-07-14 18:00:45 -06:00
13bf315a35
Refactor StorageFactory class to use registration functionality ( #1944 )
...
* Initial plan for issue
* Refactored StorageFactory to use a registration-based approach
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Added semversioner change record
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Fix Python CI test failures and improve code quality
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* ruff formatting fixes
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
Co-authored-by: Josh Bradley <joshbradley@microsoft.com >
2025-07-10 12:08:44 -06:00
e84df28e64
Improve internal logging functionality by using Python's standard logging module ( #1956 )
...
* Initial plan for issue
* Implement standard logging module and integrate with existing loggers
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Add test cases and improve documentation for standard logging
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Apply ruff formatting and add semversioner file for logging improvements
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Remove custom logger classes and refactor to use standard logging only
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Apply ruff formatting to resolve CI/CD test failures
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Add semversioner file and fix linting issues
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* ruff fixes
* fix spelling error
* Remove StandardProgressLogger and refactor to use standard logging
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Remove LoggerFactory and custom loggers, refactor to use standard logging
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Fix pyright error: use logger.info() instead of calling logger as function in cosmosdb_pipeline_storage.py
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* ruff fixes
* Remove deprecated logger files that were marked as deprecated placeholders
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Replace custom get_logger with standard Python logging
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Fix linting issues found by ruff check --fix
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* apply ruff check fixes
* add word to dictionary
* Fix type checker error in ModelManager.__new__ method
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Refactor multiple logging.getLogger() calls to use single logger per file
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Remove progress_logger parameter from build_index() and logger parameter from generate_indexing_prompts()
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Remove logger parameter from run_pipeline and standardize logger naming
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Replace logger parameter with log_level parameter in CLI commands
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Fix import ordering in notebook files to pass poetry poe check
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Remove --logger parameter from smoke test command
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Fix Windows CI/CD issue with log file cleanup in tests
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Add StreamHandler to root logger in __main__.py for CLI logging
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Only add StreamHandler if root logger doesn't have existing StreamHandler
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Fix import ordering in notebook files to pass ruff checks
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Replace logging.StreamHandler with colorlog.StreamHandler for colorized log output
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Regenerate poetry.lock file after adding colorlog dependency
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Fix import ordering in notebook files to pass ruff checks
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* move printing of dataframes to debug level
* remove colorlog for now
* Refactor workflow callbacks to inherit from logging.Handler
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Fix linting issues in workflow callback handlers
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Fix pyright type errors in blob and file workflow callbacks
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Refactor pipeline logging to use pure logging.Handler subclasses
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Rename workflow callback classes to workflow logger classes and move to logger directory
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* update dictionary
* apply ruff fixes
* fix function name
* simplify logger code
* update
* Remove error, warning, and log methods from WorkflowCallbacks and replace with standard logging
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* ruff fixes
* Fix pyright errors by removing WorkflowCallbacks from strategy type signatures
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Remove ConsoleWorkflowLogger and apply consistent formatter to all handlers
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* apply ruff fixes
* Refactor pipeline_logger.py to use standard FileHandler and remove FileWorkflowLogger
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Remove conditional azure import checks from blob_workflow_logger.py
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Fix pyright type checking errors in mock_provider.py and utils.py
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Run ruff check --fix to fix import ordering in notebooks
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Merge configure_logging and create_pipeline_logger into init_loggers function
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* Remove configure_logging and create_pipeline_logger functions, replace all usage with init_loggers
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* apply ruff fixes
* cleanup unused code
* Update init_loggers to accept GraphRagConfig instead of ReportingConfig
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* apply ruff check fixes
* Fix test failures by providing valid GraphRagConfig with required model configurations
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* apply ruff fixes
* remove logging_workflow_callback
* cleanup logging messages
* Add logging to track progress of pandas DataFrame apply operation in create_base_text_units
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* cleanup logger logic throughout codebase
* update
* more cleanup of old loggers
* small logger cleanup
* final code cleanup and added loggers to query
* add verbose logging to query
* minor code cleanup
* Fix broken unit tests for chunk_text and standard_logging
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* apply ruff fixes
* Fix test_chunk_text by mocking progress_ticker function instead of ProgressTicker class
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
* remove unnecessary logger
* remove rich and fix type annotation
* revert test formatting changes my by copilot
* promote graphrag logs to root logger
* add correct semversioner file
* revert change to file
* revert formatting changes that have no effect
* fix changes after merge with main
* revert unnecessary copilot changes
* remove whitespace
* cleanup docstring
* simplify some logic with less code
* update poetry lock file
* ruff fixes
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: jgbradley1 <654554+jgbradley1@users.noreply.github.com >
Co-authored-by: Josh Bradley <joshbradley@microsoft.com >
2025-07-09 18:29:03 -06:00
27c6de846f
Update docs for 2.0+ ( #1984 )
...
* Update docs
* Fix prompt links
2025-06-23 13:49:47 -07:00
1df89727c3
Pipeline registration ( #1940 )
...
* Move covariate run conditional
* All pipeline registration
* Fix method name construction
* Rename context storage -> output_storage
* Rename OutputConfig as generic StorageConfig
* Reuse Storage model under InputConfig
* Move input storage creation out of document loading
* Move document loading into workflows
* Semver
* Fix smoke test config for new workflows
* Fix unit tests
---------
Co-authored-by: Alonso Guevara <alonsog@microsoft.com >
2025-06-12 16:14:39 -07:00
17e431cf42
Update typer ( #1958 )
2025-06-02 14:20:21 -07:00
4a42ac81af
Release v2.3.0 ( #1951 )
2025-05-23 15:19:29 -06:00
f1e2041f07
Fix/drift search reduce ( #1948 )
...
* Fix Reduce Response for non streaming calls
* Semver
2025-05-23 08:07:09 -06:00
7fba9522d4
Task/raw model answer ( #1947 )
...
* Add full_response to llm provider output
* Semver
* Small leftover cleanup
* Add pyi to suppress Pyright errors. full_content is optional
* Format
* Add missing stubs
2025-05-22 08:22:44 -06:00
fb4fe72a73
Fix/global reduce prompt ( #1942 )
...
* Add missing string formatter
* Semver
2025-05-20 17:00:32 -06:00
f5a472ab14
Upgrade pyarrow dependency to >=17.0.0 to fix CVE-2024-52338 ( #1939 )
2025-05-20 18:34:28 -04:00
24018c6155
Task/remove dynamic retries ( #1941 )
...
* Remove max retries. Update Typer args
* Format
* Semver
* Fix typo
* Ruff and Typos
* Format
2025-05-20 11:48:27 -06:00
36948b8d2e
Various minor updates ( #1932 )
...
* Add text unit ids to Community model
* Add graph utilities
* Turn off LCC for clustering by default
* Simplify embeddings config/flow
* Semver
2025-05-16 14:48:53 -07:00
ee1b2db4a0
Update to latest fnllm ( #1930 )
...
* Update to latest fnllm
* Semver + smoke tests
* Add --method to smoke tests indexing
* format...
* Adjust embeddings limiter
2025-05-15 14:57:01 -06:00
56a865bff0
Release v2.2.1 ( #1910 )
2025-04-30 18:15:01 -06:00
8fb95a6209
Fix/community report tuning ( #1909 )
...
* Fix community report prompt tuning
* Semver
* Format ...
2025-04-30 17:44:31 -06:00
8c81cc1563
Update Index as workflows ( #1908 )
...
* Incremental index as workflow
* Update function docs
* fix state management
* Remove update workflows when specifying workflows in the config
* Fix ruff errors
* Add semver
* Remove callbacks param
2025-04-30 16:25:36 -06:00
832abf1e0c
Fix graph creation ( #1905 )
...
* Add edge weight to all graph creation
* Semver
2025-04-29 18:18:49 -07:00
25bbae8642
Docs: Add models page ( #1842 )
...
* Add models page
* Update config docs for new params
* Spelling
* Add comment on CoT with o-series
* Add notes about managed identity
* Update the viz guide
* Spruce up the getting started wording
* Capitalization
* Add BYOG page
* More BYOG edits
* Update dictionary
* Change example model name
2025-04-28 17:36:08 -07:00
c8621477ed
Release/v2.2.0 ( #1897 )
...
* Release v2.2.0
* Missing patch
2025-04-25 18:19:29 -06:00
fbf11f3a7b
Optional embeddings ( #1890 )
...
* Make all tables optional for embeddings
* Semver
---------
Co-authored-by: Alonso Guevara <alonsog@microsoft.com >
2025-04-25 16:20:56 -07:00
56e0fad218
NLP graph parity ( #1888 )
...
* Update stopwords config
* Minor edits
* Update PMI
* Format
* Perf improvements
* Semver
* Remove edge collection apply
* Remove source/target apply
* Add edge weight to graph snapshot
* Revert breaking optimizations
* Add perf fixes back in
* Format/types
* Update defaults
* Fix source/target ordering
* Fix test
2025-04-25 17:09:06 -06:00
25b605b6cd
Snapshot full graph ( #1889 )
...
* Snapshot un-merged entities and relationships
* Semver
* Fix raw df modification
2025-04-25 14:14:48 -07:00
e2a448170a
Fix/minor query fixes ( #1893 )
...
* fixed token count for drift search
* basic search fixes
* updated basic search prompt
* fixed text splitting logic
* Lint/format
* Semver
* Fix text splitting tests
---------
Co-authored-by: ha2trinh <trinhha@microsoft.com >
2025-04-25 14:12:18 -07:00
ad4cdd685f
Support OpenAI reasoning models ( #1841 )
...
* Update tiktoken
* Add max_completion_tokens to model config
* Update/remove outdated comments
* Remove max_tokens from report generation
* Remove max_tokens from entity summarization
* Remove logit_bias from graph extraction
* Remove logit_bias from claim extraction
* Swap params if reasoning model
* Add reasoning model support to basic search
* Add reasoning model support for local and global search
* Support reasoning models with dynamic community selection
* Support reasoning models in DRIFT search
* Remove unused num_threads entry
* Semver
* Update openai
* Add reasoning_effort param
2025-04-22 14:15:26 -07:00
74ad1d4a0c
Update .vsts-ci.yml ( #1874 )
2025-04-10 10:31:03 -06:00
89381296c3
fix yaml path in unified-search-app ( #1873 )
2025-04-10 13:02:00 -03:00
66aab4267e
add vsts deploy file for unified search app ( #1869 )
...
* add vsts deploy file for un ified search app
* fix file name
* remove unused tasks
* remove unused file
2025-04-08 17:08:02 -03:00
0e1a6e3770
Unified search added to graphrag ( #1862 )
...
* unified search app added to graphrag repository
* ignore print statements
* update words for unified-search
* fix lint errors
* fix lint error
* fix module name
---------
Co-authored-by: Gaudy Blanco <gaudy-microsoft@MacBook-Pro-m4-Gaudy-For-Work.local >
2025-04-07 11:59:02 -06:00
61769dd47e
Vector Store Integration Tests ( #1856 )
...
* Add vector store id reference to embeddings config.
* generated initial vector store pytests
* cleaned up cosmosdb vector store test
* fixed class name typo and debugged cosmosdb vector store test
* reset emulator connection string
* remove unneccessary comments
* removed extra comments from azure ai search test
* ruff
* semversioner
* fix cicd issues
* bypass diskANN policy for test env
* handle floating point inprecisions
---------
Co-authored-by: Derek Worthen <worthend.derek@gmail.com >
2025-04-01 11:05:04 -04:00
ffd8db7104
Gnievesponce prompt tune embedd chunking ( #1826 )
...
* Added support for embeddings chunking as defined by the config.
* ran semvisor -t patch
* Eliminated redunant code by using the embed_text strategy directly
* Added fix to support brakets within the corpus text; For example, inline LaTeX within a markdown file
---------
Co-authored-by: Gabriel Nieves <gnievesponce@microsoft.com >
2025-03-31 12:38:01 -04:00
b7b2b562ce
fnllm version fix ( #1835 )
...
* Fix fnllm version
* Semver
2025-03-21 22:13:56 -07:00
3b1e70c06b
Update config docs (2.1.0) ( #1818 )
...
* Align docs with config
* Semver
* Spelling
* Format
* Spelling
2025-03-18 12:39:30 -07:00
813b4de99f
Fix API key reference for gh-pages ( #1821 )
2025-03-18 11:10:11 -07:00
ddc6541ab6
Add docs page about input formats ( #1784 )
...
* Add docs page about input formats
* Add json example
* Spelling
2025-03-11 17:37:46 -07:00
Nathan Evans