* Revert "Sync Haystack API reference on Docusaurus (#11118)"
This reverts commit 8d9df4c156.
* Revert "docs(snippets): replace HTML comments with JSX comments for MDX v4 compatibility (#11117)"
This reverts commit 2b1ca620a0.
Replace all `<!-- test-ignore -->` HTML comment markers in Python docstrings
and generated API reference docs with JSX comments `{/* test-ignore */}` to be
compatible with strict MDX parsing in Docusaurus v4.
Also update test_python_snippets.py to use the new JSX marker format for
`TEST_IGNORE_MARK`, `TEST_CONCEPT_MARK`, `TEST_RUN_MARK`, and
`TEST_REQUIRE_FILES_PREFIX` constants, and remove the `mdx1Compat: { comments: true }`
compatibility setting from docusaurus.config.js.
Closes#11104
* adding test-ignore to snippets that require API keys
* adding test-ignore to snippets that require extra dependencies
* fixing wrong placement of ignore tag
* adding ignore to fixture files / bad placeholders / network calls
* Revert "adding test-ignore to snippets that require API keys"
This reverts commit 35ca3cd6a00cc5192089e6f3b79ce393a23fcc4a.
* Reapply "adding test-ignore to snippets that require API keys"
This reverts commit e3cca7a8544e528a863fd18f5dd1b5c8d421f8b4.
* Revert "adding ignore to fixture files / bad placeholders / network calls"
This reverts commit ba8f685de9653f1d5f144ab6ac5de8cf4bae7557.
* Revert "fixing wrong placement of ignore tag"
This reverts commit f3869316732a58bf8a966b872548f896610a0716.
* Revert "adding test-ignore to snippets that require extra dependencies"
This reverts commit 55b1fdb8a1d3ed89aa9768fd6cdf7e2d7787714a.
* Revert "adding test-ignore to snippets that require API keys"
This reverts commit 35ca3cd6a00cc5192089e6f3b79ce393a23fcc4a.
* fixing some more code snippets and ignoring some
* fixing some more code snippets and ignoring some
* adding fixture files from tests
* setting path in workflow file
* activating push triggers
* using gtp-4o-mini
* generating new md files
* install hatch
* install hatch
* install ffmpeg
* debugging
* debugging
* debugging
* adding extra requirements
* Update .github/workflows/docs-website-test-docs-snippets.yml
Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
* Apply suggestion from @anakin87
Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
* Apply suggestion from @anakin87
Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
* Apply suggestion from @anakin87
Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
* wip
* wip
* wip
* wip
* running OpenAPIServiceConnector docstring code
* running OpenAPIConnector docstring code
* running OpenAI Embedder docstring code
* temporary enabling running on PR
* running serperdev code snippets
* using test files for the llm_document_content_extractor.py docstring
* adding AZURE env vars and step summary to github
* adding more embbeders to run
* temporary disabling openai text embedder
* debugging error on text embedders docstring
* cleaning + installing dev deps from pyproject.toml
* trying use deps from pyproject.toml
* skip script to generate dependencies
* debugging error embedders docstring
* debugging
* debugging
* debugging: reverting to external requirements
* using pyproject.toml dependencies + disabling SERPERDEV code snippets
* removing extra requirements.txt
* adding slack notification + disabling one more snippet relying on SERPERDEV
* removing all triggers runs only daily at 3:17 and has a workflow_dispatch
---------
Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
* chore: enable ANN ruff ruleset, exclude components and testing dirs
- Add ANN (flake8-annotations) to ruff select in pyproject.toml
- Globally ignore ANN401 (Any) for legitimate dynamic types
- Exclude haystack/components/** and haystack/testing/** from ANN checks
(mirrors existing mypy disallow_incomplete_defs=false overrides)
- Fix all 43 ANN violations in the remaining modules:
- Add -> None to __post_init__ in breakpoints, file_content,
image_content, sparse_embedding, streaming_chunk, tool, toolset, auth
- Add -> str to __str__ in filter_policy, auth, hf (2x)
- Add *args: Any, **kwargs: Any + return types to metaclass __call__
in component.py and document.py, and __new__ in chat_message.py
- Add -> None to async _runner() in async_pipeline.py
- Type _check_callable_compatibility args and return bool
- Add Callable return type to _dispatch_bm25
- Type send_telemetry decorator params and return
- Type __init_track__ and __setattr_warn__ wrapper functions
- Type _parse_date, _parse_generic_args, async run()
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: fix ANN violations in .github/utils and docs-website scripts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: fix mypy errors introduced by ANN annotations
- async_pipeline.py: _runner() returns Mapping[str, Any], not None
- dataclasses.py: extend type: ignore to cover arg-type in addition to call-arg
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* alphabetical order of ruff rules
* fix: add ANN type annotations to haystack/components and remove per-file-ignore
Fix all ANN (flake8-annotations) violations in haystack/components/ so
the per-file-ignore for that directory can be removed entirely.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore: remove haystack.components.* from mypy disallow_incomplete_defs override
All ANN type annotations have been added to haystack/components/, so the
mypy override is no longer needed for that module.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* add type annotations to run,warm_up, __str__
* none and any type annotations
* type annotations
* __init__ return type annotation
* list[Document] instead of Any return type annotations
* fmt
* typing
* add run_async return type annotation for LinkContentFetcher
* run_async return type annotations
* fixing docstrings
---------
Co-authored-by: David S. Batista <dsbatista@gmail.com>
* feat: Add ToolSearchToolset for dynamic tool discovery from large catalogs
Implements ToolSearchToolset - a Toolset subclass that enables dynamic tool
discovery from large catalogs. Tools are discovered via `search_tools` bm25
based special search tool and become available to the LLM.
Key features:
- Single discovery mode: "bm25", postpone "embedding" for future
- Passthrough mode for small catalogs (< search_threshold)
- Self-contained BM25L search engine implementation
- Full serialization support (to_dict/from_dict)
- Auto warm-up when iterating to ensure bootstrap tool availability
* fix: Correct misleading search_tools description about return value
The description claimed to return "a JSON array of tool definitions"
but actually returns a plain text confirmation message with tool names.
* fix: Warm up discovered tools before making them available
Tools discovered via search_tools were added to _discovered_tools
without calling warm_up(), causing tools that require initialization
(connections, model loading) to fail when invoked.
* fix: Override __getitem__ to return tools from dynamic iteration
The inherited __getitem__ accessed self.tools which is always empty
in ToolSearchToolset. This caused IndexError for valid indexes even
when tools were available through __iter__.
* fix: Warm up catalog tools in passthrough mode
* feat: Add clear method to reset discovered tools
* docs: Add release note for ToolSearchToolset
* fix: reno note format
* Improve search_tools prompt to elicit tool keywords over user intent
Renamed the `query` parameter to `tool_keywords` and refined the
description to guide LLMs toward providing vocabulary from tool
names/descriptions rather than echoing user requests.
Before: LLMs often passed user intent like "south of france highlights"
After: LLMs provide tool vocabulary like "route weather search"
This improves BM25 matching since it relies on lexical overlap with
indexed tool names and descriptions.
* Rename ToolSearchToolset to SearchableToolset
Rename class and module for clarity:
- tool_search_toolset.py -> searchable_toolset.py
- ToolSearchToolset -> SearchableToolset
- Update all imports, tests, and release notes
* Replace _BM25SearchEngine with InMemoryDocumentStore
Remove the hand-rolled BM25L engine (~107 lines) and delegate to
Haystack's built-in InMemoryDocumentStore.bm25_retrieval(), which
uses the same algorithm and tokenization. Update tests accordingly.
* Use create_tool_from_function for search_tools and test fixtures
Replace manual Tool construction with create_tool_from_function and
Annotated type hints in _create_search_tool. Reduce test duplication
by using create_tool_from_function for fixtures and large_catalog.
Consolidate 4 integration tests into 1 deterministic math test and
remove the redundant integration_catalog fixture.
* Simplify SearchableToolset
* Minor fix
* Add support for lazy toolsets like MCPToolset with lazy flag
* improvements
* fix and simplify
* more unit tests
* amke is_passthrough private
* raise notimplementederror
* improve/simplify serde
* do not mention clear in relnote
---------
Co-authored-by: anakin87 <stefanofiorucci@gmail.com>
* fix: Raise error when async function is passed to Tool
- Added validation in Tool.__post_init__ to check if function is async
- Added validation in create_tool_from_function for early error detection
- Updated docstring to clarify that functions must be synchronous
- Added tests for both Tool init and @tool decorator with async functions
Closes#9580
* fix: Remove redundant async check from create_tool_from_function
Per reviewer feedback, the async validation is only needed in the
Tool class itself, since create_tool_from_function creates a Tool.
- Remove async check from create_tool_from_function
- Update docstrings to remove async-related notes
- Remove redundant tests for async functions in from_function tests
- Add release note
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add docstring note and tests for async function validation
- Updated Tool class docstring to indicate function must be synchronous
- Added test_from_function_async_raises_error test
- Added test_tool_decorator_async_raises_error test
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Update default OpenAI model to gpt-5-mini
* Update tests
* Fix tool invoker tests
* Update integration tests
* Update fragile test
* More robust tests
* Harden test
* More test improvements
* Better prompting, to trigger tools
* Simplify
* Simplify more
* Simplify even more, less maintenance in the future
* Mark two integration tests flaky
* Fixing fine details
* Add release note for OpenAI default model update to gpt-5-mini
* Add upgrade section to OpenAI default model release note
* fix: warm up individual tools inside Toolsets in warm_up_tools()
Related Issues:
* Follows up on PR #9942 (feat: Add warm_up() method to ChatGenerators)
* Addresses bug discovered during implementation of PR #9942 for issue #9907
Proposed Changes:
The warm_up_tools() utility function was only calling warm_up() on
Toolset objects themselves, but not on the individual Tool instances
contained within them. This meant tools inside a Toolset were not
properly initialized before use.
This PR modifies warm_up_tools() to iterate through Toolsets and call
warm_up() on each individual tool, in addition to calling warm_up() on
the Toolset itself.
Changes:
- Modified warm_up_tools() in haystack/tools/utils.py to iterate through
Toolsets when encountered (both as single argument and within lists)
- Added iteration to call warm_up() on each individual Tool inside Toolsets
- Added comprehensive test class TestWarmUpTools with 7 test cases
How did you test it:
- Added 7 comprehensive unit tests in test/tools/test_tools_utils.py:
* test_warm_up_tools_with_none - handles None input
* test_warm_up_tools_with_single_tool - single tool in list
* test_warm_up_tools_with_single_toolset - KEY TEST: verifies both
Toolset and individual tools are warmed
* test_warm_up_tools_with_list_containing_toolset - toolset within list
* test_warm_up_tools_with_multiple_toolsets - multiple toolsets
* test_warm_up_tools_with_mixed_tools_and_toolsets - mixed scenarios
* test_warm_up_tools_idempotency - safe to call multiple times
Notes for the reviewer:
I discovered this bug while implementing PR #9942 (for issue #9907).
When a Toolset object is passed to a component's tools parameter, the
warm_up_tools() function only calls Toolset.warm_up(), which is a no-op.
It doesn't iterate through the individual tools inside the Toolset to
warm them up.
acknowledged by @vblagoje and @sjrl
This implementation:
- Modified warm_up_tools() to iterate through Toolsets and call warm_up() on each individual tool
- Added comprehensive tests for Toolset warming behavior
- Verified both the Toolset and its contained tools are warmed up
Checklist:
I have read the contributors guidelines and the code of conduct
I have updated the related issue with new insights and changes
I added unit tests and updated the docstrings
I've used one of the conventional commit types for my PR title: fix:
I documented my code
I ran pre-commit hooks and fixed any issue
* added release note
* refactor: move tool warm-up iteration to Toolset.warm_up()
Addresses architectural feedback - moved iteration logic from warm_up_tools()
to base Toolset.warm_up() for better encapsulation. Subclasses can now
override warm_up() to customize initialization without breaking the contract.
- Toolset.warm_up() now iterates and warms tools by default
- warm_up_tools() simplified to delegate to warm_up()
- Updated tests and release notes
---------
Co-authored-by: HamidOna13 <abdulhamid.onawole@aizatron.com>
Docker image release / Build base image (push) Has been cancelled
* Tools warmup initial
* Fix lint
* Improve pydocs for warm_up
* Further improve pydocs for warm_up
* No need to warm_up tools in Agent as they are warmed up by ToolInvoker
* Simplify Toolset __add__ logic
* Simplify _ToolsetWrapper
* Add unit tests
* ToolInvoker warm_up
* Improve Tool pydoc
* Resurrect serde_utils.py
* Update tests
* Call ToolInvoker warm_up in agent warm_up
* Lint
* Move warm_up tests to ToolInvoker
* Update tests
* Remove tests
* Pydoc nit
* PR feedback
* ToolInvoker's warm_up is idempotent
* Add reno note
* Update releasenotes/notes/tools-warm-up-support-e16cc043fed3653f.yaml
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
* Make ComponentTool warm_up idempotent
* Update warm_up_tools to use ToolsType
* Linting
* Add warm up test for mixed list of Tool/Toolset instances
---------
Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
LHMQ878