github-featured/langchain4j--langchain4j

Files

T

Dmytro Liubarskyi 2101288ad7 EmbeddingModel: request/response API with per-call parameters, multimodal inputs, and observability (#5735 )

## Issue
Closes #1153 — distinguish APIs for embedding queries vs. documents/keys
(adds `EmbeddingInputType.QUERY`/`DOCUMENT` as a per-call parameter,
plus opt-in `embeddingInputType(...)` on
  `EmbeddingStoreContentRetriever` / `EmbeddingStoreIngestor`).

Partially addresses #4019 — adds the multimodal image-embedding API at
the core level (`EmbeddingInput` of `Content` parts) and wires Cohere,
Voyage, Jina, Google (Gemini Embedding 2), and Bedrock Titan; does
  not implement it for `OnnxEmbeddingModel`.

Relates to #5142 — provider-specific / per-call parameters for OpenAI
embeddings (`OpenAiEmbeddingRequestParameters`: `user`,
`encodingFormat`, `customParameters`; e.g. NVIDIA NIM `input_type` via
custom
  parameters).

Relates to #4273 — observability for `EmbeddingModel` via listeners
(`EmbeddingModelListener` + request/response/error contexts, wired
across providers).

  ## Change

Introduces an `EmbeddingModel.embed(EmbeddingRequest) →
EmbeddingResponse` API, structured like `ChatModel`'s request/response
API, so embeddings can carry **per-call parameters** and **multimodal
inputs** and
participate in **observability**. Everything is additive and
`@Experimental`; the existing `embed(String)` / `embed(TextSegment)` /
`embedAll(List)` methods keep working unchanged.

  ### Core (`langchain4j-core`)
- New request/response types: `EmbeddingRequest`, `EmbeddingResponse`,
`EmbeddingResponseMetadata`, `EmbeddingRequestParameters` (+
`DefaultEmbeddingRequestParameters` and typed `EmbeddingParameter<T>`
  tokens), `EmbeddingInput`, `EmbeddingInputType`.
- New default methods on `EmbeddingModel`: `embed(EmbeddingRequest)`,
`doEmbed(...)`, `defaultRequestParameters()`, `supportedParameters()`,
`supportedContentTypes()`, `provider()`, `listeners()`.
- **Strict opt-in / fail-fast:** per-call parameters and content types
are token/type-checked; a request that uses something the model doesn't
declare is rejected with `UnsupportedFeatureException` instead of
being silently ignored. `overrideWith` preserves the provider-specific
parameters subtype (as on the chat side).
- **Multimodal:** an `EmbeddingInput` is an ordered list of `Content`
parts (text/image); models fuse them into one embedding (or
one-per-item, per provider). Modality is auto-detected — no manual flag.
- **Observability:** `EmbeddingModelListener` + request/response/error
contexts (same shape as `ChatModelListener`), fired inline from
`embed(EmbeddingRequest)`. `addListener(...)` still works.
- **RAG opt-in:** `EmbeddingStoreContentRetriever` and
`EmbeddingStoreIngestor` gain an optional `embeddingInputType(...)`
(QUERY / DOCUMENT). Default behavior is unchanged (no input type sent).
- `ModelProvider`: added `COHERE`, `VOYAGE_AI`, `JINA`, with matching
OpenTelemetry `gen_ai.provider.name` mappings (`cohere` is a well-known
OTel value; `voyage_ai` / `jina` are custom, as permitted by the
  spec).

### Providers
- **OpenAI** (dimensions, `user`/`encodingFormat`/custom params),
**Cohere** (Embed v4 multimodal + input types), **Voyage** (multimodal +
input types), **Jina** (CLIP multimodal), **Google AI Gemini** (input
types; **Gemini Embedding 2** multimodal), **Amazon Bedrock Titan**
(multimodal).
- **Google Gen AI** (`langchain4j-google-genai`): input type → SDK
`task_type`, per-call dimensions → `outputDimensionality`, `provider()`,
listeners.
- **Ollama**: text-only — `provider()` + listeners (per-call params
correctly fail fast).
- **In-process models** (ONNX / `AbstractInProcessEmbeddingModel`):
already work via the default `doEmbed→embedAll` bridge (text-only,
image/param requests fail fast); observability via `addListener(...)`.
No
code change (no builders to wire listeners into, no dedicated
`ModelProvider`).
- **Gemini Embedding 2** dropped the `task_type` parameter, so input
types are applied as prompt instructions (`task: search result | query:
…` / `title: none | text: …`) automatically; `gemini-embedding-001`
  still uses `task_type`.
- `modelName` in the response metadata reflects the API-reported model
where the provider returns one (OpenAI/Voyage/Jina), falling back to the
configured name.

  ### Tests
- `AbstractEmbeddingModelIT` — a shared IT base (like
`AbstractChatModelIT`) covering the new API, convenience methods,
listeners, and fail-fast; each provider adds a small
`common/…EmbeddingModelIT` that
parameterizes it and declares its capabilities via `supports*()`
overrides.
- Mock-based unit tests per provider for wire format / routing /
fail-fast (run in CI without keys), plus core value-type and listener
tests.

### Docs
- Embedding-model section in the RAG tutorial (request/response,
multimodal, query-vs-document opt-in), the EmbeddingModel listener
section in the Observability tutorial, the embedding contribution
guidance in
  `CONTRIBUTING.md`, and the six provider integration pages.

  ### Notes
- `EmbeddingResponseMetadata` intentionally has no `finishReason`
(embeddings have no finish reason). No real provider is affected: the
only provider that emits `STOP` (Cloudflare WorkersAI) overrides the
convenience methods directly, and every other provider always returned
`null` here.
- `revapi.json` suppressions were added where the new (non-breaking)
types are exposed in provider APIs.

  ## General checklist
  - [x] There are no breaking changes (API, behaviour)
  - [x] I have added unit and/or integration tests for my change
  - [x] The tests cover both positive and negative cases
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
  - [x] I have added/updated the documentation
- [ ] I have manually run all the unit and integration tests in the core
and main modules, and they are all green
- [ ] I have added an example in the examples repo (only for "big"
features)
  - [ ] I have added/updated Spring Boot starter(s) (if applicable)

---------

Co-authored-by: agent <agent@langchain4j.dev>

2026-07-09 22:09:34 +02:00

8.1 KiB

Raw Permalink Blame History

Thank you for investing your time and effort in contributing to our project, we appreciate it a lot! 🤗

General guidelines

For new integrations, please consider adding it in community repo first.
If you want to contribute a bug fix or a new feature that isn't listed in the issues yet, please open a new issue for it. We will triage it shortly.
Follow Google's Best Practices for Java Libraries
Keep the code compatible with Java 17.
When integrating third-party services, use the official SDK whenever possible. If no official SDK is available, implement the client using langchain4j-http-client and Jackson.
Avoid adding new dependencies as much as possible (new dependencies with test scope are OK). If absolutely necessary, try to use the same libraries which are already used in the project. Make sure you run mvn dependency:analyze to identify unnecessary dependencies.
Write unit and/or integration tests for your code. This is critical: no tests, no review!
The tests should cover both positive and negative cases.
Make sure you run all unit tests on all modules with mvn clean test. Some integration tests need the API token (key) to be set up as an environment variable in order to communicate with the configured model provider (look for "EnabledIfEnvironmentVariable" annotation to find out the name of this token).
Avoid making breaking changes. Always keep backward compatibility in mind. For example, instead of removing fields/methods/etc, mark them @Deprecated and make sure they still work as before.
Follow existing naming conventions.
Add Javadoc where necessary. There's no need to duplicate Javadoc from the implemented interfaces.
Follow existing code style present in the project. Run make lint and make format before commit.
Large features should be discussed with maintainers before implementation.

Opening an issue

Please fill in all sections of the issue template.

Opening a PR

Please open the PR as ready for review, not as a draft.
Before opening the PR, please make sure it is complete:
- Add unit and/or integration tests for your change (see the testing guidelines above). This is critical: no tests, no review!
- Add documentation (if required).
- Run ./mvnw spotless:check and ./mvnw spotless:apply to ensure compliance with the source code formatting of the project.
Fill in all the sections of the PR template.
Please make it easier to review your PR:
- Keep changes as small as possible.
- Do not combine refactoring with changes in a single PR.
- Avoid reformatting existing code.

Please note that we do not have the capacity to review PRs immediately. We ask for your patience. We are doing our best to review your PR as quickly as possible.

Guidelines on adding a new model integration

Please open PRs with new model integrations in the langchain4j-community repository
Integration with OpenAI is a good example.
Create integration test classes that extend from AbstractChatModelIT, AbstractStreamingChatModelIT, AbstractChatModelListenerIT, AbstractStreamingChatModelListenerIT and AbstractStreamingAiServiceIT. There are many examples in existing modules.
If the model provider supports embeddings, implement EmbeddingModel using the request/response API (embed(EmbeddingRequest), supportedParameters(), supportedContentTypes(), listeners()), and create an integration test class that extends from AbstractEmbeddingModelIT. Override the supports*() methods to declare the model's capabilities (per-call parameters such as input_type/dimensions, image/multimodal inputs); the base test then verifies embed(EmbeddingRequest), the convenience methods, listeners, and the fail-fast behavior for unsupported parameters/modalities. See the OpenAI module for an example.
If model provider supports tools, create an integration test class that extends from AbstractAiServiceWithToolsIT. There are many examples in existing modules.
If model provider supports structured outputs, create an integration test class that extends from AbstractAiServiceWithJsonSchemaIT. There are many examples in existing modules.
Document the new integration here, here and here.
Add an example to the examples repository, similar to this.
Add a new module to the appropriate section of the BOM.
It would be great if you could add a Spring Boot starter.

Guidelines on adding a new embedding store integration

Please open PRs with new embedding store integrations in the langchain4j-community repository
Integration with Chroma is a good example.
Add a {IntegrationName}EmbeddingStoreIT. It should extend from EmbeddingStoreWithFilteringIT (when store supports metadata filtering) or EmbeddingStoreIT and pass all tests.
Add a {IntegrationName}EmbeddingStoreRemovalIT. It should extend from EmbeddingStoreWithRemovalIT and pass all tests.
Document the new integration here, here and here.
Add an example to the examples repository, similar to this.
Add a new module to the appropriate section of the BOM.
It would be great if you could add a Spring Boot starter. (after

Guidelines on changing an existing embedding store integration

Ensure that your changes are backwards compatible. Embeddings and TextSegments persisted with the latest released version of LangChain4j should still work.

8.1 KiB Raw Permalink Blame History