bradyyie
0a5abcacb5
Backports the Orkes workflow classifier concept to OSS. The classifier is a tag derived from WorkflowDef.metadata (explicit metadata.classifier wins, else 'agent' when the AgentSpan stamp agent_sdk/agentDef is present, else 'workflow' for plain untagged defs). - common: new WorkflowClassifier utility; WorkflowSummary gains a classifier field (proto field 23) populated at index time - rest: GET /metadata/workflow supports ?classifier= filtering; workflow /search and /search-v2 support &classifier=, folded into the structured query as 'classifier IN (...)' - postgres/sqlite: classifier column on workflow_index (Flyway V17 / V5), written on index and filterable via the query builders; the 'workflow' token also matches pre-migration NULL rows - es7/es8/os2/os3: classifier keyword mapping for workflow documents
ES8 Persistence
This module provides Elasticsearch 8.x persistence for indexing workflows and tasks.
It uses the Elasticsearch Java API Client (elasticsearch-java) aligned with ES8.
Index management (ES8 best practices)
This module uses composable index templates, write aliases, and an ILM policy:
- ILM policy:
${indexPrefix}-default-ilm-policy - Rollover conditions (hot phase):
max_primary_shard_size=50gb - Write aliases:
${indexPrefix}_workflow,${indexPrefix}_task,${indexPrefix}_task_log,${indexPrefix}_message,${indexPrefix}_event - Initial indices:
${alias}-000001(created automatically when index management is enabled) - Refresh interval: defaults to
30sfor all indices (configurable)
Build and usage
1) Build-time module selection
When building the server, select the ES8 persistence module to avoid Lucene conflicts:
./gradlew :conductor-server:bootJar -PindexingBackend=elasticsearch8
-PindexingBackend=es8 is also accepted.
2) Runtime configuration
Select the Elasticsearch 8 backend in your configuration:
conductor.indexing.type=elasticsearch8
Elasticsearch version compatibility
The ES8 module uses elasticsearch-java client version 8.19.11.
For local Docker-based setups, use Elasticsearch 8.19.x (the provided compose file pins
8.19.11).
All other conductor.elasticsearch.* properties are shared with the ES7 module.
Configuration
(Default values shown below)
# A comma separated list of scheme/host/port of the ES nodes to communicate with.
# Scheme can be `http` or `https`. If scheme is omitted then `http` will be used.
conductor.elasticsearch.url=localhost:9200
# The name of the workflow and task index.
conductor.elasticsearch.indexPrefix=conductor
# Default refresh interval applied via the component template.
conductor.elasticsearch.indexRefreshInterval=30s
# Path to a PEM-encoded certificate to trust for HTTPS connections.
conductor.elasticsearch.trustCertPath=
# Worker queue size used in executor service for async methods in IndexDao.
conductor.elasticsearch.asyncWorkerQueueSize=100
# Maximum thread pool size in executor service for async methods in IndexDao
conductor.elasticsearch.asyncMaxPoolSize=12
# Timeout (in seconds) for the in-memory to be flushed if not explicitly indexed
conductor.elasticsearch.asyncBufferFlushTimeout=10
BASIC Authentication
If you need to pass user/password to connect to ES, add the following properties to your config file:
conductor.elasticsearch.username=someusername
conductor.elasticsearch.password=somepassword