fix/docs-deploy-pythonpath
435 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Nicholas Cole
|
5c16c85322 | feat(agents): aggregate execution token usage (#1484) | ||
|
bradyyie
|
0da7d51136 |
feat(events): add start_agent event handler action
Adds a first-class "Start Agent" action to event handlers, alongside the
existing start_workflow/complete_task/fail_task/terminate_workflow/
update_workflow_variables. This starts a registered agent execution (via the
core WorkflowExecutor.startAgentExecution(AgentStartRequest), already
available on origin/main) directly from an event, without going through
start_workflow against an agent's underlying workflow definition — which
would bypass the agent input contract (prompt/media/context/session_id),
per-run model/timeout overrides, and idempotency handling that
startAgentExecution provides.
Java (common, core):
- EventHandler.Action.Type += start_agent; new flat Action.StartAgent payload
(name, version, prompt, sessionId, media, context, idempotencyKey) as
@ProtoField(id = 8) — kept flat rather than reusing AgentStartRequest to
keep proto generation simple.
- SimpleActionProcessor: new case templating the payload's fields against
the event via ParametersUtils (mirroring the existing startWorkflow
handler), then calling workflowExecutor.startAgentExecution(...). No SPI
needed — WorkflowExecutor and AgentStartRequest are both already visible
from core.
- grpc/AbstractProtoMapper.java and grpc/eventhandler.proto are regenerated
by the :conductor-common:protogen task (wired into :conductor-grpc's
build) — no hand-written gRPC code.
ui-next:
- Action enum + actionLabel ("Start Agent"), persistNewAction, the
START_AGENT_ACTION schema template, the EventHandlerForm render switch, and
the StartAgentAction TS type all follow the existing start_workflow
pattern.
- New StartAgentActionForm (ActionForms/StartAgentTask.tsx), modeled on
StartWorkflowActionForm: agent name/version pickers sourced from the
existing /agent/list endpoint and AgentSummary type (already used by
AgentTaskForm for the same purpose), prompt/session/idempotency-key text
fields, a newline-delimited media list, and a context key-value editor.
Tests: TestSimpleActionProcessor#testStartAgent (core) and
StartAgentTask.test.tsx (ui-next) cover payload templating/mapping and the
form's field wiring, respectively.
Note: this action is OSS + ui-next only in this change. It requires a mirror
in orkes-conductor (its own EventHandler model + OrkesActionProcessor case)
to work on Orkes, tracked separately.
|
||
|
Shailesh Padave
|
de6da295fc | Merge branch 'main' into feature/a2a-expose-all | ||
|
Viren Baraiya
|
0ff680b786 | feat(file-storage): replace LOCAL storage type with CONDUCTOR (#1412) | ||
|
Shailesh Padave
|
6e3d61130d | Merge branch 'main' into feature/a2a-expose-all | ||
|
Nicholas Cole
|
21ee2875ed | feat(ui): align agent definitions with workflows (#1411) | ||
|
Shailesh Padave
|
a9e6316ca1 | Merge branch 'main' into feature/a2a-expose-all | ||
|
nicholascole
|
1e0072f97d | refactor(ai): consolidate agent integration configuration | ||
|
Shailesh Padave
|
40cf12294e |
fix(agent): accept agentName as alias for name in conductor agent task
A2ACallRequest documents agentName as the field for specifying a
conductor agent, but ConductorAgentDelegate re-parses task input as
AgentStartRequest which only has name. Adding @JsonAlias("agentName")
lets both field names work, preventing silent null and the misleading
"AGENT (conductor) requires 'name'" error.
|
||
|
Viren Baraiya
|
3ea601884d | Enhances A2A/AgentSpan execution (#1356) | ||
|
Naomi Most
|
c8e63df574 |
feat(rest): add task signal endpoints (#1205)
* feat(rest): add task signal endpoints
Adds the task signal endpoints the Go SDK / CLI already call but OSS never
implemented (conductor-oss/conductor#1197):
POST /api/tasks/{workflowId}/{status}/signal (async)
POST /api/tasks/{workflowId}/{status}/signal/sync (sync, returns SignalResponse)
Before this, the SDK's SignalAsync hit POST /tasks/{wfId}/{status}/signal, which
had no matching route, so Spring fell through to the all-variable update route
/{workflowId}/{taskRefName}/{status} and tried to coerce the literal "signal"
into TaskResult.Status -> MethodArgumentTypeMismatchException. Adding the literal
/signal segment makes that pattern more specific, so it now wins; a MockMvc
routing test (with PathPatternParser, mirroring production) locks this in.
"Signal" finds the first non-terminal WAIT task in the workflow (descending into
running sub-workflows) and applies the given status + output to it, via the new
TaskService.signalTask. The sync variant then waits for the workflow to settle
into its next blocking/terminal state and renders a SignalResponse per the
returnStrategy, reusing the same poll-to-response logic as executeWorkflow --
extracted into WorkflowSignalResponder so both controllers share it.
Tests: TaskServiceTest (signalTask found / not-found), TaskResourceTest (async,
sync, not-found, route resolution). Full conductor-rest suite green.
* fix(signal): add signalTimeout field and E2E Groovy integration tests
Two issues raised in PR #1205 review:
1. `SignalResponse` was missing the `signalTimeout` boolean that Orkes sets
when the sync-signal poll times out. Without it a timed-out response looks
identical to a successful one. Added `signalTimeout` to `SignalResponse`
(absorbed by `WorkflowRun` and `TaskRun` via inheritance), propagated it
through `NotificationResult.toResponse()`, and set it to `true` in
`WorkflowSignalResponder`'s timeout fallback path.
2. Added `SignalTaskSpec` to the test-harness — a Spring Boot integration test
backed by a real Redis testcontainer that exercises `TaskService.signalTask()`
without any mocking: direct-WAIT-task signal, signal-with-no-blocker,
signal-on-nonexistent-workflow, and sub-workflow descent.
* test(signal): remove mocked signal tests — covered by SignalTaskSpec E2E
* fix(test): SignalTaskSpec — expect NotFoundException for missing workflow
ExecutionDAOFacade.getWorkflow() throws NotFoundException for unknown IDs
(does not return null). The test was asserting null — corrected to thrown().
The e2e WorkflowRerunTests failure in the same CI run is pre-existing and
unrelated to signal changes (it was already failing on the prior commit).
* test(e2e): HTTP-level signal endpoint tests (async + sync)
* fix: spotless formatting violations in SignalTaskTest
|
||
|
Viren Baraiya
|
80156f58ff | Task cancel contract (#1342) | ||
|
Viren Baraiya
|
c85fa0fb4f | support for agent task and other UI fixes | ||
|
Viren Baraiya
|
0110401072 | Resolve #1286: Support conductor agents in the AGENT tasks (#1288) | ||
|
Viren Baraiya
|
182d0d966b | Add Agentspan and Agentic Workflows to Conductor | ||
|
Viren Baraiya
|
978c47a9f0 |
Move SecretsDAO to org.conductoross package and simplify agentspan integration
Relocates SecretsDAO out of the legacy com.netflix.conductor.dao package, adapts SkillMetadataDAO implementations accordingly, and removes the AgentSpan embedded environment post-processor, principal filter, and env-backed credential store in favor of a simpler configuration wired directly through application.properties (agentspan.embedded). |
||
|
Viren Baraiya
|
21d1ff9f5a | ignore getExecutionMetadataIfHasData when serializing to JSON | ||
|
Miguel Prieto
|
411a800f2f |
Secrets & environment: workflow references (${workflow.secrets/env}) + task-declared runtimeMetadata (#1251)
Env-backed secrets & environment + task-declared runtimeMetadata (#1251) - Resolve ${workflow.secrets.X} / ${workflow.env.X} from CONDUCTOR_SECRET_* / CONDUCTOR_ENV_* env vars (secrets deferred, never persisted). - Inject task-declared secret/env values onto polled tasks via TaskDef.runtimeMetadata (wire-only). - Read-only GET /api/environment and /api/secrets. Part of #1152. |
||
|
bradyyie
|
0a5abcacb5 |
Add workflow classifier support for definitions and execution search
Backports the Orkes workflow classifier concept to OSS. The classifier is a tag derived from WorkflowDef.metadata (explicit metadata.classifier wins, else 'agent' when the AgentSpan stamp agent_sdk/agentDef is present, else 'workflow' for plain untagged defs). - common: new WorkflowClassifier utility; WorkflowSummary gains a classifier field (proto field 23) populated at index time - rest: GET /metadata/workflow supports ?classifier= filtering; workflow /search and /search-v2 support &classifier=, folded into the structured query as 'classifier IN (...)' - postgres/sqlite: classifier column on workflow_index (Flyway V17 / V5), written on index and filterable via the query builders; the 'workflow' token also matches pre-migration NULL rows - es7/es8/os2/os3: classifier keyword mapping for workflow documents |
||
|
Naomi Most
|
1c7ae90efb |
feat(rest): add GET /workflow/{workflowId}/status endpoint (#1200)
Adds a lightweight workflow status summary endpoint that the Go SDK's
GetWorkflowState (and the `conductor workflow status` CLI command) already
expects. Previously OSS only exposed GET /workflow/{workflowId}, so the SDK's
call to /workflow/{id}/status returned a 404 (conductor-oss/conductor#1197).
Mirrors the Orkes WorkflowResource.getWorkflowStatusSummary endpoint: returns
workflowId, correlationId, status, and optionally output/variables. The new
WorkflowStatus model sits alongside the existing SignalResponse/WorkflowRun
models in org.conductoross.conductor.model.
|
||
|
Pedro Becker
|
42a9ed69f2 |
Terminate workflow with custom failure workflow version (#696)
* Failure Workflow Version Adds method to allow terminating a workflow with a custom failure definition version. * Adds documentation * Improve "failureWorkflowVersion" usage description * Failure Workflow Version Adds method to allow terminating a workflow with a custom failure definition version. * Adds documentation * Improve "failureWorkflowVersion" usage description * Code review changes from @nthmost-orkes * Sets "failureWorkflowVersion" JSON snippet as Integer * E-mail field typo * Reverts interface method, avoiding brake changes on custom implementations * Checks for protobuf default value before setting "failureWorkflowVersion" on "fromProto" implementation * Code style --------- Co-authored-by: Naomi Most <naomi.most@orkes.io> |
||
|
Viren Baraiya
|
cde7b8e934 |
feat: Durable A2A (Agent2Agent) protocol — client, server, observability (#1195)
* feat(a2a): durable A2A protocol — client, server, observability
Add Agent2Agent (A2A) protocol support to the AI module, in both directions.
Client: CALL_AGENT / GET_AGENT_CARD / CANCEL_AGENT_TASK system tasks call remote
A2A agents over JSON-RPC with poll, streaming (SSE), and push modes. Durable by
design — deterministic messageId, state in the execution (not a thread), liveness
guards, and a push backstop. SSRF-guarded (IPv6 ULA + cloud-metadata; redirects off).
Server: expose any workflow as an A2A agent (one agent per workflow), idempotent
message/send -> startWorkflow (RETURN_EXISTING), tasks/get / tasks/cancel, and
multi-turn resume (a follow-up message/send completes the paused HUMAN/WAIT task
instead of starting a duplicate).
Observability: Micrometer counters via the shared Monitors registry and MDC
correlation keys across the A2A code paths.
Gated by conductor.integrations.ai.enabled (client) and conductor.a2a.server.enabled
(server).
* test(a2a): unit, real-agent interop, and real-engine durability tests
- Unit/wire tests (MockWebServer), embedded-agent e2e, push callback, server
JSON-RPC dispatch + multi-turn resume, mapper/worker, and observability.
- A2ASdkInteropTest: drives the client against the official a2a-sdk reference
agent launched as a subprocess (discovery, send, poll, streaming, message-mode);
self-skips when no Python with a2a-sdk is available.
- A2ADurableEngineEndToEndTest (test-harness): CALL_AGENT through the real decider
+ AsyncSystemTaskExecutor + Redis, proving crash/restart resume from persistence.
* docs(a2a): integration guide, examples, UI task types, design notes
- docs/devguide/ai/a2a-integration.md (AI Cookbook nav) with worked examples:
call/expose, multi-turn resume request/response, and push end-to-end.
- ai/examples: call / get-card / server + streaming / push / multi-turn / cancel,
indexed in the examples README; runnable interop + durable demos under test
resources.
- ui-next: register CALL_AGENT / GET_AGENT_CARD / CANCEL_AGENT_TASK task types.
- design/a2a: protocol study, durability proposal, and server design notes.
Note: a CI step to install a2a-sdk (so A2ASdkInteropTest runs in the build job)
is left as a follow-up — it needs a token with the `workflow` scope to land.
* refactor(a2a): move inbound server auth to enterprise (OSS open by default)
The A2A server's optional shared-secret api-key (conductor.a2a.server.api-key)
is removed from OSS: the server is now open by default, matching OSS Conductor
REST. Inbound authentication (API keys, OAuth/OIDC, mTLS, per-skill scopes,
signed Agent Cards) belongs to the enterprise build; front OSS with a
gateway/firewall.
Unchanged and kept in OSS as safe-by-default guards: client SSRF protection,
push-callback token auth, cross-agent execution isolation, and client→remote
per-call auth headers.
Removes authorized()/apiKey + the two api-key tests; docs and design notes
updated.
* docs(a2a): error-handling/troubleshooting, multi-agent + client multi-turn + LLM-pick examples
Docs: add an error-handling & retries section (FAILED vs FAILED_WITH_TERMINAL_ERROR
mapping for HTTP/JSON-RPC/SSRF/liveness) + a troubleshooting table; flesh out the
client multi-turn section with a worked SWITCH-on-input-required snippet; add an
"orchestrating multiple agents" subsection.
Examples (validated by ExampleWorkflowValidationTest):
- 27-a2a-multi-agent: FORK_JOIN calling agents in parallel → JOIN.
- 28-a2a-llm-pick-skill: GET_AGENT_CARD → LLM_CHAT_COMPLETE → CALL_AGENT.
- 29-a2a-client-multi-turn: SWITCH on input-required, re-call with same context/taskId.
* refactor(a2a)!: rename CALL_AGENT→AGENT, CANCEL_AGENT_TASK→CANCEL_AGENT + add agentType
Generalize the agent task types ahead of multi-runtime support:
- CALL_AGENT → AGENT (class CallAgentTask → AgentTask), CANCEL_AGENT_TASK →
CANCEL_AGENT. GET_AGENT_CARD kept (discovery/"Agent Card" is A2A-specific).
- New input field `agentType` (default "a2a") on all three tasks — the extension
point for native runtimes (langgraph, openai, …). Unknown values are rejected
with a clear error; only "a2a" is implemented today.
Updated across enum, handlers, request models, mapper, callback, tests,
test-harness, examples, docs, design notes, and the ui-next task registration.
BREAKING CHANGE: workflows using type "CALL_AGENT"/"CANCEL_AGENT_TASK" must switch
to "AGENT"/"CANCEL_AGENT".
* docs(a2a): clean up post-rename residue (grammar, example names, phrasing)
Fix 'A AGENT'→'An AGENT' grammar (A2AMetrics, design), reword example descriptions and a doc sentence left awkward by the mechanical rename, and update legacy example workflow names (a2a_call_agent_* → a2a_agent_*).
* feat(a2a): server-side message/stream (SSE)
Exposed workflow-agents now support A2A message/stream. The endpoint returns an
SSE stream driven off a dedicated daemon pool: the initial Task, status-update
events as the workflow's A2A state changes, artifact-update events as output is
produced, and a final status-update at a terminal/input-required state (or when
the stream window elapses). The agent advertises capabilities.streaming=true.
A web-agnostic A2AStreamSink keeps SseEmitter out of A2AWorkflowAgent so the
stream driver stays unit-testable. Config: conductor.a2a.server.stream-poll-
interval-millis (500), stream-max-duration-seconds (300).
Tests: a mocked unit test asserting the task → artifact-update → final
status-update sequence, and a real-HTTP loopback (our client message/stream
against the server's SSE) aggregating to a completed task with artifacts.
Docs + design notes updated (streaming moves from follow-up to shipped).
* Update A2ADurableEngineEndToEndTest.java
* Update A2ADurableEngineEndToEndTest.java
* refactor(a2a): simplify client/server task code (no behavior change)
- AgentTask: replace hand-rolled isBlank/firstNonBlank/stripTrailingSlash/
asInt/asLong helpers with commons-lang3 StringUtils + Number-only reads;
reuse parseRequest().getPollIntervalSeconds() for pollInterval; dedup
stateOf via A2AResults; fix the stale {@link #isAsyncComplete} javadoc to
describe the real backstop-poll mechanism
- A2ACallbackResource: drop the deprecated query-param token path (untested,
no legacy clients on a feature branch); StringUtils for isBlank
- A2AServerResource/A2AWorkflowAgent: delete the duplicate basePath()/URL
building (delegate to the agent's agentUrl/new agentCardUrl); drop the
no-op `id == null ? null : id` ternaries
- remove dead fields A2AMessage.referenceTaskIds and PushNotificationConfig.id
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(a2a): add mermaid use-case diagrams (client & server directions)
Diagrams rendering both A2A directions — Conductor as client (workflow calls
remote agents via the AGENT task) and as server (a workflow exposed as an A2A
agent):
- docs/devguide/ai/a2a-integration.md: both-directions overview flowchart,
AGENT lifecycle sequence (poll/push/stream), FORK_JOIN→JOIN multi-agent
fan-out, and the server message/send→workflow→tasks/get sequence
- design/a2a/08: overview flowchart + a sequence per direction (A/B)
- design/a2a/10: server sequence emphasizing the streaming and multi-turn
resume paths
Rendered via pymdownx.superfences (mkdocs) and natively on GitHub.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai)!: rename package ai.models -> ai.model (singular)
Align the AI module with conductor-oss package conventions: every model
package in the codebase (com.netflix.conductor.common.model, core.model,
grpc/proto/model, ai.a2a.model) is singular `model` — `ai.models` was the
only plural `models` package in the entire repo, and sat one letter away
from the singular ai.a2a.model, which read as an inconsistency.
Pure mechanical rename: git mv of the package dir (history preserved) plus
package-decl/import/FQN updates across the module (main + test) and one doc
reference in ai/CONTRIBUTING.md. No type or behavior changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix /a2a endpoint not being reachable
* style: spotlessApply
* fix and upgrade version
* refactor for agentspan
* feat: add SQLite (sqlite-vec) vector database backend
Add an embedded, zero-infrastructure VectorDB backend implemented with the
sqlite-vec extension, alongside the existing pgvector/MongoDB/Pinecone backends.
- SqliteVectorDB stores embeddings in a vec0 virtual table and runs KNN via the
sqlite-vec MATCH operator (l2/cosine/l1).
- The native vec0 binaries (linux/macos/windows) are downloaded + checksum-verified
at build time (downloadSqliteVec) and bundled as jar resources; SqliteVecExtensions
extracts the right one at runtime.
- SqliteVectorDBAutoConfiguration registers a "default" instance automatically when
conductor.db.type=sqlite and conductor.integrations.ai.enabled=true, so the
all-in-one SQLite server gets a working vector store with no external dependency.
- Adds unit tests, a real end-to-end round-trip test, a RAG example
(30-rag-sqlite-vec.json), and documentation.
Note: a dedicated Linux e2e CI job (.github/workflows/ci.yml) is intentionally not
included here because the push token lacks the GitHub `workflow` scope; it must be
committed separately with a workflow-scoped token.
* test: enable workflow execution lock in integration profile to fix flaky specs
The integration-test profile disabled conductor.app.workflow-execution-lock-enabled
(since 2021), so acquireLock() was a no-op and the background WorkflowSweeper could
decide a workflow concurrently with specs' manual sweep()/asyncSystemTaskExecutor
calls. Two simultaneous decides could schedule duplicate tasks, intermittently
failing exact task-count assertions (FailureWorkflowSpec, and
HierarchicalForkJoinSubworkflowRetrySpec).
Enable the lock (matching production defaults; local_only lock is already configured)
so decides are serialized. Full :conductor-test-harness:test passes (252 tests).
* fix(build): wire downloadSqliteVec as a resource source so all consumers depend on it
The generated sqlite-vec resource directory is consumed by sourcesJar (and other
resource consumers), but only processResources depended on downloadSqliteVec. Gradle's
task-dependency validation (run as part of `build`) failed sourcesJar with an implicit
dependency error. Register the task provider itself as the resource srcDir so every
consumer depends on it automatically.
* fix(ai): harden sqlite-vec backend after review
Follow-up fixes from a deep review of the SQLite/sqlite-vec backend:
- SqliteVectorDB.upsertEmbeddings: restore autoCommit(true) in a finally block
so connections are never returned to the Hikari pool in manual-commit mode.
- SqliteVectorDB.getSqliteClient: validate the extension path against an
allowlist before interpolating it into load_extension(...).
- SqliteVectorDBAutoConfiguration.resolveDbPath: strip JDBC URL query params
(e.g. ?busy_timeout=15000&journal_mode=WAL) so the derived *_vectordb.db path
is a valid filename.
- SqliteVecExtensions: re-extract the bundled binary if the cached temp file was
deleted between calls (stale-cache guard).
- VectorDBProvider: warn when an auto-registered default is skipped because an
explicit instance with the same name already exists.
Adds SqliteVectorDBAutoConfigurationTest (resolveDbPath cases incl. query
params), SqliteVecExtensionsTest (platform detection), and a rollback/
autocommit-restore test in SqliteVectorDBTest.
* feat(ui-next): config forms for all AI / agentic task types
Wire up the workflow-editor task form dispatcher, Add-Task menu, icons, task
generators and JSON schemas for every AI task type exposed by the `ai/` module,
so each gets a real config form instead of the generic JSON editor.
- New bespoke forms: AGENT, GET_AGENT_CARD, CANCEL_AGENT, LIST_MCP_TOOLS,
CALL_MCP_TOOL, GENERATE_IMAGE, GENERATE_AUDIO, GENERATE_VIDEO, GENERATE_PDF,
LLM_SEARCH_EMBEDDINGS.
- Wire the existing-but-unwired LLM forms (chat/text complete, embeddings,
index text/document, search index, get document) into TaskFormContent.
- LLM_CHAT_COMPLETE: OSS plain "Instructions" textarea (enterprise prompt-name
picker remains pluggable via the plugin registry).
- A2A forms: agentType as a fixed A2A/Conductor radio; multi-row headers via the
shared HTTP headers editor; pushNotification is a boolean toggle.
- CALL_MCP_TOOL arguments accept inline JSON or a ${variable} reference.
- Media-gen + search-embeddings reuse the field-driven LLMFormFields for
provider/model/vectorDB/embedding selection (server-backed dropdowns).
- Full field coverage vs the backend input models; numeric fields use coerceTo
so the workflow JSON stores real numbers while still accepting ${variables}.
- A2A tasks added to the Agentic Orchestration quick-add section.
* fix(ui-next): address PR review on AI task forms
- Schemas.ts: add the new AI/agentic task types to genericSchema's type enum so
the code (JSON) editor no longer warns "type is not one of ..." for AGENT,
GENERATE_*, *_MCP_*, LLM_SEARCH_EMBEDDINGS, CHUNK_TEXT, LIST_FILES, PARSE_DOCUMENT.
- AgentTaskForm: fix spacing so the "Push backstop poll" field's floating label
no longer collides with the push-notification toggle (group switches, separate
the input row).
- LLMSearchEmbeddingsTaskForm: add spacing so the helper text no longer collides
with the Query field's floating label.
- QuickAddMenu: fill the two empty Agentic Orchestration slots (Call MCP Tool,
Generate Image) for a clean two-row grid.
* fix(ui-next): snug node height for AI/agentic task cards
AI task cards render header-only content (no custom body), but fell through to
the DEFAULT node height (100px), leaving empty space below the label on the
canvas. Size the AI/agentic header-only task family to 80px (matching the
header-only FORK_JOIN node) so the cards fit their content.
---------
Co-authored-by: Nicholas Cole <68611647+NicholasDCole@users.noreply.github.com>
Co-authored-by: nicholascole <nicholas.colesd@gmail.com>
|
||
|
Naomi Most
|
797126ee0f |
Revert "feat(webhooks-oss): WAIT_FOR_WEBHOOK end-to-end — port + persistence …" (#1202)
This reverts commit
|
||
|
Naomi Most
|
227f0e24c5 |
feat(webhooks-oss): WAIT_FOR_WEBHOOK end-to-end — port + persistence across 5 backings (#1106)
* feat: port webhooks-oss module from orkes-conductor
Lifts the new webhooks-oss/ module created by orkes-io/orkes-conductor#3612
(scheduler-style OSS/enterprise split) into conductor-oss. All io.orkes.conductor.*
packages translated to org.conductoross.conductor.* per repo convention.
Replaces the prior webhook-task/ module (deleted on main, only stale build/
remnants existed) with a structurally faithful copy of webhooks-oss from
orkes-conductor.
Ported
- 17 main + 7 test files comprising the webhooks-oss module
(verifiers, hashing, IncomingWebhookService, WebhookTaskMapper,
WebhookWorkerProperties, etc.)
- 10 supporting classes (EventMessage, ErrorList, Tag, WebhookConfig,
IncomingWebhookEvent, WebhookExecutionHistory, WebhookDAO,
WebhookTaskService, TargetWorkflowCollector, TimeBasedUUIDGenerator)
into common/ and core/
Code rewrites beyond mechanical namespace translation
1. ApplicationException -> NonTransientException (2 sites in
IncomingWebhookService, 3 sites in TimeBasedUUIDGenerator).
OSS retired ApplicationException.
2. TimeBasedUUIDGenerator stripped of multi-tenant OrkesRequestContext
lookups. generate() still produces real time-based UUIDs via log4j
UuidUtil. generate(long) (test-only, unused in OSS) falls back to
UUID.randomUUID() with TODO. getOrgId() returns "_".
3. EventMessage: removed orgId field and List<ExtendedEventExecution>
eventExecutions field (transitive class not ported; orgId is tenant
context).
4. IncomingWebhookService: replaced two executionDAOFacade.addEventMessage()
calls with log.warn(). OSS's ExecutionDAOFacade has no addEventMessage
(Orkes-only audit feature). Rejected events still observable via logs;
audit-table persistence is a future enhancement.
5. Deleted 3 postgres DAO tests (AbstractTestDAO, PostgresDAOTestUtil,
PostgresWebhookTaskServiceTest). They exercise a PostgresWebhookTaskService
impl that has not been ported yet.
6. Inlined FeatureFlags.SECURITY as literal "conductor.security.enabled"
in 5 verifier tests. Avoided porting a 20-line constants class for one
string reference.
7. TargetWorkflowCollectorTest: org.apache.groovy.util.Maps.of(...) ->
java.util.Map.of(...). Same semantics.
8. WebhookTaskMapper relocated from com.netflix.conductor.core.execution.mapper
to org.conductoross.conductor.tasks.webhook (per repo namespace
convention). Added explicit imports for TaskMapper and TaskMapperContext.
9. webhooks-oss/build.gradle adds commons-lang3, spring-web (test), and
spring-boot-starter-web (compileOnly) for the at-compile-time imports.
Wiring
- settings.gradle: include 'webhooks-oss'
- server/build.gradle: implementation project(':conductor-webhooks-oss')
Verified
- ./gradlew :conductor-webhooks-oss:compileJava + compileTestJava: clean
- ./gradlew :conductor-webhooks-oss:test: all green
- ./gradlew :conductor-server:compileJava: clean (full server tree)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(webhooks-oss): in-memory DAO + REST endpoint for first end-to-end webhook
Adds the runtime pieces needed to receive a webhook over HTTP and store it
in the queue for later dispatch. The dispatch worker and config-CRUD API
land in the next iteration.
Added
- InMemoryWebhookDAO: lifted from the prior feat/webhook-foundation
attempt, extended to satisfy the richer WebhookDAO interface ported
from orkes. Annotated @Component @ConditionalOnMissingBean(name=
"webhookDAO") so a persistent impl can override.
- InMemoryWebhookTaskService: lifted similarly, with @Component
@ConditionalOnMissingBean(name="webhookTaskService").
- IncomingWebhookResource: tenant-free OSS port of the
webhooks-enterprise REST controller. POST /webhook/{id} and GET
/webhook/{id}. ~60 lines.
Code rewrite
- IncomingWebhookService: swapped constructor-injected TimeBasedUUIDGenerator
for IDGenerator (already a @Bean in ConductorCoreConfiguration). Removes
the @ConditionalOnProperty wiring trap and avoids needing
conductor.id.generator=time_based. Webhook event IDs are now random UUIDs
rather than time-based UUIDs, which is fine for the queue path that uses
them.
Matchers note
InMemoryWebhookDAO.createMatchers/getMatchers/removeMatchers are no-ops.
Orkes' impl computes matchers by reading WorkflowDefs and extracting
inputParameters.matches from WAIT_FOR_WEBHOOK tasks, which requires
MetadataDAO injection. Inert until a worker consumes them, so deferred
with an inline comment.
Verified
- :conductor-webhooks-oss:compileJava + compileTestJava: clean
- :conductor-webhooks-oss:test: all green
- :conductor-server:compileJava: clean
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(webhooks-oss): WebhookConfigService + WebhookConfigResource for CRUD
Adds the management surface so webhook configs can be registered before
they receive events. Combined with the previous commit's REST receive
endpoint, this is the full register-then-receive arc.
Added
- WebhookConfigService: tenant-free OSS port of orkes' enterprise
service. CRUD over WebhookDAO, calls TargetWorkflowCollector to
populate matchers, sanitizes secrets on read. No audit logging
(AuditUtils is enterprise-only); no FeatureFlags conditional
(always enabled in OSS).
- WebhookConfigResource: REST controller mounted at
/api/metadata/webhook (mirrors orkes path). POST/PUT/GET/DELETE +
GET-all. No security annotations (@PreAuthorize / @PostFilter are
enterprise-only); no tag endpoints (TagsService is enterprise).
Validation rules preserved.
Code rewrites vs orkes
- ApplicationException(CONFLICT, ...) -> ConflictException
- ApplicationException(NOT_FOUND, ...) -> NotFoundException
- ApplicationException(INVALID_INPUT, ...) -> NonTransientException
- TimeBasedUUIDGenerator -> IDGenerator (consistent with prior commit)
- Dropped AuditUtils, FeatureFlags.WEBHOOKS conditional, all security
annotations, all tag endpoints
- Dropped createdBy = OrkesAuthentication.getAuthenticatedUser().getId()
since OSS has no authenticated-user concept here
Verified
- :conductor-webhooks-oss:compileJava + compileTestJava: clean
- :conductor-webhooks-oss:test: all green
- :conductor-server:compileJava: clean
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(webhooks-oss): WebhookWorker + real matchers for end-to-end dispatch
Completes the end-to-end webhook arc. An OSS server can now:
1. Register a webhook config (POST /api/metadata/webhook)
2. Receive an incoming webhook (POST /webhook/{id})
3. Verify the signature
4. Store the event + enqueue
5. The worker polls the queue, matches against waiting WAIT_FOR_WEBHOOK
tasks via the hash service, completes those tasks, and starts any
workflows declared in workflowsToStart.
Added
- WebhookWorker: OSS port of the Orkes Enterprise WebhookWorker. Plain
@Component with @PostConstruct-started ScheduledExecutorService and
@PreDestroy shutdown (no LifecycleAwareComponent base, no
MetricsCollector / Monitors, no OrkesRequestContext orgId switching,
no ExtendedEventExecution audit bookkeeping). Failures log instead
of persist. ~250 lines.
Changes
- InMemoryWebhookDAO.createMatchers now actually computes matchers:
injects MetadataDAO, reads WorkflowDef by name+version, finds
WAIT_FOR_WEBHOOK / WAIT tasks, extracts inputParameters.matches and
keys them by workflowName;version;taskRef. Mirrors orkes'
PostgresWebhookDAO.createMatchers minus the orgId prefix and
workflowDefToUpdateMap cache.
Code rewrites vs orkes WebhookWorker
- LifecycleAwareComponent -> plain @Component with @PostConstruct /
@PreDestroy
- MetricsCollector, Monitors.error -> log statements
- OrkesRequestContext.setOrgId per message -> dropped
- recordEventExecution(...) (writes ExtendedEventExecution audit
records via ExecutionDAOFacade.updateEventExecution) -> log
statements. ExtendedEventExecution / ExtendedEventHandler are not
ported to OSS.
- executionDAOFacade.addEventMessage(...) -> dropped (method doesn't
exist in OSS; same call previously stubbed in IncomingWebhookService)
- orkesRedisExecutionDAO.getTask(taskId) -> executionDAOFacade.getTaskModel(taskId)
- WorkflowConsistency.DURABLE -> dropped (enum doesn't exist in OSS)
- ApplicationException -> dropped (we already log + throw via log line)
Verified
- :conductor-webhooks-oss:compileJava + compileTestJava: clean
- :conductor-webhooks-oss:test: green
- :conductor-server:compileJava: clean
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(webhooks-oss): cover OSS-added pieces (62 tests, 0 failures)
Adds unit-test coverage for the four OSS-original classes added in
prior commits on this branch. None of these had tests previously.
Added
- WebhookConfigServiceTest (8 tests): id generation, conflict on
existing id, fresh-id passthrough, secret sanitization on read,
redacted-secret preservation on update, real-secret overwrite,
TargetWorkflowCollector delegation, removeWebhook ordering.
- InMemoryWebhookDAOTest (8 tests): CRUD for configs and events,
plus the matchers computation logic added in the previous commit:
null override stores empty, missing WorkflowDef skipped, real
WAIT_FOR_WEBHOOK task with matches stored under
workflowName;version;taskRef, task without matches skipped,
non-webhook task skipped, removeMatchers drops.
- WebhookConfigResourceTest (8 tests): validation (missing all three
targets, HEADER_BASED without headers), valid delegation, path-id
override on update, 404 on missing get/delete, secret sanitization
on get.
- WebhookWorkerTest (6 tests): event-not-found and config-not-found
early returns, workflowsToStart invokes WorkflowExecutor with the
right StartWorkflowInput (name, version, input merge, event name,
createdBy), non-integer version skipped, matcher hit completes the
waiting WAIT_FOR_WEBHOOK task and removes it from the task service,
terminal task is not re-completed.
Change
- WebhookWorker.handleMessage made package-private so the test can
invoke it directly without driving the ScheduledExecutorService.
Verified
- :conductor-webhooks-oss:test passes 62 tests, 0 failures
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(webhooks-oss): WebhookTest for the put() delegation
Closes the last untested link in the integration chain. When a workflow
reaches a WAIT_FOR_WEBHOOK task, the Webhook WorkflowSystemTask's
start() must register the task with WebhookTaskService so the worker
can find it via hash later.
- WebhookTest (1 test): verifies start() calls
webhookTaskService.put(taskModel, workflow.getWorkflowVersion()) and
sets status to IN_PROGRESS.
Verified: :conductor-webhooks-oss:test 63/63 pass.
* test(webhooks-oss): end-to-end integration test wiring real beans
Exercises the full webhooks-oss bean graph end-to-end with only the
WorkflowExecutor, QueueDAO, ExecutionDAOFacade, MetadataDAO, and
ParametersUtils mocked. Validates the integration story without
needing a full @SpringBootTest (which would require a circular dep
on test-harness).
Added
- WebhooksOssEndToEndTest (4 tests):
- register_then_receive_storesEventAndPushesQueue: full POST path
from REST controller through config DAO → IncomingWebhookService
→ verification → event store → queue push.
- receive_unknownWebhookId_returnsNullAndNoEnqueue: idempotent-safe
drop when no config is registered.
- register_workflow_completion_path_end_to_end: the load-bearing
one. Wires WorkflowDef → matchers computation → system-task
put() → receive → worker.handleMessage → workflowExecutor.updateTask
with status=COMPLETED. Validates the full task-completion chain.
- receive_failedVerification_throwsAndNoEnqueue: verifier rejects,
NonTransientException thrown, no queue push.
Wires real instances of InMemoryWebhookDAO, InMemoryWebhookTaskService,
TargetWorkflowCollector, WebhookConfigService, IncomingWebhookService,
IncomingWebhookResource, WebhookHashingService, WebhookWorker, and
Webhook. Uses a stub verifier that rejects bodies starting with
"REJECT" to drive the failure path deterministically.
Total tests on this branch now 67 across 9 test classes, 0 failures.
* fix(webhooks-oss): four real bugs found by smoke test against running server
Started server-lite, registered a webhook config, delivered an HMAC-signed
event, watched it dispatch into a workflow that completed. Found and fixed
four issues along the way.
Bug 1: REST routes missing /api prefix
IncomingWebhookResource mounted at /webhook; OSS convention is /api/...
(see RequestMappingConstants.API_PREFIX). Without the prefix, requests
fell through to the static resource handler and returned 500. Fixed:
/webhook -> /api/webhook.
Bug 2: @PathVariable name inference
Bare @PathVariable String id failed at runtime with "parameter name
information not available via reflection. Ensure that the compiler
uses the '-parameters' flag." OSS controllers explicitly name each
binding (see e.g. MetadataResource). Fixed: all @PathVariable in
IncomingWebhookResource and WebhookConfigResource now use explicit
("id") names.
Bug 3: Mutation of stored WebhookConfig on read
WebhookConfigService.getWebhooks() and WebhookConfigResource.getWebhook()
mutated the returned config's secretValue in place to redact it. With
the in-memory DAO that returns shared references, this corrupted the
persisted secret. Next webhook delivery read secretValue=*** and HMAC
verification failed with "Illegal base64 character 2a" trying to decode
the *** placeholder. Fix: WebhookConfig gained toBuilder=true; both
sites now clone-then-redact instead of mutating. Added regression
assertions in WebhookConfigServiceTest and WebhookConfigResourceTest.
Bug 4: server-lite did not depend on webhooks-oss
Only server/build.gradle pulled in :conductor-webhooks-oss. server-lite
is the lighter local-dev target; users running it via :conductor-server-lite:bootRun
would not get the webhooks module loaded at all. Fixed: added the dep
alongside the other system tasks (http-task, json-jq-task, kafka).
Verified end-to-end against running server
1. POST /api/metadata/webhook -> 200, returns config with generated id
2. GET /api/metadata/webhook/{id} -> 200, secret redacted to ***
3. POST /api/metadata/workflow -> 200, register wf-smoke v1
4. POST /api/webhook/{id} with HMAC-SHA256 signed body -> 200
5. WebhookWorker dequeued the event, invoked WorkflowExecutor.startWorkflow,
wf-smoke v1 ran to COMPLETED in 655ms
Tests: 68 across 9 classes, 0 failures.
* fix(webhooks-oss): round 1 ruthless cleanup — license headers, contract bugs, dead code
Adversarial review surfaced three blockers before this PR can flip from
draft to reviewable. All addressed in this commit.
License headers — would have failed spotlessJavaCheck on first CI build
42 files reformatted by spotlessApply. Every "Copyright 2022 Orkes, Inc."
+ "Orkes Enterprise License" header now Apache 2.0 / Conductor Authors.
TimeBasedUUIDGenerator had no header at all — fixed. Import ordering
normalized across the affected modules per repo Spotless config.
WebhookConfigService.updateWebhook NPE on missing id
PUT /api/metadata/webhook/{id} with an id that doesn't exist would
dereference null and 500 instead of 404. Added an explicit
NotFoundException at the top of the method, matching the pattern in
WebhookConfigResource.getWebhook + deleteWebhook. Regression test added.
IncomingWebhookService.handleWebhook 200/null masquerade for unknown id
POST /api/webhook/{id} to an unregistered webhook id returned HTTP 200
with empty body — looked successful to the caller. Now throws
NotFoundException so the global exception mapper returns 404. Updated
WebhooksOssEndToEndTest accordingly.
Dead code
- IncomingWebhookService no longer builds EventMessage objects that
were discarded immediately (left over from the addEventMessage ->
log.warn rewrite). 28 lines removed.
- Dropped unused ExecutionDAOFacade dependency — constructor down from
5 to 4 args.
- Dropped unused EventMessage / DEAD_LETTER_QUEUE imports.
Verified
- :conductor-webhooks-oss:compileJava + compileTestJava: clean
- :conductor-webhooks-oss:spotlessCheck: clean
- :conductor-webhooks-oss:test: 69 tests / 0 failures
* fix(webhooks-oss): round 2 ruthless cleanup — should-fix items from adversarial review
Addresses the should-fix items surfaced by the pre-review adversarial pass.
WebhookTaskMapper relocated for OSS convention
org.conductoross.conductor.tasks.webhook.WebhookTaskMapper ->
org.conductoross.conductor.webhook.tasks.mapper.WebhookTaskMapper
Matches the AI module's precedent (org.conductoross.conductor.ai.tasks.mapper.*)
and collocates the mapper with the rest of the webhook code instead of
splitting it across two top-level packages.
Dead code in TimeBasedUUIDGenerator
Dropped generate(long) and getOrgId(String) — both had TODO stubs with
no OSS callers (verified by grep). Net: 16 lines removed, generate()
gets an @Override annotation, three "OSS-OrkesRequestContext-removed"
TODOs gone. The remaining methods (generate() and getDate(String)) are
the only ones reachable in OSS.
Build.gradle hygiene
- compileOnly 'spring-boot-starter-web' -> implementation: the module's
REST controllers use @RestController/@RequestMapping etc. at runtime,
not just compile time. compileOnly only worked because :conductor-server
happens to pull starter-web transitively; module would be non-functional
if consumed standalone.
- Same upgrade for spring-boot-autoconfigure (used by
@ConditionalOnMissingBean).
- stripe-java 29.4.0 and sendgrid-java 4.10.2 hardcoded -> bound to new
revStripe and revSendgrid in dependencies.gradle. Added a PINNED
comment explaining they're webhook-verifier-specific clients with no
shared rev.
- Dropped now-orphan test deps: spring-web (no longer needed after
IncomingWebhookService dropped HttpHeaders test usage indirectly),
postgres-persistence, testcontainers (postgresql + base), HikariCP,
duplicate spring-retry. The DAO tests these supported moved to
postgres-persistence in the parent split PR.
WebhookWorker.recordHistory off-by-one
if (hist.size() > lastRunWorkflowIdSize) -> >=. Previous form let the
list grow to N+1 before trimming, contradicting the property name.
Missing test coverage
- IncomingWebhookResourceTest (2 tests): delegation + return value
passthrough for both handleWebhook and handlePing endpoints.
- InMemoryWebhookTaskServiceTest (6 tests): put/get hash semantics,
missing matches throws, bucketing of multiple tasks at the same hash,
remove preserves vs drops bucket, unknown hash returns empty.
WebhookConfigServiceTest extended with the NotFoundException-on-missing-id
regression (mirrors the contract fix in round 1).
Verified
- :conductor-webhooks-oss:compileJava + compileTestJava clean
- :conductor-webhooks-oss:spotlessCheck clean
- :conductor-webhooks-oss:test 76 tests / 0 failures
- :conductor-server:compileJava clean
* chore: revert accidentally-committed unrelated files from prior session cruft
* fix(webhooks-oss): correctness — DLQ-friendly worker + matcher recomputation
Two production-impacting bugs from the adversarial review.
WebhookWorker.pollAndExecute no longer acks on failure
Previously a try/finally acked unconditionally, so any exception in
handleMessage (DB blip, OOM, NPE) silently dropped the webhook event.
The TODO comment "retries are not yet modelled" was not a fix.
Now: ack only after handleMessage returns normally. On failure, log
and let the queue's unack timeout redeliver. Poison messages still
get retried, but they hit the underlying QueueDAO impl's retry policy
(e.g. postgres queue's max-retry → dead-letter) rather than vanishing.
InMemoryWebhookDAO.createMatchers no longer caches stale criteria
Previously createMatchers walked the WorkflowDefs at registration
time and stored the extracted `matches` criteria. If a workflow def
was later updated to change its WAIT_FOR_WEBHOOK task's matches
inputParameter, getMatchers kept returning the old criteria until
someone re-PUT the webhook config.
Now: createMatchers stores only the receiverWorkflowNamesToVersions
*targets* (which DO need to be captured at registration time so the
expression-based override is preserved). getMatchers looks up the
WorkflowDefs from MetadataDAO and extracts matches on every call.
Slightly more expensive (one MetadataDAO read per workflow per webhook
event), but always correct.
Tests
- InMemoryWebhookDAOTest gains getMatchers_reflectsWorkflowDefUpdates_noStaleCache:
register matchers, assert criteria, swap the mock to return an updated
WorkflowDef, assert getMatchers returns the new criteria without
re-running createMatchers. Pins the regression.
- 77 tests / 0 failures.
* test(webhooks-oss): rewrite WebhookWorkerTest with real beans + cover pollAndExecute
Per AGENTS.md preference: use real implementations over mocks. The prior
WebhookWorkerTest was 100% mocks. Rewrote to wire real instances of
InMemoryWebhookDAO, InMemoryWebhookTaskService, WebhookHashingService, and
TargetWorkflowCollector. Only the deep infra is still mocked: QueueDAO,
WorkflowExecutor, ExecutionDAOFacade, MetadataDAO, ParametersUtils.
The full register→receive→dispatch flow remains covered by
WebhooksOssEndToEndTest. This class now focuses on worker-internal
semantics that don't surface end-to-end.
Added pollAndExecute coverage
Made pollAndExecute() package-private (was private — same change pattern
already applied to handleMessage for the same reason). Four new tests:
- pollAndExecute_emptyBatch_noop: empty pop, no ack called.
- pollAndExecute_success_acks: happy path acks the message after
handleMessage returns cleanly.
- pollAndExecute_handleMessageThrows_doesNotAck: REGRESSION TEST. Prior
impl had a try/finally that acked even on failure, silently dropping
webhook events on any exception. Test uses an anonymous InMemoryWebhookDAO
subclass that throws from getWebhookEvent to simulate a DB blip.
- pollAndExecute_mixedBatch_acksOnlySuccesses: batch with one good +
one poison message. Only the good one gets acked; the poison one is
left for the queue's unack timeout to redeliver.
Existing handleMessage tests rewired with real beans
- handleMessage_eventNotFound_returnsEarly: stores nothing, calls
handleMessage, expects no executor invocation.
- handleMessage_configNotFound_returnsEarly: stores event but not config;
asserts the event is NOT removed (so retry can happen).
- handleMessage_workflowsToStart_invokesExecutor: registers config with
workflowsToStart, asserts StartWorkflowInput contents (name, version,
input merge, event name) and that the event is removed after success.
- handleMessage_workflowsToStart_nonIntegerVersion_skipped: version is a
string, no executor call.
- handleMessage_matcherHit_completesWaitingTask: registers config + uses
real WebhookTaskService.put to register a waiting task; asserts that
webhookTaskService.get returns empty after handleMessage completes.
- handleMessage_matcherHit_terminalTask_skipped: pre-COMPLETED task is
not re-completed.
Total: 81 tests across 11 classes, 0 failures.
* docs(webhooks-oss): WAIT_FOR_WEBHOOK task + REST surface
Per project CLAUDE.md, new user-facing surfaces need source-derived docs.
The webhooks-oss module added /api/metadata/webhook (config CRUD) and
/api/webhook/{id} (event receive) without any documentation; users had no
on-ramp.
Adds docs/.../systemtasks/wait-for-webhook-task.md following the existing
system task doc pattern (wait-task.md template). Contents:
- Task type + how the dispatch chain works (mapper, hash, worker)
- inputParameters table (matches)
- WebhookConfig registration: endpoint, body, verifier types table
(HMAC_BASED, SIGNATURE_BASED, HEADER_BASED, SLACK_BASED, STRIPE,
TWITTER, SENDGRID — pulled from the actual Verifier enum)
- Delivery endpoint: status codes, response semantics
- Verified end-to-end curl example: same flow exercised during the
smoke test (register config, register workflow def, deliver HMAC-signed
event)
- conductor.webhook.worker.* properties table (pulled from
WebhookWorkerProperties.java)
- Failure semantics: verification failure / worker dispatch failure /
rejected event logging — matches the round-3 ack-only-on-success
worker change.
Registered in mkdocs.yml nav alongside wait-task.md.
* docs(webhooks-oss): restore Javadoc on WEBHOOK_QUEUE constant for Orkes parity
* fix(webhooks-oss): log verification failures at ERROR to match Orkes
* test(webhooks-oss): property test for registration/inbound hash agreement
Random (workflow, version, taskRef, matches, body) tuples assert the two
hash sites in webhooks-oss canonicalize identically: InMemoryWebhookTaskService
(registration) and WebhookHashingService.computeJsonHash (inbound). Catches
silent divergence that would cause inbound events to miss registered tasks.
100 reps per direction (match / no-match). Seed defaults to System.nanoTime();
on failure the TestWatcher prints -Dwebhook.hash.seed=<n> for reproducibility.
* fix(webhooks-oss): guard StripeVerifier against null api_version
Stripe events whose body omits api_version (older API payloads, synthetic
test payloads, and apparently some real Stripe deliveries) NPE on
rawApiVersion.split(...). The signature has already been verified at this
point, and both branches of the apiVersion check ultimately return
ErrorList.empty(), so a missing api_version is treated as "old api / no
deserializer support" — same as the dated pre-deserializer path.
Surfaced by the smoke harness firing synthetic Stripe-shaped bodies; saw
500 NPE in the verifier path. Adds StripeVerifierTest (3 cases): null
api_version accepted, bad signature rejected, missing header rejected.
Orkes has identical code in io.orkes.conductor.webhook.verifier.StripeVerifier
— mirror fix needed upstream to keep parity.
* feat(postgres-persistence): PostgresWebhookDAO + PostgresWebhookTaskService
Production multi-node backing for the WebhookDAO + WebhookTaskService
interfaces. Designed clean for OSS (no orgId anywhere in the schema or
queries) and applies the matcher-cache fix from the parent PR's Round 3:
target workflows are persisted, match criteria are recomputed from
MetadataDAO on every getMatchers() call so WorkflowDef updates take
effect without re-registering the webhook.
Stacked PR — base is feat/webhooks-from-orkes-split. Don't merge until
the parent (#1106) lands. After parent merges this rebases onto main.
Added
- V16__webhook.sql: four tables — webhook, incoming_webhook_event,
webhook_target_workflows, webhook_hash_to_taskid. Renamed
webhook_matchers -> webhook_target_workflows to reflect that we
store target workflow ids (override snapshot), NOT pre-computed
matchers. Index on incoming_webhook_event(created_on) for the
cleanup follow-up.
- org.conductoross.conductor.postgres.dao.PostgresWebhookDAO:
9 interface methods + private computeMatchers helper. Extends
PostgresBaseDAO. ON CONFLICT DO UPDATE for upserts.
- org.conductoross.conductor.postgres.dao.PostgresWebhookTaskService:
3 interface methods (put/get/remove). ON CONFLICT DO NOTHING on
put for idempotency.
- PostgresWebhookDAOTest (8 tests with testcontainers + @MockBean MetadataDAO),
PostgresWebhookTaskServiceTest (6 tests with testcontainers).
Critical regression test: getMatchers_recomputesFromMetadataDAO_onEachCall_noStaleCache.
Shared hash extraction
- WebhookTaskHashing helper added to core
(org.conductoross.conductor.service.webhook). Static methods that
both InMemory and Postgres impls use, so tasks registered via one
backing are findable by hash via any other. Validates matches
before composing the hash (regression fix in this commit: NPE in
removeIterationFromTaskRefName would mask the missing-matches
exception).
- InMemoryWebhookTaskService refactored to delegate to the helper
(-50 LoC). Test surface unchanged.
PostgresConfiguration wiring
- Two new @Bean methods with @DependsOn({"flywayForPrimaryDb"}).
Bean names "webhookDAO" and "webhookTaskService" match the
@ConditionalOnMissingBean(name=...) on the InMemory impls, so the
postgres beans take precedence when this module is on the classpath.
Always-on for now (no @ConditionalOnProperty); the whole
postgres-persistence module is gated by conductor.db.type=postgres
at the upstream configuration.
OSS-side design changes vs orkes' PostgresWebhookDAO
- All org_id columns and WHERE clauses dropped. Tables are tenant-free.
- ConductorLimitsDAO/LimitUtils quota check dropped (Orkes-only).
- Preconditions.checkNotNull replaced with NotFoundException where
appropriate; nulls otherwise propagate per OSS convention.
- Scheduled refreshMatchers executor dropped — matcher staleness is
fixed by design (recompute on read).
- workflowDefToUpdateMap cache dropped — same reason.
- The "matches" persistence (webhook_matchers table in orkes)
replaced with "targets" persistence (webhook_target_workflows).
Different shape, same external contract.
Verified locally
- :conductor-postgres-persistence:compileJava + compileTestJava clean
- :conductor-postgres-persistence:spotlessCheck clean
- :conductor-webhooks-oss:test green (81 tests, hashing refactor
doesn't break anything)
- :conductor-server:compileJava clean
Note: postgres tests need Docker daemon for testcontainers; not run
locally. CI will validate.
* fix(postgres-persistence): tolerate orphan migrations in webhook DAO tests
CI 'build' job on #1110 failed with FlywayValidateException:
Detected applied migration not resolved locally: 10.1
V10.1__notify.sql lives in db/migration_postgres_notify/, which
PostgresConfiguration only loads when
conductor.postgres.experimental-queue-notify=true. When another test
class in the same Gradle test executor JVM enables that flag, V10.1
gets applied to the shared testcontainers postgres. When our webhook
tests run without that flag, Flyway validates schema_history against
our (smaller) resolved-migration list and refuses to migrate.
Fix: add spring.flyway.ignore-migration-patterns=*:missing to both
PostgresWebhookDAOTest and PostgresWebhookTaskServiceTest. This tells
Flyway it's OK for the DB to contain applied migrations that aren't in
the current location set — the validation behavior we actually want for
test contexts that share a postgres container across configurations.
The before() @Before hook still flyway.clean()s the DB between tests so
schema state stays predictable within a class.
Local: still passes spotlessCheck; can't run tests (no Docker on dev
box). CI will validate.
* feat(persistence): WebhookDAO + WebhookTaskService impls for SQLite and MySQL
Adds the same in-DB shape we landed for postgres to the two other SQL
backends OSS conductor supports. Cassandra and Redis are deferred to a
follow-up — they need different data modeling (not SQL).
Pattern is identical across backends, dialect specifics aside:
SQLite (sqlite-persistence)
- V4__webhook.sql: 4 tables matching the postgres layout (no org_id)
- SqliteWebhookDAO + SqliteWebhookTaskService extending SqliteBaseDAO
- Uses SQLite's INSERT ... ON CONFLICT (col) DO UPDATE SET col = excluded.col
- Bean wiring in SqliteConfiguration with names webhookDAO + webhookTaskService
MySQL (mysql-persistence)
- V10__webhook.sql: same 4 tables with InnoDB / utf8mb4 + ON UPDATE
CURRENT_TIMESTAMP for modified_on
- MySQLWebhookDAO + MySQLWebhookTaskService extending MySQLBaseDAO
- Uses MySQL's INSERT ... ON DUPLICATE KEY UPDATE col = VALUES(col)
and INSERT IGNORE for the conflict-do-nothing case
- Bean wiring in MySQLConfiguration with the same bean names
Bean name collision is intentional: webhookDAO and webhookTaskService
are the names the InMemory impls' @ConditionalOnMissingBean checks for
in webhooks-oss, so the SQL impls take precedence whenever their
persistence module is on the classpath.
Shared
- All three SQL impls use WebhookTaskHashing from core so the hash a
task is stored under is identical across backings.
- Same matcher-recomputation-on-read design as PostgresWebhookDAO —
target workflows persisted, matchers computed from MetadataDAO on
every getMatchers() call (no stale-cache).
Tests
- SqliteWebhookTaskServiceTest (6) + SqliteWebhookDAOTest (7) — run
against a local SQLite file (no Docker). All 13 pass locally.
Notable per-test workflow names in the task service test to avoid
hash collisions across the shared SQLite file.
- MySQLWebhookTaskServiceTest (6) + MySQLWebhookDAOTest (6) — same
shape; uses jdbc:tc:mysql:8.0.29 testcontainers. CI will validate.
Not run locally — needs Docker.
Known follow-ups (separate PRs)
- Redis adapter (different data model — Redis hashes / sorted sets)
- Cassandra adapter (denormalized CQL)
- WebhookCleanupJob for retention of incoming_webhook_event rows
* feat(redis-persistence): WebhookDAO + WebhookTaskService impls for Redis
Adds the Redis backing. Different paradigm than the SQL impls (no
schema, no SQL) but the same external behavior including matcher
recomputation on read.
Data model
- nsKey("WEBHOOK_CONFIG") hash: webhook_id -> JSON(WebhookConfig)
- nsKey("WEBHOOK_EVENT") hash: event_id -> JSON(IncomingWebhookEvent)
- nsKey("WEBHOOK_TARGETS") hash: webhook_id -> JSON(Map<workflowName,version>)
- nsKey("WEBHOOK_HASH", hash) set: members are task_ids waiting on
that hash
Single hash per concept rather than per-id keys keeps getAllWebhooks()
fast (HVALS) and avoids polluting the Redis keyspace.
OSS pattern alignment
- Both classes extend BaseDynoDAO (gives jedisProxy, nsKey, toJson,
readValue).
- @Component(value="...") matches the bean name the InMemory impls
guard against via @ConditionalOnMissingBean — so Redis wins when its
module is on the classpath and conductor.db.type is set to a Redis
flavor.
- @Conditional(AnyRedisCondition.class) — AnyNestedCondition that
activates for db type memory, redis_cluster, redis_sentinel, or
redis_standalone. Same gate RedisFileMetadataDAO uses.
- WebhookTaskHashing is shared with the SQL impls so the hash a task
is stored under is identical across backings.
Tests
- RedisWebhookTaskServiceTest (6) + RedisWebhookDAOTest (7) — both
use the existing pattern of GenericContainer("redis:7-alpine") +
JedisPool + flushAll() between tests (mirrors RedisMetadataDAOTest).
- @MockBean MetadataDAO in the DAO test to drive the
recomputes-on-read regression.
Local: compiles + spotless clean. Tests not run locally (need Docker).
CI will validate.
* feat(cassandra-persistence): WebhookDAO + WebhookTaskService impls for Cassandra
Final SQL/NoSQL backend for the matching-scheduler-precedent series.
Cassandra-flavored impls of WebhookDAO + WebhookTaskService, following
the org.conductoross.conductor.cassandra.dao.* layout the scheduler
cassandra port uses.
Data model
- webhook (bucket, webhook_id, json_data) with composite PK
((bucket), webhook_id). Single ALL bucket so getAllWebhooks() is a
single-partition scan. Anti-pattern at high cardinality, fine for
admin-scale config lists.
- incoming_webhook_event (event_id PRIMARY KEY, json_data) — high
cardinality, no listing needed.
- webhook_target_workflows (webhook_id PRIMARY KEY, json_data) —
target workflow versions snapshot per webhook. Matchers themselves
are recomputed from MetadataDAO on read (same design as the SQL
backings — no stale cache).
- webhook_hash_to_taskid (hash, task_id) with composite PK
((hash), task_id) — efficient get(hash) via single-partition scan,
efficient point delete by (hash, task_id).
Tables created on construction via ensureTables() — mirrors the
scheduler-cassandra pattern. PreparedStatements built once at
construction, bound per call.
Shared
- WebhookTaskHashing from core — hashes match across backings.
- Local toJson/readValue helpers because the ones in CassandraBaseDAO
are package-private. Same workaround the scheduler-cassandra DAO
applies (and that the file says explicitly: 'objectMapper is private
in CassandraBaseDAO; keep a local reference for serialization').
Tests
- CassandraWebhookDAOTest (7) + CassandraWebhookTaskServiceTest (6)
using testcontainers CassandraContainer('cassandra:3.11.2') +
@ClassRule. Drop tables in @Before for test isolation. Mirrors
CassandraSchedulerDAOTest setup verbatim.
Local: compiles + spotless clean. Tests not run locally (need Docker).
CI will validate.
Matching-scheduler-precedent series complete
All 5 OSS persistence backends now have WebhookDAO + WebhookTaskService
impls: postgres, mysql, sqlite, redis, cassandra. Same external
contract, backing-specific data model.
Still on the deferred list: WebhookCleanupJob for retention of
incoming_webhook_event rows.
* feat(postgres-persistence): PostgresWebhookCleanupJob — scheduled retention for incoming_webhook_event
Periodically deletes old rows from the incoming_webhook_event table so it
doesn't grow unbounded. Postgres-only for now — the other backends will
get their own impls (or rely on Cassandra TTL / Redis per-key expiry) in
follow-up PRs.
Behavior
- @Scheduled cron defaults to hourly ("0 0 * * * *")
- Default retention: 7 days
- Default batch size: 1000 rows per pass
- Default max-runtime: 60s per tick
- Properties: conductor.webhooks.cleanup.{cron,retention-duration,batch-size,max-runtime,enabled}
- enabled defaults to true; users can disable with conductor.webhooks.cleanup.enabled=false
SQL: WITH old_records AS (SELECT ctid WHERE created_on < ? LIMIT ?),
deleted AS (DELETE WHERE ctid IN (...) RETURNING ctid) SELECT count(1)
FROM deleted. CTE + RETURNING keeps each batch's transaction small and
cancellable.
Wired in PostgresConfiguration as a @Bean (not @Component) so it only
loads when conductor.db.type=postgres. The @ConditionalOnProperty for
the enabled flag gates loading further. @EnableScheduling is already
declared in core/SchedulerConfiguration so the @Scheduled annotation
fires once the bean is registered.
Tests
- PostgresWebhookCleanupJobTest (4 tests with testcontainers postgres):
- deletes rows older than retention, keeps recent ones
- empty table is a no-op (no exception)
- all-recent table keeps everything
- batched delete spans multiple passes correctly (batch=2, 7 rows,
forces 4 passes)
Local: compiles + spotless clean. Tests not run locally (need Docker).
CI will validate.
* feat(persistence): MySQL + SQLite WebhookCleanupJob siblings
Completes scheduled retention of incoming_webhook_event rows across all
SQL backings (after PostgresWebhookCleanupJob from the prior commit).
MySQL impl (MySQLWebhookCleanupJob)
- MySQL doesn't support DELETE ... RETURNING the way postgres does, so
uses DELETE FROM ... WHERE created_on < ? LIMIT ? and relies on
PreparedStatement.executeUpdate() return value to know when to stop.
SQLite impl (SqliteWebhookCleanupJob)
- SQLite's DELETE ... LIMIT is only available when compiled with
SQLITE_ENABLE_UPDATE_DELETE_LIMIT (not the default in many distros),
so uses a rowid subquery: DELETE FROM t WHERE rowid IN
(SELECT rowid FROM t WHERE created_on < ? LIMIT ?).
Both follow the postgres impl's contract:
- Default retention: 7 days
- Default cron: hourly
- Default batch size: 1000
- Default max-runtime: 60s per tick
- Properties: conductor.webhooks.cleanup.{cron,retention-duration,
batch-size,max-runtime,enabled}
Wired as @Bean in MySQLConfiguration and SqliteConfiguration with the
same @ConditionalOnProperty(enabled, matchIfMissing=true) gate.
Tests
- MySQLWebhookCleanupJobTest (3) — uses testcontainers MySQL. Not run
locally; CI will validate.
- SqliteWebhookCleanupJobTest (3) — uses local sqlite file; passes
locally (3/3). Required an explicit DELETE FROM in @Before because
flyway.clean() leaks rows across tests on the shared sqlite file.
Cassandra and Redis cleanup are deferred to follow-ups — both have
different idioms (Cassandra TTL on table or per-row, Redis per-key
expiry or hash field walker) that warrant their own design pass.
* feat(persistence): Cassandra TTL + Redis cleanup job — completes retention coverage
All 5 OSS persistence backends now have a retention mechanism for
incoming_webhook_event:
Cassandra (CassandraWebhookDAO)
- Adds default_time_to_live to incoming_webhook_event at CREATE TABLE
time. Cassandra auto-expires rows after the configured window — no
scheduled job needed, no tombstone churn beyond what the SSTable
layer already manages.
- Default TTL: 7 days (matches the SQL cleanup-job default).
- Configurable via conductor.webhooks.cleanup.retention-duration on
the CassandraConfiguration @Bean wiring (Duration → toSeconds for
default_time_to_live).
- Caveat documented inline: CREATE TABLE IF NOT EXISTS won't update an
existing table's TTL; operators changing retention on an existing
deployment need to ALTER TABLE manually.
- The other 3 tables (webhook, webhook_target_workflows,
webhook_hash_to_taskid) don't get TTL — they're durable config /
task-routing state, not auditable events.
Redis (RedisWebhookCleanupJob)
- @Component @Conditional(AnyRedisCondition.class)
@ConditionalOnProperty(conductor.webhooks.cleanup.enabled,
matchIfMissing=true).
- @Scheduled walker on the same cron as the SQL backings: HGETALL on
the WEBHOOK_EVENT hash, parse each value as IncomingWebhookEvent,
HDEL any whose timeStamp is older than retentionDuration.
- Cheap at low cardinality; documented that high-volume deployments
should migrate to per-key TTL storage (HEXPIRE landed in Redis 7.4
but isn't ubiquitous yet).
- 4 tests with testcontainers Redis: deletes old, keeps recent, empty
hash no-op, all-recent kept, timeStamp=0 fixtures preserved.
Verified locally: full compile clean across cassandra/redis modules
and server. Tests not run locally — both need Docker. CI will validate.
Deferred items now all addressed:
- ~~Worker DLQ semantics~~ (in #1106)
- ~~Matcher cache staleness~~ (in #1106 + #1110)
- ~~Postgres impls~~ (#1110)
- ~~MySQL/SQLite/Redis/Cassandra adapters~~ (#1110)
- ~~WebhookCleanupJob (postgres)~~ (#1110)
- ~~MySQL + SQLite cleanup~~ (#1110)
- ~~Cassandra TTL + Redis cleanup~~ (this commit)
Remaining open: EventMessage audit-table persistence (the addEventMessage
→ log.warn debt) — separate, larger architectural decision.
* refactor(webhooks): extract WebhookMatcherComputer to core
Lifts the identical `computeMatchers` body from all 6 WebhookDAO impls
(InMemory + Postgres/MySQL/SQLite/Redis/Cassandra) into a single static
helper in core. Each impl's getMatchers() collapses to a one-liner that
loads the target-workflow snapshot and delegates the WorkflowDef →
matchers transformation.
Behavior identical — the loop body is lifted verbatim. Net –166 LoC.
Same parity-of-behavior argument as WebhookTaskHashing: a WorkflowDef
edit now produces identical matcher output regardless of which
persistence module is backing WebhookDAO, enforced by a shared call
rather than five hand-maintained copies.
* fix(webhook/verifier): stop leaking Stripe signing secret in error response
The 'header is not present' error path concatenated webhookConfig.getSecretValue()
into the response body. Renames the local to signingSecret and uses the literal
STRIPE_SIGNATURE constant in the error message.
* fix(webhook/verifier): guard SignatureBasedVerifier against short header
substring(SHA_256.length()) on a header that doesn't start with 'sha256='
threw an uncaught StringIndexOutOfBoundsException, surfacing as HTTP 500.
Validate the prefix and return a clean ErrorList instead.
* fix(webhook/verifier): use constant-time signature comparison
String.equals on HMAC signatures short-circuits on the first differing byte,
which leaks signature bytes to a network-adjacent attacker with a stopwatch.
Switch to MessageDigest.isEqual in both HMACVerifier and SignatureBasedVerifier.
* fix(webhook): drop signature/payload material from debug logs
HMACVerifier and SignatureBasedVerifier logged both signatures at debug level,
and IncomingWebhookResource logged the full request body, params, and headers map.
With DEBUG enabled in any deployment those become live secret/PII channels — log
only the webhook id and sizes.
* fix(webhook): null-check verifier lookup before invocation
The verifier name comes from a deserialized config; the read path doesn't
re-validate against the live enum. A drift between persisted verifier strings
and registered @Component beans NPE'd. Reject with a clean error instead.
* fix(webhook): drop @SneakyThrows from handlePing
handlePing has no checked-exception surface — verifier.handlePing() declares
none — so the annotation was vestigial obfuscation. Removed it and applied the
same null-verifier guard that handleWebhook now uses for symmetry.
* fix(webhook): validate matches map key types before downstream use
Replaces the @SuppressWarnings(unchecked) cast with a key-type check, copying
into a typed Map<String, Object> when valid and skipping the entry otherwise.
A non-String key now surfaces as a skipped matcher, not a downstream CCE.
* docs(webhook): document MetadataDAO fetch cost on the matcher hot path
The compute() loop issues one getWorkflowDef per target workflow on every read.
Cassandra wraps with CacheableMetadataDAO, but postgres/mysql/sqlite/redis hit
the backend each time — acceptable for small fanout, revisit if matchers fan out.
* docs(webhook/migrations): clarify webhook_hash_to_taskid column intent
The 'hash' column actually holds a delimited deterministic key, not a SHA-256
hash — TEXT in postgres/sqlite is intentional, but MySQL's VARCHAR(255) is at
latent truncation risk on long keys. Documented and flagged for follow-up.
* docs(webhook): document auth model for /api/metadata/webhook endpoints
The webhook config endpoints follow conductor OSS's standard 'no auth in the
server, secure via deployment wrapper' pattern. Added a deployment section
that calls out the two surfaces (public event vs operator metadata) explicitly.
* fix(webhook/verifier): implement Slack's v0= signing-secret protocol
Replaces the urlVerified-only check (which became a no-op after the first
successful event) with the real Slack v0=hex(hmac-sha256(timestamp:body))
verification, plus a 5-min replay-tolerance window. Tests rewritten to
exercise the signed protocol end-to-end.
* feat(webhook/cleanup): cluster-safe cleanup lease across cleanup jobs
Without a lease every replica ran the cleanup cron simultaneously, hammering
the incoming_webhook_event table. SQL backends gain a webhook_cleanup_lease
table acquired via a single conditional UPDATE; Redis uses SET NX PX.
* feat(webhook): signature-dedup replay protection across verifiers + DAOs
Verifiers expose dedupKey() (signature header value); IncomingWebhookService
records it via webhookDAO.tryRecordSignature() after verify() and rejects
duplicates within REPLAY_DEDUP_TTL (5 min). Cassandra retains the default
no-op pending an IF-NOT-EXISTS + TTL impl in a follow-up.
* fix(cassandra/webhook): qualify prepared statements with keyspace
CassandraWebhookDAO and CassandraWebhookTaskService created tables as
<keyspace>.<table> in ensureTables() but their session.prepare() calls
referenced bare table names. With a session not bound to a keyspace via
USE, prepare fails with "No keyspace has been specified" and every test
in CassandraWebhookDAOTest / CassandraWebhookTaskServiceTest aborts at
setUp. Mirror the qualification ensureTables() already does.
* fix(persistence): spring-web on cassandra+redis test classpath
IncomingWebhookEvent.headers is typed as org.springframework.http.HttpHeaders
(mirroring Orkes). Jackson reflects over all declared fields during
serialization, so a NoClassDefFoundError fires whenever a persistence test
serializes the event and spring-web isn't on its test runtime classpath.
postgres/mysql/sqlite already pull spring-web transitively via
:conductor-server testImpl; cassandra and redis only have spring-boot-starter
compileOnly which doesn't include spring-web. Add spring-web as
testImplementation in those two modules.
* style: apply spotless formatting to webhook DAO and matcher computer
* style: apply spotless formatting across webhook modules
* fix(persistence/test): commit inserts in MySQL+Postgres cleanup tests
src/test/resources/application.properties sets
spring.datasource.hikari.auto-commit=false in both modules. The new
MySQLWebhookCleanupJobTest and PostgresWebhookCleanupJobTest insert seed
rows via raw JDBC (dataSource.getConnection() + executeUpdate()) and
let the try-with-resources close the connection without committing —
so the inserts roll back when the connection returns to the pool and
the precondition assertEquals(N, rowCount()) sees 0.
Add an explicit conn.commit() after each insertEvent() executeUpdate.
The cleanup job itself already calls setAutoCommit(true) on its own
connection so its DELETEs are unaffected.
* fix(postgres/test): include notify migration location in webhook tests
The Postgres webhook tests set spring.flyway.ignore-migration-patterns
in @TestPropertySource, but PostgresConfiguration builds its Flyway bean
via Flyway.configure() directly rather than via Spring's auto-config,
so that property has no effect.
Other tests in this module (PostgresQueueListenerTest,
PostgresGrpcEndToEndTest) set experimentalQueueNotify=true and apply
V10.1 from migration_postgres_notify to the shared testcontainer DB.
When the new webhook tests run after them in the same JVM, their
Flyway has only migration_postgres on its locations, sees V10.1 in
flyway_schema_history, and aborts context creation with "Detected
applied migration not resolved locally: 10.1" — taking all 18 tests
in the three webhook test classes down with it.
Switch the webhook tests to also set experimentalQueueNotify=true so
their Flyway includes the notify location and validation passes
regardless of test ordering. The notify triggers only fire on
queue_message writes, which these tests never do, so there's no
behavioral impact.
* fix(postgres/test): truncate tables instead of flyway.clean()
PostgresConfiguration builds its Flyway bean via Flyway.configure()
directly rather than via Spring's auto-config, so neither
spring.flyway.clean-disabled=false nor any other spring.flyway.* test
property applies to it. Flyway defaults to cleanDisabled=true, so the
@Before's flyway.clean() throws FlywayException across all 18 webhook
tests.
Drop flyway.clean()/migrate() and truncate the relevant tables
directly in each test class:
- PostgresWebhookCleanupJobTest: truncate incoming_webhook_event +
reset the webhook_cleanup_lease row (tryAcquireLease updates but
never releases — second test in this class needs the lease back).
- PostgresWebhookDAOTest: truncate webhook, incoming_webhook_event,
webhook_target_workflows.
- PostgresWebhookTaskServiceTest: truncate webhook_hash_to_taskid.
PostgresQueueListenerTest in this module uses the same TRUNCATE pattern
for the same reason.
* fix(sqlite/test): reset cleanup lease row in @Before
SqliteWebhookCleanupJobTest already worked around flyway.clean()'s
incomplete reset on the shared sqlite file by adding an explicit
DELETE FROM incoming_webhook_event in @Before, but missed the
webhook_cleanup_lease row. V5__webhook_cleanup_lease.sql seeds the
row with INSERT OR IGNORE, so re-running flyway.migrate() after a
partial-clean is a no-op and the prior test's expires_at = now + 5min
sticks. The first test acquires the lease and the next two tests'
tryAcquireLease() returns false, so job.run() exits early and
nothing is deleted — failing the post-cleanup row-count assertions.
Add an explicit UPDATE of the lease row alongside the existing
incoming_webhook_event delete so every test starts with the lease
available.
* test(sendgrid): add SendGridVerifierTest — missing headers, bad key, sig pass/fail
* test(incoming-webhook): add IncomingWebhookServiceTest — all handleWebhook + handlePing branches
* test(webhook-worker): add non-Map body and recordHistory trim-when-full tests
* test(webhook-hashing): add WebhookHashingServiceTest — non-Map body, empty array, malformed JSON
* refactor(test): replace mocks with real in-memory impls — IncomingWebhookServiceTest (0 mocks), WebhookWorkerTest (2 mocks from 5)
* fix(sqlite/test): reset cleanup lease with parameterized Timestamp
The previous attempt set webhook_cleanup_lease.expires_at via a text
literal ('1970-01-01T00:00:00'), but the xerial sqlite-jdbc driver
stores java.sql.Timestamp as INTEGER (epoch millis) and SQLite's type
comparison rule says INTEGER < TEXT — so the cleanup job's
"expires_at < ?" check (with ? bound as a Timestamp/INTEGER) compares
TEXT > INTEGER → false, and the lease could never be re-acquired.
Switch to INSERT OR REPLACE with the expires_at value bound via
ps.setTimestamp() so it lands as INTEGER in the same format the
cleanup job's comparison uses.
Note: V5__webhook_cleanup_lease.sql seeds the row with the same broken
text literal, so a single-instance SQLite deployment running the
cleanup job for the first time would also fail to acquire its lease.
Follow-up needed on the V5 migration itself, but out of scope here.
* fix(sqlite/webhook): seed cleanup lease expires_at as INTEGER
Closes #1142
V5 seeded webhook_cleanup_lease.expires_at with a text literal
('1970-01-01T00:00:00'). The xerial sqlite-jdbc driver stores
java.sql.Timestamp as INTEGER (epoch millis), and SQLite's type-
comparison rule says INTEGER < TEXT — so SqliteWebhookCleanupJob's
"expires_at < ?" check (with ? bound as a Timestamp) was always
false on a freshly migrated DB. The cleanup job would log "lease
held elsewhere" forever and never delete an incoming_webhook_event
row.
V5 hasn't shipped to main yet, so fix the seed value in place
rather than adding a patch migration. Use INTEGER 0 so the seed's
storage class matches what the cleanup job writes via setTimestamp().
The test's @Before lease reset (added earlier in this PR for between-
test isolation, since flyway.clean() leaks rows on the shared sqlite
file) remains necessary — it covers a different concern. Trim its
comments to drop the now-stale "V5 seed is broken" wording.
Postgres and MySQL aren't affected: their native TIMESTAMP types parse
the text literal as a real timestamp. No Orkes parallel — sqlite-
persistence and the cleanup-lease pattern are both OSS-side additions.
* style: apply spotless to webhooks-oss test files
* remove Tag.java from common — tags are enterprise RBAC, no OSS backing
Tag.java was introduced by the webhooks-oss port but serves no purpose in OSS:
tags in Orkes power TagsService/AutomaticTagService-based access control, neither
of which exists here. The class had no callers other than WebhookConfig.tags.
* drop tags field from WebhookConfig — never populated in OSS
In Orkes, WebhookConfig.tags is populated at query-time by TagsService from
a separate enterprise tags table. OSS has no TagsService, so the field was
always null. Dead API surface that would mislead users.
* remove EventMessage.java — ported but never wired, no OSS backing
EventMessage was added to common to mirror Orkes' webhook event audit trail,
but OSS has no ExecutionDAO.addEventMessage() and IncomingWebhookService
never references it. Zero callers anywhere in OSS — pure dead code.
The event audit feature (DLQ persistence for rejected/unmatched webhook events)
requires a separate architectural decision before it belongs here.
* remove WebhookExecutionHistory — enterprise-only, both sides marked TODO Remove
WebhookExecutionHistory embeds a rolling execution log inside WebhookConfig
and is only implemented in orkes-conductor/webhooks-enterprise. Both the Orkes
and OSS models carried a '// TODO Remove this' comment. The recordHistory()
call also held the only webhookDAO.createWebhook() write-back after dispatch,
but urlVerified is already persisted by IncomingWebhookService before the event
is enqueued, so that write was redundant. Drops lastRunWorkflowIdSize property
and its associated test.
* drop WebhookConfig.getWorkflowNames() — zero callers, duplicates map getter
Method was @JsonIgnore and only extracted keys from receiverWorkflowNamesToVersions,
which is already accessible directly. No call site exists anywhere in OSS.
* map NonTransientException → 400 in ApplicationExceptionMapper
NonTransientException previously fell through to the default 500, which
told webhook senders (Stripe, GitHub, Slack) that the server was broken
and to retry. Signature verification failures are client errors — the
sender's signature is wrong — and must return 4xx so senders stop
retrying.
NonTransientException by semantics ("this won't work no matter how many
times you retry") maps cleanly to 400 Bad Request.
Verified: bad_signature and replay cases in negative_smoke.py now return
400 instead of 500, 5/5 negative cases passing on enki.
* fix(webhook): SREM correlation set on task cancel to stop orphan-set growth
Ports orkes-io/orkes-conductor#3663. Webhook.start() does a put() into the
WAIT_FOR_WEBHOOK hash set; the only removal path was WebhookWorker.handleEvent
firing SREM on a matching event. Workflows terminated before a match arrived
left their taskId in the set indefinitely — Redis memory leak / Postgres row
accumulation in webhook_hash_to_taskid.
Fix: add Webhook.cancel() that calls webhookTaskService.remove(TaskModel, int)
on all six backings (postgres, mysql, redis, cassandra, sqlite, in-memory).
WebhookTaskHashing.computeHashIfPresent() handles tasks cancelled before
IN_PROGRESS (null/missing matches) as a safe no-op instead of throwing.
Cleanup failures are caught and logged so the terminate path is never blocked.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Viren Baraiya
|
5cf15d6a4e | Revert "MCP for Workflow" (#1137) | ||
|
kowser
|
1d398c2b30 | MCP for Workflow | ||
|
Viren Baraiya
|
6ceb9208b4 | resolve ${...} expressions in subWorkflowParam.workflowDefinition at … (#1068) | ||
|
Kowser
|
061b61673a | Documentation for file storage feature (#1062) | ||
|
kowser-orkes
|
09aa35e196 | Initial implementation for file storage feature | ||
|
Viren Baraiya
|
6456b564fa | Improve task retry policy (#1031) | ||
|
Naomi Most
|
1ca7ba5c4e |
fix: slow GET /metadata/workflow — Redis cache + UI switch to names-and-versions (#844) (#974)
* fix: cache workflow defs in Redis to fix slow GET /metadata/workflow (#844) getAllWorkflowDefs() previously issued N+1 serial Redis round-trips on every request (1 SMEMBERS + 1 HGETALL per workflow name), returning full definition bodies for every version. Under load with hundreds of workflow definitions this caused multi-second response times; under concurrent load, requests backed up into tens-of-seconds latency and ~30% HTTP 500s. Reproduction: 2000 workflows x 5 versions on localhost Redis - Before: ~3s serial, 5-70s under concurrent load, ~30% HTTP 500s - After: sub-millisecond (cache hit) Fix mirrors the existing taskDefCache pattern: - Add in-memory workflowDefCache (volatile List<WorkflowDef>) - getAllWorkflowDefs() returns a copy of the cache - getAllWorkflowDefsLatestVersions() derives latest-per-name from cache (O(n) in-memory instead of N Redis round-trips per workflow name) - Cache is refreshed synchronously after every create/update/remove so writes are immediately visible to subsequent reads - Periodic background refresh uses existing metadataCacheRefreshInterval property (default 60s) - Named executor threads + @PreDestroy shutdown for proper lifecycle - Fixed pre-existing: taskDefCache was not declared volatile Tests: two new tests verify the cache is updated after create/remove and that getAllWorkflowDefsLatestVersions correctly picks highest version. Closes #844 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * ui : using /metadata/workflow/names-and-versions instead of /metadata/workflow for better efficency * test: add updateTime assertion and clean up dead instanceof Map branches * fix: make workflow def cache opt-in, consolidate cache refresh executors * fix: defensive copy in getAllWorkflowDefsLatestVersions, clean shutdown, remove dead Array.from branch * style: spotless formatting fix * fix: update Cypress intercept and fixture for names-and-versions endpoint --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: kousik.sv <kousikmahadeva.s@freshworks.com> |
||
|
Viren Baraiya
|
fec4f90e20 | feat: wildcard search and sub-workflow filtering (#996) | ||
|
Miguel Prieto
|
e7659e6ae4 | feat: Workflow Message Queue (WMQ) — push messages into running workflows (#982) | ||
|
Viren Baraiya
|
4cb030bf08 | Refactor redis and upgrade jedis (#927) | ||
|
Akhil Pathivada
|
ccc822a8dc | feat: Add TaskDef-level control for task status event publishing (#756) | ||
|
Uriel Nudelman
|
c969d9b82d | Add fail_task failure reason support (#879) | ||
|
nthmost-orkes
|
1055736ce4 |
test: add integration tests for FORK_JOIN joinMode=SYNC (#619)
- Add ForkJoinSyncModeIntegrationTest with 9 test scenarios covering: all-succeed, backward compat (no joinMode), explicit ASYNC, optional branch fails, required branch fails, sequential tasks in branch, joinOn subset, nested fork/join, and 10-branch fan-out - Fix WorkflowTask.JoinMode proto field ID from 33→34 (conflict with `items` field added for DO_WHILE list iteration in a prior PR) - Add @ProtoEnum annotation to JoinMode enum so protogen can map it - Regenerate workflowtask.proto and AbstractProtoMapper with JoinMode - Add JSON workflow definition resources for manual/documentation use - Use raw JSON + RestTemplate for workflow registration to avoid classpath conflict with bundled conductor-client WorkflowTask Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
nthmost-orkes
|
0c80f3161b |
refactor: joinMode → enum; read from workflowTask instead of inputData
Addresses reviewer feedback on PR #680 (v1r3n): 1. Change WorkflowTask.joinMode from String to the inner enum WorkflowTask.JoinMode { SYNC, ASYNC }, eliminating free-form string comparison and making the API self-documenting. 2. Read joinMode directly from taskModel.getWorkflowTask() in Join.getEvaluationOffset() instead of pulling it from the task's inputData map. The value is now never duplicated into the task payload. 3. Remove the joinMode copy from JoinTaskMapper.getMappedTasks() since Join no longer reads it from input data. 4. Update tests accordingly: - JoinTest: construct WorkflowTask with JoinMode enum; replace the now-redundant case-insensitivity test with a null-workflowTask test that verifies the async default. - JoinTaskMapperTest: use JoinMode enum; assert joinMode is absent from inputData and present on the workflowTask. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Naomi Most
|
6ca9af706a | Merge branch 'main' into 619-add-synchronous-join-support-to-fork_join | ||
|
Viren Baraiya
|
7a048ae356 |
AI task tracking, module reorganization, and new REST endpoints (#801)
* AI task tracking, module reorganization, and new REST endpoints |
||
|
Naomi Most
|
99ed9f96b4 |
Add native list iteration support to DO_WHILE tasks (#624) (#760)
* Add native list iteration support to DO_WHILE tasks Implements Epic #624 - Native list iteration for DO_WHILE loops ## Changes - Add 'items' parameter to WorkflowTask for direct list iteration - Implement automatic list traversal in DoWhile task executor - Inject loopItem and loopIndex variables into workflow context - Add 13 comprehensive unit tests (23 total tests passing) - Maintain 100% backward compatibility with counter-based loops ## Usage Example ```json { "name": "process_items", "type": "DO_WHILE", "items": "${workflow.input.myList}", "loopOver": [ { "name": "process_task", "type": "SIMPLE", "inputParameters": { "item": "${do_while_task.output.loopItem}", "index": "${do_while_task.output.loopIndex}" } } ] } ``` ## Features - Eliminates manual counter management boilerplate - Provides loopItem and loopIndex variables automatically - Supports workflow expressions for items parameter - Optional custom loop conditions - Handles edge cases (empty lists, null values) ## Testing - All 23 DoWhileTest unit tests passing - Full conductor-core test suite passing (no regressions) - Backward compatibility verified with existing counter-based loops Closes #624 Part of #582 * Add Orkes compatibility: support _items in inputParameters Enables seamless migration from Orkes Conductor's OrkesDoWhile to OSS DoWhile. ## Changes - Modified `isListIteration()` to check both: 1. `workflowTask.items` (new OSS approach - priority) 2. `_items` in inputParameters (Orkes compatibility - fallback) - Modified `evaluateItemsList()` to evaluate items from both sources: - Priority 1: workflowTask.items field - Priority 2: _items from inputParameters (Orkes) - Added 4 comprehensive Orkes compatibility tests: - Detection of _items parameter - Evaluation of _items list - Priority verification (OSS field takes precedence) - Loop variable injection with _items ## Compatibility Strategy This allows Orkes to delete OrkesDoWhile and use OSS DoWhile directly: - Existing Orkes workflows using `_items` continue working - New workflows can use either approach - OSS `items` field takes precedence when both are present ## Testing - All 27 DoWhileTest tests passing (23 original + 4 Orkes compatibility) - Full conductor-core test suite passing - Code formatting (Spotless) applied Addresses CTO feedback for Orkes migration alignment. * Add comprehensive functionality tests per Epic #624 requirements Covers all critical test requirements from Epic #624 checklist. ## New Tests Added (9 tests) ### List Iteration Functionality 1. **Array support** - Tests Object[] array type conversion 2. **Complex objects** - Tests iteration with nested objects and custom conditions 3. **loopIndex conditions** - Tests using loopIndex in loop conditions 4. **Single item** - Tests graceful handling of single-element lists 5. **Direct list values** - Tests _items with direct list (non-expression) ### Integration & Edge Cases 6. **keepLastN integration** - Tests list iteration works with Phase 1 cleanup 7. **Exceeds list size** - Tests graceful stop when iteration exceeds list bounds 8. **LOC reduction verification** - Documents 53% LOC reduction with concrete example ## Epic #624 Checklist Coverage ### Technical Scope ✅ Complete - [x] Add items parameter - [x] Implement automatic list traversal - [x] Inject iteration variables - [x] Support workflow expressions - [x] Handle edge cases - [x] Maintain backward compatibility - [x] Integrate with keepLastN cleanup ← NOW TESTED ### Success Criteria ✅ Complete - [x] 50-70% LOC reduction (53% verified) - [x] items parameter accepts arrays/collections - [x] loopItem accessible - [x] loopIndex accessible - [x] Empty lists handled - [x] Backward compatibility maintained - [x] Works with keepLastN ← NOW TESTED - [x] Workflow expressions work ### Testing Requirements ✅ Complete (36 total tests) - [x] Unit tests for items parsing (3 tests) - [x] Unit tests for list iteration logic (13 tests) - [x] Integration with various data types (4 tests) - [x] Integration with workflow expressions (4 tests) - [x] Backward compatibility (2 tests) - [x] Integration with keepLastN ← NOW TESTED (1 test) - [x] Edge cases (5 tests) - [x] Orkes compatibility (4 tests) ## Test Counts - Phase 1 (keepLastN): 10 tests - List iteration (OSS): 13 tests - Orkes compatibility: 4 tests - Comprehensive functionality: 9 tests - **Total: 36 tests, all passing** Addresses Epic #624 testing requirements comprehensively. |
||
|
Viren Baraiya
|
0c723f7aa4 |
Support Agentic flows (#745)
🤖 Support for agentic workflows in Conductor
|
||
|
Viren Baraiya
|
b502f9591b | initial commit | ||
|
Naomi Most
|
d91c6f596b | Merge remote-tracking branch 'origin/main' into 619-add-synchronous-join-support-to-fork_join | ||
|
manan164
|
72301d1f0d | Added rate limit policy | ||
|
Naomi Most
|
b88c800d14 | Add joinMode property to WorkflowTask | ||
|
Shailesh Padave
|
3cee62a3dc | Merge branch 'main' into CCOR-195 | ||
|
Viren Baraiya
|
cbf53e8702 |
Merge pull request #552 from ruben-mathews/551
[Bug Fix] Allow Json Keys with spaces in the placeholders |
||
|
Shailesh Jagannath Padave
|
dbacd3bf8c | Applied spotless | ||
|
Shailesh Jagannath Padave
|
a7730f6aa5 | Introduce maskedFields in workflow def |