feat(workflow): experimental-search authoring boundary, reshaped and verified

Land the in-flight experimental-search slice as a real, honest v0.9.4
piece: a provider-neutral WorkflowSearchSpec authoring + freeze boundary
(parse/validate, preregistration hashes over baseline/requested+resolved
model/public evidence/evaluator, deterministic candidate ids and
admission batches) plus the operate best-of-N recipe's structured
'search' strategy (2-16 independent worktree candidates with
responseSchema contracts and a read-only judge). The module explicitly
remains an authoring boundary, not a runtime: hard_gates/score commands
are parsed and validated only, and docs say so.

Fixes found during the takeover operation:
- The checked-in recipe test failed against the real driver contract
  (parse_task_options lets prompt win over description, so fake-driver
  needles never matched and replies fell back to non-JSON). The recipe
  now puts the full instruction in the single driver-visible description
  and pins the prompt-wins contract with a new test.
- Queue claims verified against the runtime: the Workflow host's
  per-run concurrency gate (Semaphore, 16 live) is where larger
  populations wait; docs now name the gate instead of vague 'queues
  through Fleet'. WORKFLOW_SEARCH_MAX_CONCURRENT documents 16 as today's
  default with a cross-reference (a crate cycle prevents importing the
  host constant).
- TournamentOrdering (ScoreThenCost) and the 1,000-agent validation
  test bump retained from the slice.

Verified: cargo test -p codewhale-workflow -p codewhale-workflow-js
--locked green (250 + 16 + 9 + 49).
This commit is contained in:
Hmbown
2026-07-31 20:38:27 -07:00
parent 90cdb7346c
commit f3e3232ef2
12 changed files with 1016 additions and 61 deletions
+18
View File
@@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- Added a provider-neutral `WorkflowSearchSpec` authoring and freeze boundary
for experimental search inside Workflow, plus structured 216 candidate
generation in the best-of-N starter. Searches bind the baseline, requested
and resolved model ids, public evidence, evaluator identity, gates, scoring,
budget, and review-only integration policy before admission.
### Changed
- Workflow scale documentation now matches the runtime: up to 1,000 tasks per
run, admitted at most 16 live at a time (additional `task()` calls block on
the host's per-run concurrency gate, then route through Fleet).
`BranchTournament` keeps its historical cost-first default and supports
explicit score-first ordering. Runtime-owned hidden gates, benchmark scoring,
and clean replay remain an explicit host seam rather than being inferred from
worker self-reports.
## [0.9.3] - 2026-07-31
This is the Codewhale v0.9.3 source candidate. It is not a published release
+1
View File
@@ -80,6 +80,7 @@ path.
- [docs/PROVIDERS.md](docs/PROVIDERS.md) — every provider route: hosted,
gateway, and local
- [docs/FLEET.md](docs/FLEET.md) — fleets, the ledger, and resume
- [docs/WORKFLOW_EXPERIMENTAL_SEARCH.md](docs/WORKFLOW_EXPERIMENTAL_SEARCH.md) — frozen, provider-neutral experimental search within Workflow
- [docs/CONFIGURATION.md](docs/CONFIGURATION.md) — `config.toml`, hooks, and
the constitution
- [docs/AUTHORIZATION_ORDER.md](docs/AUTHORIZATION_ORDER.md) — how modes,
+3
View File
@@ -9,6 +9,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- Added a Workflow experimental-search authoring foundation and structured
search option to the bundled best-of-N recipe. This remains Workflow/Fleet,
never a new mode or scheduler, and never auto-applies a winner.
- `codewhale account login`, `status`, `logout`, and `keys` connect a CLI
profile to the managed Codewhale account through the browser device flow.
Sessions use the OS credential manager, remote BYOK values are never printed,
+26 -12
View File
@@ -18,7 +18,10 @@ the user has already chosen the approach.
1. Define one task, one evidence packet, and one explicit scoring rubric before
launching candidates. Include correctness, fit to the request, simplicity,
risk, and verification.
2. Choose `N` from 2 to 4. Default to 3. More candidates need a concrete reason.
2. Choose `N` from 2 to 4 for a quick comparison (default 3). For an explicit
experimental search, use the Workflow search option: 216 live candidates,
with larger validated populations queued at the Workflow host's 16-worker
concurrency gate rather than launched at once.
3. Give every candidate the same task and rubric. Add only a candidate number;
do not steer candidates toward different conclusions unless diversity is an
explicit part of the request.
@@ -54,20 +57,21 @@ When candidates must implement code, give each one:
- the same bounded `write_roots` or `exact_files`
Never run parallel writers in the parent checkout. Each implementer must return
`VERDICT: PASS|FAIL` with command evidence (tests/lint) — a diff alone is not
completion.
the structured candidate contract (candidate id, hypothesis, paths, commands,
self-verdict, risks, and artifact references). A self-verdict is evidence to
inspect, not a hard-gate result.
Optional diversity: pin different `model` / Fleet `fleet_profile` values when
the project has multiple capable routes; otherwise keep model strength `same`.
## Judge Once
Use one read-only reviewer/verifier worker, or the parent when the result is
small, to score all candidates against the original rubric. The judge must:
Use one read-only reviewer worker, or the parent when the result is small, to
score all candidates against the original rubric. The judge must:
- cite evidence from each candidate rather than vote by style;
- reject candidates that violate authority, scope, or verification gates;
- reject any code candidate without PASS evidence (or re-verify it);
- treat candidate-reported commands and PASS claims as untrusted until replay;
- name the winner and the decisive reasons;
- identify useful pieces worth combining, if any;
- say when the candidates are tied or all fail.
@@ -80,12 +84,22 @@ approaches into a new unreviewed solution.
For proposal-only work, return the winning answer with a compact score summary.
For code work:
1. Inspect the winning worktree diff.
2. Apply/merge the winner into the parent checkout only after the judge (or a
follow-up verifier) reports PASS with real checks.
3. Re-run the repository's checks in the parent after apply. Candidate
self-reports and style scores are not final verification.
4. Leave losing worktrees unmerged; do not pollute the parent with partials.
1. Freeze the baseline, evaluator, hard gates, score rule, and authority before
a larger search admits candidates. Any evaluator change starts a revision.
2. After a worker loses write authority, apply its patch to a clean baseline
and let the runtime—not that worker—run hard gates and scoring.
3. Inspect the winning worktree diff and independently replay it on the clean
baseline. A different read-only model may look for gaming, but deterministic
tests remain the authority.
4. Present the verified winner for review. Applying or merging is a separate,
explicit user action; `NONE` is valid when every candidate fails.
5. Preserve losing and failed candidate receipts as useful negative results.
The checked-in `operate_best_of_n.workflow.js` recipe supports
`strategy: "search"` for structured 216 candidate generation and review. It
does **not** yet turn prompt-listed commands into hidden runtime gates. Do not
advertise those gates until the runtime evaluator host consumes a frozen
`WorkflowSearchSpec`.
Stop early when one candidate reveals a hard constraint that invalidates the
tournament. Report the negative result rather than spending the remaining
+107
View File
@@ -57,6 +57,113 @@ async fn args_global_is_the_invocation_input() {
assert_eq!(value, json!({"sum": 42, "tag": "release"}));
}
#[tokio::test]
async fn checked_in_best_of_n_search_recipe_runs_with_structured_receipts() {
let driver = Arc::new(FakeDriver::new());
for index in 1..=2 {
driver.on(
// Rules match the driver-visible `TaskRequest.description`, which
// is the full instruction text (the VM's `prompt` alias wins over
// a short label). Match the unique per-candidate suffix line.
&format!("candidate_id=cand_{index:03} of 2."),
FakeReply::Complete(
json!({
"candidate_id": format!("cand_{index:03}"),
"hypothesis": "bounded fixture",
"modified_paths": ["src/lib.rs"],
"commands_run": ["cargo test --locked"],
"self_verdict": "pass",
"known_risks": [],
"artifact_refs": [format!("patch:cand_{index:03}")]
})
.to_string(),
),
);
}
driver.on(
"read-only tournament judge",
FakeReply::Complete(
json!({
"winner_id": "cand_001",
"ranking": ["cand_001", "cand_002"],
"verification_required": true,
"reasons": ["fixture score"]
})
.to_string(),
),
);
let value = run(
&driver,
include_str!("../../../workflows/operate_best_of_n.workflow.js"),
json!({
"brief": "Implement the fixture",
"strategy": "search",
"n": 2,
"writeRoots": ["src"],
"model": "deepseek-v4-flash",
"thinking": "max"
}),
)
.await
.expect("checked-in search recipe should execute");
assert_eq!(value["scenario"], "operate-search");
assert_eq!(value["review"]["winner_id"], "cand_001");
assert_eq!(driver.spawn_count(), 3);
let requests = driver.requests();
assert_eq!(requests[0].model.as_deref(), Some("deepseek-v4-flash"));
assert_eq!(requests[0].thinking.as_deref(), Some("max"));
assert_eq!(requests[0].write_roots, ["src"]);
assert_eq!(requests[2].write_authority.as_deref(), Some("read_only"));
// Regression: the driver-visible description is the full instruction text,
// so reply rules must target text that actually reaches the driver. If a
// future recipe reintroduces a separate short `description` next to a long
// `prompt`, these needles stop matching, the FakeDriver falls back to its
// non-JSON "done:..." reply, and the structured receipts fail loudly.
assert!(
requests[0]
.description
.starts_with("You are one independent candidate")
);
assert!(
requests[0]
.description
.contains("CANDIDATE-SPECIFIC INSTRUCTION: candidate_id=cand_001 of 2.")
);
assert!(
requests[2]
.description
.starts_with("You are the read-only tournament judge")
);
}
#[tokio::test]
async fn task_prompt_wins_over_description_as_driver_visible_text() {
let driver = Arc::new(FakeDriver::new());
let value = run(
&driver,
r#"
return await task({
description: "short progress label",
prompt: "the real instruction",
});
"#,
json!(null),
)
.await
.unwrap();
// No rules were registered, so the FakeDriver fallback echoes the
// driver-visible description. The reply text proves the driver received
// the prompt, not the short label.
assert_eq!(value, json!("done:the real instruction"));
let requests = driver.requests();
assert_eq!(requests.len(), 1);
assert_eq!(requests[0].description, "the real instruction");
assert_ne!(requests[0].description, "short progress label");
}
#[tokio::test]
async fn task_round_trip_carries_all_options_and_normalizes_profile() {
let driver = Arc::new(FakeDriver::new());
+557
View File
@@ -0,0 +1,557 @@
//! Provider-neutral experimental search authoring for Workflow.
//!
//! This module is an authoring and freeze boundary, not a new runtime or
//! scheduler. A validated search still has to be lowered by the Workflow host
//! into Fleet workers plus a runtime-owned evaluator. In particular, worker
//! self-reports are never promoted to hard-gate evidence here.
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use thiserror::Error;
use crate::{DEFAULT_FLEET_WORKFLOW_MAX_AGENTS, experimental_search::SearchSpecError::*};
pub const WORKFLOW_SEARCH_SCHEMA_VERSION: u32 = 1;
/// Live-worker ceiling for one search admission batch.
///
/// 16 matches the Workflow host's live-child ceiling today
/// (`codewhale_workflow_js::WORKFLOW_MAX_CONCURRENT`, from which the tui
/// driver sizes its per-run admission semaphore). This crate cannot import
/// that constant directly because `codewhale-workflow-js` depends on
/// `codewhale-workflow`, so 16 is documented here as today's default — not a
/// new configuration knob. Keep it in sync with the host constant.
pub const WORKFLOW_SEARCH_MAX_CONCURRENT: u16 = 16;
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct WorkflowSearchSpec {
#[serde(default = "default_schema_version")]
pub schema_version: u32,
pub name: String,
pub objective: String,
pub population: u16,
pub rounds: Vec<u16>,
pub concurrency: u16,
pub worker: SearchWorkerSpec,
#[serde(default)]
pub budget: SearchBudgetSpec,
pub hard_gates: SearchHardGateSpec,
pub score: SearchScoreSpec,
#[serde(default)]
pub selection: SearchSelectionSpec,
#[serde(default)]
pub integration_policy: SearchIntegrationPolicy,
}
impl WorkflowSearchSpec {
pub fn from_toml(source: &str) -> Result<Self, SearchSpecError> {
let spec: Self = toml::from_str(source).map_err(|error| Parse(error.to_string()))?;
spec.validate()?;
Ok(spec)
}
pub fn validate(&self) -> Result<(), SearchSpecError> {
if self.schema_version != WORKFLOW_SEARCH_SCHEMA_VERSION {
return Err(UnsupportedSchemaVersion(self.schema_version));
}
validate_name(&self.name)?;
validate_text("objective", &self.objective, 32_768)?;
if !(2..=DEFAULT_FLEET_WORKFLOW_MAX_AGENTS as u16).contains(&self.population) {
return Err(InvalidPopulation(self.population));
}
if self.concurrency == 0
|| self.concurrency > WORKFLOW_SEARCH_MAX_CONCURRENT
|| self.concurrency > self.population
{
return Err(InvalidConcurrency {
concurrency: self.concurrency,
population: self.population,
});
}
validate_rounds(self.population, &self.rounds)?;
validate_text("worker.model", &self.worker.model, 256)?;
if self.worker.write_roots.is_empty() && self.worker.exact_files.is_empty() {
return Err(UnboundedWriteScope);
}
validate_string_list("worker.write_roots", &self.worker.write_roots, 128, 1_024)?;
validate_string_list("worker.exact_files", &self.worker.exact_files, 256, 1_024)?;
if self.budget.max_cost_microusd == Some(0) {
return Err(ZeroBudget("max_cost_microusd"));
}
if self.budget.max_tokens == Some(0) {
return Err(ZeroBudget("max_tokens"));
}
if !self.hard_gates.forbid_test_changes {
return Err(TestWeakeningAllowed);
}
if self.hard_gates.commands.is_empty() {
return Err(MissingHardGates);
}
validate_string_list("hard_gates.commands", &self.hard_gates.commands, 32, 4_096)?;
validate_string_list(
"hard_gates.protected_paths",
&self.hard_gates.protected_paths,
256,
1_024,
)?;
validate_text("score.command", &self.score.command, 4_096)?;
validate_text("score.metric", &self.score.metric, 256)?;
if !(1..=25).contains(&self.score.trials) {
return Err(InvalidTrials(self.score.trials));
}
if self.score.tie_breakers.is_empty() {
return Err(MissingTieBreakers);
}
Ok(())
}
/// Freeze the exact public inputs and evaluator identity before admission.
/// The evaluator bytes are hashed, not exposed to generation workers.
pub fn freeze(
&self,
baseline_commit: &str,
resolved_model: &str,
public_evidence: &[u8],
evaluator: &[u8],
) -> Result<FrozenWorkflowSearch, SearchSpecError> {
self.validate()?;
validate_commit(baseline_commit)?;
validate_text("resolved_model", resolved_model, 256)?;
if evaluator.is_empty() {
return Err(EmptyEvaluator);
}
let public_evidence_hash = sha256_label(public_evidence);
let evaluator_hash = sha256_label(evaluator);
let freeze_input = FreezeInput {
spec: self,
baseline_commit,
requested_model: &self.worker.model,
resolved_model,
public_evidence_hash: &public_evidence_hash,
evaluator_hash: &evaluator_hash,
};
let encoded =
serde_json::to_vec(&freeze_input).map_err(|error| FreezeEncoding(error.to_string()))?;
let preregistration_hash = sha256_label(&encoded);
let search_id = format!("search-{}", &preregistration_hash[7..23]);
Ok(FrozenWorkflowSearch {
schema_version: self.schema_version,
search_id,
baseline_commit: baseline_commit.to_string(),
preregistration_hash,
public_evidence_hash,
evaluator_hash,
requested_model: self.worker.model.clone(),
resolved_model: resolved_model.to_string(),
candidate_ids: self.candidate_ids(),
})
}
#[must_use]
pub fn candidate_ids(&self) -> Vec<String> {
let width = self.population.to_string().len().max(3);
(1..=self.population)
.map(|index| format!("cand_{index:0width$}"))
.collect()
}
/// Deterministic admission batches. Fleet owns actual scheduling and may
/// run fewer workers when its configured pool or provider quota is lower.
#[must_use]
pub fn admission_batches(&self) -> Vec<Vec<String>> {
self.candidate_ids()
.chunks(usize::from(self.concurrency))
.map(<[String]>::to_vec)
.collect()
}
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct SearchWorkerSpec {
#[serde(default)]
pub provider: Option<String>,
pub model: String,
#[serde(default)]
pub reasoning_effort: SearchReasoningEffort,
#[serde(default)]
pub write_authority: SearchWriteAuthority,
#[serde(default)]
pub write_roots: Vec<String>,
#[serde(default)]
pub exact_files: Vec<String>,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(rename_all = "snake_case")]
pub enum SearchReasoningEffort {
Off,
Low,
Medium,
#[default]
High,
Max,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(rename_all = "snake_case")]
pub enum SearchWriteAuthority {
#[default]
WorktreeWrite,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Default)]
pub struct SearchBudgetSpec {
#[serde(default)]
pub max_cost_microusd: Option<u64>,
#[serde(default)]
pub max_tokens: Option<u64>,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct SearchHardGateSpec {
pub commands: Vec<String>,
#[serde(default = "default_true")]
pub forbid_test_changes: bool,
#[serde(default)]
pub protected_paths: Vec<String>,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct SearchScoreSpec {
pub command: String,
pub metric: String,
#[serde(default)]
pub direction: SearchDirection,
#[serde(default = "default_trials")]
pub trials: u16,
#[serde(default)]
pub tie_breakers: Vec<SearchTieBreaker>,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(rename_all = "snake_case")]
pub enum SearchDirection {
#[default]
Minimize,
Maximize,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum SearchTieBreaker {
DiffLines,
CostMicrousd,
Score,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct SearchSelectionSpec {
#[serde(default)]
pub policy: SearchSelectionPolicy,
#[serde(default = "default_true")]
pub retain_diversity: bool,
#[serde(default)]
pub ordering: Vec<SearchSelectionMetric>,
}
impl Default for SearchSelectionSpec {
fn default() -> Self {
Self {
policy: SearchSelectionPolicy::Pareto,
retain_diversity: true,
ordering: Vec::new(),
}
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(rename_all = "snake_case")]
pub enum SearchSelectionPolicy {
#[default]
Pareto,
Ordered,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum SearchSelectionMetric {
Score,
Runtime,
DiffLines,
CostMicrousd,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(rename_all = "snake_case")]
pub enum SearchIntegrationPolicy {
/// Produce a verified, reviewable winner or NONE. Never apply or merge it.
#[default]
ReviewOnly,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct FrozenWorkflowSearch {
pub schema_version: u32,
pub search_id: String,
pub baseline_commit: String,
pub preregistration_hash: String,
pub public_evidence_hash: String,
pub evaluator_hash: String,
pub requested_model: String,
pub resolved_model: String,
pub candidate_ids: Vec<String>,
}
#[derive(Serialize)]
struct FreezeInput<'a> {
spec: &'a WorkflowSearchSpec,
baseline_commit: &'a str,
requested_model: &'a str,
resolved_model: &'a str,
public_evidence_hash: &'a str,
evaluator_hash: &'a str,
}
#[derive(Debug, Clone, PartialEq, Eq, Error)]
pub enum SearchSpecError {
#[error("failed to parse Workflow search TOML: {0}")]
Parse(String),
#[error("unsupported Workflow search schema version {0}")]
UnsupportedSchemaVersion(u32),
#[error("search name must be a 1-96 character lowercase token")]
InvalidName,
#[error("{field} must be non-empty and no longer than {max} characters")]
InvalidText { field: &'static str, max: usize },
#[error("population {0} must be between 2 and 1000")]
InvalidPopulation(u16),
#[error(
"concurrency {concurrency} must be between 1 and 16 and no greater than population {population}"
)]
InvalidConcurrency { concurrency: u16, population: u16 },
#[error("rounds must start at population, decrease strictly, and end at 1")]
InvalidRounds,
#[error("write-capable search workers require write_roots or exact_files")]
UnboundedWriteScope,
#[error("{field} contains an empty, oversized, or duplicate entry")]
InvalidStringList { field: &'static str },
#[error("{0} must be greater than zero when set")]
ZeroBudget(&'static str),
#[error("experimental search must forbid test changes")]
TestWeakeningAllowed,
#[error("experimental search requires at least one runtime-owned hard-gate command")]
MissingHardGates,
#[error("score.trials must be between 1 and 25, got {0}")]
InvalidTrials(u16),
#[error("experimental search requires at least one deterministic tie-breaker")]
MissingTieBreakers,
#[error("baseline_commit must be a 7-64 character hexadecimal commit id")]
InvalidBaselineCommit,
#[error("evaluator bytes must be non-empty")]
EmptyEvaluator,
#[error("failed to encode frozen Workflow search: {0}")]
FreezeEncoding(String),
}
fn default_schema_version() -> u32 {
WORKFLOW_SEARCH_SCHEMA_VERSION
}
fn default_trials() -> u16 {
5
}
fn default_true() -> bool {
true
}
fn validate_name(value: &str) -> Result<(), SearchSpecError> {
if value.is_empty()
|| value.len() > 96
|| !value
.bytes()
.all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || b"-_".contains(&byte))
{
return Err(InvalidName);
}
Ok(())
}
fn validate_text(field: &'static str, value: &str, max: usize) -> Result<(), SearchSpecError> {
if value.trim().is_empty() || value.chars().count() > max {
return Err(InvalidText { field, max });
}
Ok(())
}
fn validate_rounds(population: u16, rounds: &[u16]) -> Result<(), SearchSpecError> {
if rounds.len() < 2
|| rounds.first() != Some(&population)
|| rounds.last() != Some(&1)
|| rounds.windows(2).any(|pair| pair[0] <= pair[1])
{
return Err(InvalidRounds);
}
Ok(())
}
fn validate_string_list(
field: &'static str,
values: &[String],
max_items: usize,
max_chars: usize,
) -> Result<(), SearchSpecError> {
if values.len() > max_items
|| values
.iter()
.any(|value| value.trim().is_empty() || value.chars().count() > max_chars)
|| values
.iter()
.enumerate()
.any(|(index, value)| values[..index].contains(value))
{
return Err(InvalidStringList { field });
}
Ok(())
}
fn validate_commit(value: &str) -> Result<(), SearchSpecError> {
if !(7..=64).contains(&value.len()) || !value.bytes().all(|byte| byte.is_ascii_hexdigit()) {
return Err(InvalidBaselineCommit);
}
Ok(())
}
fn sha256_label(bytes: &[u8]) -> String {
let digest = Sha256::digest(bytes);
let mut output = String::with_capacity(71);
output.push_str("sha256:");
for byte in digest {
use std::fmt::Write as _;
let _ = write!(output, "{byte:02x}");
}
output
}
#[cfg(test)]
mod tests {
use super::*;
const SPEC: &str = r#"
name = "speed-up-certificate"
objective = "Reduce runtime without changing exact results"
population = 32
rounds = [32, 8, 3, 1]
concurrency = 16
integration_policy = "review_only"
[worker]
provider = "deepseek"
model = "deepseek-v4-flash"
reasoning_effort = "max"
write_authority = "worktree_write"
write_roots = ["code"]
[budget]
max_cost_microusd = 5000000
max_tokens = 10000000
[hard_gates]
commands = ["cargo test --locked", "git diff --exit-code -- expected.json"]
forbid_test_changes = true
protected_paths = ["tests", "expected.json"]
[score]
command = "./scripts/benchmark_candidate.sh"
direction = "minimize"
metric = "median_runtime_ms"
trials = 5
tie_breakers = ["diff_lines", "cost_microusd"]
[selection]
policy = "pareto"
retain_diversity = true
"#;
#[test]
fn parses_valid_search_and_queues_through_live_cap() {
let spec = WorkflowSearchSpec::from_toml(SPEC).expect("valid search spec");
assert_eq!(spec.population, 32);
assert_eq!(spec.admission_batches().len(), 2);
assert_eq!(spec.admission_batches()[0].len(), 16);
assert_eq!(spec.candidate_ids()[0], "cand_001");
assert_eq!(spec.candidate_ids()[31], "cand_032");
}
#[test]
fn freeze_is_deterministic_and_model_version_sensitive() {
let spec = WorkflowSearchSpec::from_toml(SPEC).expect("valid search spec");
let first = spec
.freeze(
"33bc6a98",
"DeepSeek-V4-Flash-0731",
b"public evidence",
b"private evaluator",
)
.expect("freeze succeeds");
let replay = spec
.freeze(
"33bc6a98",
"DeepSeek-V4-Flash-0731",
b"public evidence",
b"private evaluator",
)
.expect("freeze succeeds");
let drifted = spec
.freeze(
"33bc6a98",
"DeepSeek-V4-Flash-next",
b"public evidence",
b"private evaluator",
)
.expect("freeze succeeds");
assert_eq!(first, replay);
assert_ne!(first.search_id, drifted.search_id);
assert_ne!(first.preregistration_hash, drifted.preregistration_hash);
assert_eq!(first.requested_model, "deepseek-v4-flash");
assert_eq!(first.resolved_model, "DeepSeek-V4-Flash-0731");
}
#[test]
fn rejects_unsafe_or_unbounded_searches() {
let mut spec = WorkflowSearchSpec::from_toml(SPEC).expect("valid search spec");
spec.hard_gates.forbid_test_changes = false;
assert_eq!(spec.validate(), Err(SearchSpecError::TestWeakeningAllowed));
spec.hard_gates.forbid_test_changes = true;
spec.worker.write_roots.clear();
assert_eq!(spec.validate(), Err(SearchSpecError::UnboundedWriteScope));
}
#[test]
fn rejects_invalid_rounds_and_excess_live_concurrency() {
let mut spec = WorkflowSearchSpec::from_toml(SPEC).expect("valid search spec");
spec.rounds = vec![32, 8, 8, 1];
assert_eq!(spec.validate(), Err(SearchSpecError::InvalidRounds));
spec.rounds = vec![32, 1];
spec.concurrency = 17;
assert_eq!(
spec.validate(),
Err(SearchSpecError::InvalidConcurrency {
concurrency: 17,
population: 32,
})
);
}
#[test]
fn deserialization_refuses_auto_merge_policy() {
let source = SPEC.replace("review_only", "auto_merge");
let error = WorkflowSearchSpec::from_toml(&source).expect_err("must reject auto merge");
assert!(matches!(error, SearchSpecError::Parse(_)));
}
}
+48 -9
View File
@@ -5,6 +5,7 @@
//! top only after their cancellation and evidence semantics are proven.
mod elevation;
pub mod experimental_search;
/// Setup-time Fleet composition: a suggestion schema with no runtime authority.
///
/// Deliberately **not** re-exported from the crate root. Nothing consumes it
@@ -1638,18 +1639,35 @@ fn truncate_evidence(value: &str) -> String {
pub struct BranchTournament {
#[serde(default)]
pub min_score: u32,
#[serde(default)]
pub ordering: TournamentOrdering,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
#[serde(rename_all = "snake_case")]
pub enum TournamentOrdering {
/// Historical behavior: choose the cheapest passing branch, then score.
#[default]
CostThenScore,
/// Quality-first behavior for explicitly configured evaluation workflows.
ScoreThenCost,
}
impl BranchTournament {
pub fn select(&self, candidates: &[BranchCandidate]) -> Option<BranchCandidate> {
candidates
.iter()
.filter(|candidate| {
let passing = || {
candidates.iter().filter(|candidate| {
candidate.status == WorkflowRunStatus::Succeeded
&& candidate.score >= self.min_score
})
.min_by_key(|candidate| (candidate.cost, std::cmp::Reverse(candidate.score)))
.cloned()
};
match self.ordering {
TournamentOrdering::CostThenScore => passing()
.min_by_key(|candidate| (candidate.cost, std::cmp::Reverse(candidate.score))),
TournamentOrdering::ScoreThenCost => passing()
.min_by_key(|candidate| (std::cmp::Reverse(candidate.score), candidate.cost)),
}
.cloned()
}
}
@@ -2790,7 +2808,7 @@ mod tests {
}
#[test]
fn fleet_validation_accepts_one_hundred_agents_and_variable_models() {
fn fleet_validation_accepts_one_thousand_agents_and_variable_models() {
let nodes = (0..DEFAULT_FLEET_WORKFLOW_MAX_AGENTS)
.map(|index| {
let mut leaf = match leaf_node(&format!("agent-{index}")) {
@@ -2817,14 +2835,14 @@ mod tests {
let shape = workflow
.validate_for_fleet()
.expect("one hundred agents should fit the Fleet Workflow limit");
.expect("one thousand agents should fit the Fleet Workflow limit");
assert_eq!(shape.total_agents, DEFAULT_FLEET_WORKFLOW_MAX_AGENTS);
assert_eq!(shape.max_depth, 1);
}
#[test]
fn fleet_validation_rejects_more_than_one_hundred_agents() {
fn fleet_validation_rejects_more_than_one_thousand_agents() {
let nodes = (0..=DEFAULT_FLEET_WORKFLOW_MAX_AGENTS)
.map(|index| leaf_node(&format!("agent-{index}")))
.collect();
@@ -4052,7 +4070,10 @@ mod tests {
#[test]
fn tournament_selects_passing_minimal_branch() {
let tournament = BranchTournament { min_score: 60 };
let tournament = BranchTournament {
min_score: 60,
ordering: TournamentOrdering::CostThenScore,
};
let candidates = vec![
candidate(
"expensive-pass",
@@ -4079,6 +4100,24 @@ mod tests {
assert_eq!(selected.branch_id, "cheap-pass");
}
#[test]
fn tournament_can_select_score_before_cost_explicitly() {
let tournament = BranchTournament {
min_score: 60,
ordering: TournamentOrdering::ScoreThenCost,
};
let candidates = vec![
candidate("quality", WorkflowRunStatus::Succeeded, 95, 100, "quality"),
candidate("minimal", WorkflowRunStatus::Succeeded, 70, 10, "minimal"),
];
let selected = tournament
.select(&candidates)
.expect("one passing branch should be selected");
assert_eq!(selected.branch_id, "quality");
}
#[test]
fn pareto_frontier_keeps_diverse_candidates() {
let frontier = ParetoFrontier { max_items: 4 };
+6 -3
View File
@@ -322,13 +322,16 @@ perform real work as `codewhale exec` processes.
Default Workflow-to-Fleet validation is intentionally bounded:
- 100 total worker agents per workflow run;
- 5 recursive Fleet rings;
- 1,000 total worker agents per Workflow run;
- 16 live worker agents at once; larger populations queue (block) on the host's
per-run concurrency gate until a live slot frees, then route through Fleet;
- 5 recursive Fleet rings as the hard ceiling (default user configuration: 2);
- bounded loops only (`max_iterations` required);
- bounded dynamic expansion only (`max_children` plus a template required).
These are population limits, not a demand to launch everything at once. A
100-agent workflow should still drain through the configured Fleet worker pool.
1,000-agent Workflow should still drain through the configured Fleet worker
pool.
Recommended model layouts, such as a DeepSeek Pro orchestrator with Flash
workers in the first ring and cheaper workers farther out, are presets only.
Every slot can inherit the active model or carry an explicit model override.
+14
View File
@@ -123,12 +123,26 @@ Planned scope:
## Eval Workset
Implemented authoring foundation:
- Provider-neutral `WorkflowSearchSpec` validation and deterministic freeze
receipts for an experimental-search option within Workflow. The freeze binds
the baseline, requested and resolved model names, public evidence, and
evaluator identity before candidate admission.
- The best-of-N Workflow starter can generate 216 structured, independent
worktree candidates with cache-stable shared instructions and a read-only
review. This is generation/review evidence, not runtime-owned hard-gate proof.
Planned scope:
- Reproducible task suites for coding, review, docs, release checks, and
long-context workflows.
- Side-by-side route comparisons where the exact model, provider, thinking
level, prompt, and tool policy are captured.
- Runtime-owned hard gates and command scoring after worker write authority is
revoked; clean-baseline replay; duplicate-patch detection; multi-round
Pareto/diversity promotion; aggregate receipts over Fleet receipts; and a
Workflow-panel leaderboard. No winner is applied or merged automatically.
## Observability Workset
+21 -4
View File
@@ -123,13 +123,30 @@ or filesystem authority.
Fleet launch validation applies a conservative default shape before any
Workflow IR is lowered to workers:
- up to 100 total worker agents per workflow run;
- up to 5 recursive Fleet rings;
- up to 1,000 total worker agents per Workflow run;
- up to 16 live worker agents at once; larger populations queue (block) on the
host's per-run concurrency gate until a live slot frees, then route through
Fleet;
- up to 5 recursive Fleet rings (the default user configuration is 2);
- loops require `max_iterations`;
- dynamic `expand` nodes require `max_children` and a template.
Those limits bound the workflow population, not instantaneous launch
concurrency. A valid 100-agent workflow can still drain through a smaller Fleet
Those limits distinguish population from instantaneous launch concurrency. A
valid 1,000-agent Workflow can still drain through a smaller Fleet
worker pool. Model selection stays per slot: a DeepSeek preset can suggest
`deepseek-v4-pro` for the orchestrator and `deepseek-v4-flash` for nearby
workers, but users and agents may override any slot when the task calls for it.
## Experimental search is a Workflow option
Experimental search generalizes the existing best-of-N recipe without adding a
new product mode, scheduler, or sub-agent API. A provider-neutral
`WorkflowSearchSpec` freezes the objective, baseline, model request and resolved
version, public evidence, evaluator hash, hard gates, scoring rule, budgets,
write scope, rounds, and review-only integration policy before admission.
The current JS starter supports structured generation and read-only review with
`strategy: "search"`. Runtime-owned command gates, hidden evaluation, benchmark
scoring, and clean-baseline replay are an explicit host seam still to wire; a
candidate's self-verdict must never be promoted into evaluator truth. See
[Workflow Experimental Search](WORKFLOW_EXPERIMENTAL_SEARCH.md).
+115
View File
@@ -0,0 +1,115 @@
# Workflow Experimental Search
Experimental search is an option for **Workflow**, not a fourth mode and not a
second scheduler. Fleet still owns workers, routes, concurrency, leases, and
receipts. Workflow owns the frozen order: independent generation, evaluation,
selection, repair, and verification.
The product vocabulary is Fleet (who), Workflow (what order), Lane (one running
Workflow), Runtime (where/how), and Operate (the mode). Do not introduce
"WhaleFlow" as a current synonym.
## Current foundation
- The imperative Workflow VM admits at most 1,000 tasks per run and 16 live
tasks at once. The host's per-run concurrency gate is a semaphore sized
`WORKFLOW_MAX_CONCURRENT = 16` (`codewhale-workflow-js`); additional
`task()` spawns block on that gate until a live slot frees, then route
through Fleet. A larger declared population therefore queues at the gate,
not through Fleet itself.
- `WorkflowSearchSpec` is a provider-neutral TOML/Rust authoring boundary. It
validates bounded worktree writes, rounds, budgets, mandatory anti-test-
weakening posture, hard-gate commands, deterministic scoring, selection, and
review-only integration.
- `hard_gates.commands` and `score.command` are parsed and validated only;
nothing in this slice executes them. Runtime-owned gate execution and
benchmark scoring are the evaluator host seam described below.
- Freezing a spec records one deterministic search id and preregistration hash
over the baseline commit, requested and resolved model ids, public evidence
hash, evaluator hash, and the complete spec.
- `operate_best_of_n.workflow.js` supports `strategy: "search"` for 216
structured independent candidates and one read-only reviewer. The stable
shared instructions precede the candidate-specific suffix to favor provider
prefix caching.
- `BranchTournament` preserves its historical cost-first default but now
supports explicit score-first ordering. Pareto selection remains available
in the typed Workflow core.
## Security and truth boundary
The JS starter does not own the shell or evaluator. It therefore cannot turn a
command mentioned in a prompt into a hidden, runtime-owned gate. Candidate
self-verdicts and claimed commands are untrusted. Until the evaluator host seam
lands, the starter produces generation and review evidence only.
The evaluator host must:
1. freeze a real Git baseline and evaluator before admitting candidates;
2. give every writer its own worktree and the same public evidence;
3. revoke writer authority before injecting hidden tests or scorer details;
4. apply each patch to a clean baseline, reject forbidden/test changes, then
run hard gates before performance scoring;
5. record commands, exit codes, environment, token/cache/cost usage, artifacts,
promotion reasons, and failures on top of Fleet receipts;
6. replay the provisional winner cleanly and run an independent read-only
adversarial review; and
7. return `NONE` when all candidates fail and never apply or merge a winner
without a later explicit user action.
## Provider presets
The abstraction remains provider-neutral. A DeepSeek Flash preset can exploit
its automatically managed prefix cache by keeping shared instructions,
experiment rules, repository evidence, and the response contract stable, with
the candidate id last. Preliminary scouts can use lower effort while promoted
implementers/finalists use high or max effort.
Record both the requested API model id and the resolved provider version.
Provider account concurrency is not Fleet concurrency: the runtime keeps its
16-live-worker ceiling, handles 429 responses and keep-alives outside the
deterministic VM, and stops new admissions when the shared budget is exhausted.
## Example authoring shape
```toml
name = "speed-up-certificate"
objective = "Reduce runtime without changing exact results"
population = 32
rounds = [32, 8, 3, 1]
concurrency = 16
integration_policy = "review_only"
[worker]
provider = "deepseek"
model = "deepseek-v4-flash"
reasoning_effort = "high"
write_authority = "worktree_write"
write_roots = ["code"]
[budget]
max_cost_microusd = 5000000
max_tokens = 10000000
[hard_gates]
commands = [
"PYTHONWARNINGS=error python certificate.py",
"git diff --exit-code -- expected_result.json",
]
forbid_test_changes = true
protected_paths = ["tests", "expected_result.json"]
[score]
command = "./scripts/benchmark_candidate.sh"
direction = "minimize"
metric = "median_runtime_ms"
trials = 5
tie_breakers = ["diff_lines", "cost_microusd"]
[selection]
policy = "pareto"
retain_diversity = true
```
This file is authoring input, not yet a runnable CLI promise. The next runtime
slice is the evaluator host and aggregate receipt; after that, the natural-
language authoring layer can safely compile a user's request into this shape.
+100 -33
View File
@@ -1,21 +1,56 @@
/**
* Operate starter best-of-N implementers in worktrees, then reviewer.
* Operate starter independent worktree candidates, then one reviewer.
*
* Spawns N worktree implementers with the same brief, then a read-only
* reviewer that picks a winner. The parent must apply the winner only after
* PASS evidence (skill: best-of-n).
* Set strategy="search" for a bounded 2-16 candidate search. This remains a
* Workflow recipe, not a new mode or scheduler. Runtime-owned command gates
* and clean-baseline scoring require the typed search/evaluator host seam.
*
* Run: /workflow run workflows/operate_best_of_n.workflow.js
* Args: { brief, n?, targetFiles?, writeRoots? }
* Args: { brief, n?, strategy?, rubric?, model?, thinking?, targetFiles?, writeRoots? }
*/
export default async function (args) {
const brief =
args?.brief ??
args?.task ??
"Propose and implement the smallest correct fix for the open failure.";
const n = Math.min(4, Math.max(2, Number(args?.n ?? 3) || 3));
const strategy = args?.strategy === "search" ? "search" : "best_of_n";
const maxCandidates = strategy === "search" ? 16 : 4;
const defaultCandidates = strategy === "search" ? 8 : 3;
const n = Math.min(
maxCandidates,
Math.max(2, Number(args?.n ?? defaultCandidates) || defaultCandidates)
);
const exactFiles = Array.isArray(args?.targetFiles) ? args.targetFiles : [];
const writeRoots = Array.isArray(args?.writeRoots) ? args.writeRoots : [];
const rubric =
args?.rubric ??
"Correctness first; then fit, measured quality, simplicity, risk, and verification evidence.";
const model = typeof args?.model === "string" ? args.model : undefined;
const thinking =
typeof args?.thinking === "string" ? args.thinking : undefined;
const candidateSchema = {
type: "object",
additionalProperties: false,
required: [
"candidate_id",
"hypothesis",
"modified_paths",
"commands_run",
"self_verdict",
"known_risks",
"artifact_refs",
],
properties: {
candidate_id: { type: "string" },
hypothesis: { type: "string" },
modified_paths: { type: "array", items: { type: "string" } },
commands_run: { type: "array", items: { type: "string" } },
self_verdict: { type: "string", enum: ["pass", "fail"] },
known_risks: { type: "array", items: { type: "string" } },
artifact_refs: { type: "array", items: { type: "string" } },
},
};
phase("Candidates");
const candidateFns = [];
@@ -23,9 +58,30 @@ export default async function (args) {
const index = i;
candidateFns.push(() =>
task({
description: `Best-of-N candidate ${index}/${n}`,
// The VM delivers one text to the driver: `prompt` (alias) wins over
// `description`, so the full instruction lives in `description` and
// `label` carries the short progress name. A separate short
// `description` would never reach the driver.
description: [
"You are one independent candidate in a Codewhale Workflow search.",
"Implement the same frozen brief and rubric in this isolated worktree only.",
"Do not inspect other candidates, rankings, hidden tests, or evaluator internals.",
"Do not push. Do not merge. Do not touch the parent checkout.",
"Your self_verdict is informational; only runtime-owned evaluation can pass a hard gate.",
"Return only the required structured response.",
"",
"BRIEF:",
String(brief),
"",
"RUBRIC:",
String(rubric),
"",
`CANDIDATE-SPECIFIC INSTRUCTION: candidate_id=cand_${String(index).padStart(3, "0")} of ${n}.`,
].join("\n"),
label: `candidate_${index}`,
type: "implementer",
...(model ? { model } : {}),
...(thinking ? { thinking } : {}),
worktree: true,
writeAuthority: "worktree_write",
...(exactFiles.length ? { exactFiles } : {}),
@@ -35,19 +91,8 @@ export default async function (args) {
"Do not share other candidates' answers.",
"Parent checkout must remain unchanged until apply.",
],
acceptance: [
"Return VERDICT PASS/FAIL with command evidence.",
"List every modified path with a one-line why.",
],
prompt: [
`You are candidate ${index} of ${n} in a best-of-N tournament.`,
"Implement the brief below in this isolated worktree only.",
"Run relevant checks. End with VERDICT: PASS|FAIL, COMMANDS, EVIDENCE.",
"Do not push. Do not merge. Do not touch the parent checkout.",
"",
"BRIEF:",
String(brief),
].join("\n"),
acceptance: ["Return the exact structured candidate contract."],
responseSchema: candidateSchema,
})
);
}
@@ -55,30 +100,52 @@ export default async function (args) {
phase("Review");
const review = await task({
description:
"Score candidates against one rubric; name a winner only with PASS evidence.",
label: "reviewer",
type: "review",
worktree: false,
prompt: [
"You are the tournament judge. Score every candidate against: correctness,",
"fit to the brief, simplicity, risk, and verification evidence.",
"Reject candidates without PASS command evidence for code work.",
"Name exactly one winner (or NONE if all fail), with decisive reasons.",
// Single driver-visible text; `label` carries the short progress name.
description: [
"You are the read-only tournament judge. Score every candidate against the frozen rubric.",
"Treat self_verdict and claimed commands as untrusted candidate statements.",
"Name one provisional winner_id, or NONE if all fail, with decisive reasons.",
"Do not merge or apply changes. Do not invent missing evidence.",
"Set verification_required=true for every code winner.",
"Return only the required structured response.",
"",
"candidates:",
"BRIEF:",
String(brief),
"",
"RUBRIC:",
String(rubric),
"",
"CANDIDATES:",
String(JSON.stringify(candidates, null, 2) ?? "(missing)"),
].join("\n"),
label: "reviewer",
type: "review",
writeAuthority: "read_only",
worktree: false,
responseSchema: {
type: "object",
additionalProperties: false,
required: ["winner_id", "ranking", "verification_required", "reasons"],
properties: {
winner_id: { type: "string" },
ranking: { type: "array", items: { type: "string" } },
verification_required: { type: "boolean" },
reasons: { type: "array", items: { type: "string" } },
},
},
});
return {
scenario: "operate-best-of-n",
scenario: strategy === "search" ? "operate-search" : "operate-best-of-n",
strategy,
n,
brief,
rubric,
candidates,
review,
apply_policy:
"Parent applies winner only after reviewer PASS + parent re-verify.",
"Parent applies a winner only after independent clean replay and explicit user approval.",
execution_boundary:
"This recipe generates and reviews candidates. It does not claim runtime-owned hidden gates, benchmark scoring, or clean-baseline replay; use a frozen WorkflowSearchSpec once the evaluator host is wired.",
};
}