feat(workflow): experimental-search authoring boundary, reshaped and verified
Land the in-flight experimental-search slice as a real, honest v0.9.4 piece: a provider-neutral WorkflowSearchSpec authoring + freeze boundary (parse/validate, preregistration hashes over baseline/requested+resolved model/public evidence/evaluator, deterministic candidate ids and admission batches) plus the operate best-of-N recipe's structured 'search' strategy (2-16 independent worktree candidates with responseSchema contracts and a read-only judge). The module explicitly remains an authoring boundary, not a runtime: hard_gates/score commands are parsed and validated only, and docs say so. Fixes found during the takeover operation: - The checked-in recipe test failed against the real driver contract (parse_task_options lets prompt win over description, so fake-driver needles never matched and replies fell back to non-JSON). The recipe now puts the full instruction in the single driver-visible description and pins the prompt-wins contract with a new test. - Queue claims verified against the runtime: the Workflow host's per-run concurrency gate (Semaphore, 16 live) is where larger populations wait; docs now name the gate instead of vague 'queues through Fleet'. WORKFLOW_SEARCH_MAX_CONCURRENT documents 16 as today's default with a cross-reference (a crate cycle prevents importing the host constant). - TournamentOrdering (ScoreThenCost) and the 1,000-agent validation test bump retained from the slice. Verified: cargo test -p codewhale-workflow -p codewhale-workflow-js --locked green (250 + 16 + 9 + 49).
This commit is contained in:
@@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- Added a provider-neutral `WorkflowSearchSpec` authoring and freeze boundary
|
||||
for experimental search inside Workflow, plus structured 2–16 candidate
|
||||
generation in the best-of-N starter. Searches bind the baseline, requested
|
||||
and resolved model ids, public evidence, evaluator identity, gates, scoring,
|
||||
budget, and review-only integration policy before admission.
|
||||
|
||||
### Changed
|
||||
|
||||
- Workflow scale documentation now matches the runtime: up to 1,000 tasks per
|
||||
run, admitted at most 16 live at a time (additional `task()` calls block on
|
||||
the host's per-run concurrency gate, then route through Fleet).
|
||||
`BranchTournament` keeps its historical cost-first default and supports
|
||||
explicit score-first ordering. Runtime-owned hidden gates, benchmark scoring,
|
||||
and clean replay remain an explicit host seam rather than being inferred from
|
||||
worker self-reports.
|
||||
|
||||
## [0.9.3] - 2026-07-31
|
||||
|
||||
This is the Codewhale v0.9.3 source candidate. It is not a published release
|
||||
|
||||
@@ -80,6 +80,7 @@ path.
|
||||
- [docs/PROVIDERS.md](docs/PROVIDERS.md) — every provider route: hosted,
|
||||
gateway, and local
|
||||
- [docs/FLEET.md](docs/FLEET.md) — fleets, the ledger, and resume
|
||||
- [docs/WORKFLOW_EXPERIMENTAL_SEARCH.md](docs/WORKFLOW_EXPERIMENTAL_SEARCH.md) — frozen, provider-neutral experimental search within Workflow
|
||||
- [docs/CONFIGURATION.md](docs/CONFIGURATION.md) — `config.toml`, hooks, and
|
||||
the constitution
|
||||
- [docs/AUTHORIZATION_ORDER.md](docs/AUTHORIZATION_ORDER.md) — how modes,
|
||||
|
||||
@@ -9,6 +9,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Added
|
||||
|
||||
- Added a Workflow experimental-search authoring foundation and structured
|
||||
search option to the bundled best-of-N recipe. This remains Workflow/Fleet,
|
||||
never a new mode or scheduler, and never auto-applies a winner.
|
||||
- `codewhale account login`, `status`, `logout`, and `keys` connect a CLI
|
||||
profile to the managed Codewhale account through the browser device flow.
|
||||
Sessions use the OS credential manager, remote BYOK values are never printed,
|
||||
|
||||
@@ -18,7 +18,10 @@ the user has already chosen the approach.
|
||||
1. Define one task, one evidence packet, and one explicit scoring rubric before
|
||||
launching candidates. Include correctness, fit to the request, simplicity,
|
||||
risk, and verification.
|
||||
2. Choose `N` from 2 to 4. Default to 3. More candidates need a concrete reason.
|
||||
2. Choose `N` from 2 to 4 for a quick comparison (default 3). For an explicit
|
||||
experimental search, use the Workflow search option: 2–16 live candidates,
|
||||
with larger validated populations queued at the Workflow host's 16-worker
|
||||
concurrency gate rather than launched at once.
|
||||
3. Give every candidate the same task and rubric. Add only a candidate number;
|
||||
do not steer candidates toward different conclusions unless diversity is an
|
||||
explicit part of the request.
|
||||
@@ -54,20 +57,21 @@ When candidates must implement code, give each one:
|
||||
- the same bounded `write_roots` or `exact_files`
|
||||
|
||||
Never run parallel writers in the parent checkout. Each implementer must return
|
||||
`VERDICT: PASS|FAIL` with command evidence (tests/lint) — a diff alone is not
|
||||
completion.
|
||||
the structured candidate contract (candidate id, hypothesis, paths, commands,
|
||||
self-verdict, risks, and artifact references). A self-verdict is evidence to
|
||||
inspect, not a hard-gate result.
|
||||
|
||||
Optional diversity: pin different `model` / Fleet `fleet_profile` values when
|
||||
the project has multiple capable routes; otherwise keep model strength `same`.
|
||||
|
||||
## Judge Once
|
||||
|
||||
Use one read-only reviewer/verifier worker, or the parent when the result is
|
||||
small, to score all candidates against the original rubric. The judge must:
|
||||
Use one read-only reviewer worker, or the parent when the result is small, to
|
||||
score all candidates against the original rubric. The judge must:
|
||||
|
||||
- cite evidence from each candidate rather than vote by style;
|
||||
- reject candidates that violate authority, scope, or verification gates;
|
||||
- reject any code candidate without PASS evidence (or re-verify it);
|
||||
- treat candidate-reported commands and PASS claims as untrusted until replay;
|
||||
- name the winner and the decisive reasons;
|
||||
- identify useful pieces worth combining, if any;
|
||||
- say when the candidates are tied or all fail.
|
||||
@@ -80,12 +84,22 @@ approaches into a new unreviewed solution.
|
||||
For proposal-only work, return the winning answer with a compact score summary.
|
||||
For code work:
|
||||
|
||||
1. Inspect the winning worktree diff.
|
||||
2. Apply/merge the winner into the parent checkout only after the judge (or a
|
||||
follow-up verifier) reports PASS with real checks.
|
||||
3. Re-run the repository's checks in the parent after apply. Candidate
|
||||
self-reports and style scores are not final verification.
|
||||
4. Leave losing worktrees unmerged; do not pollute the parent with partials.
|
||||
1. Freeze the baseline, evaluator, hard gates, score rule, and authority before
|
||||
a larger search admits candidates. Any evaluator change starts a revision.
|
||||
2. After a worker loses write authority, apply its patch to a clean baseline
|
||||
and let the runtime—not that worker—run hard gates and scoring.
|
||||
3. Inspect the winning worktree diff and independently replay it on the clean
|
||||
baseline. A different read-only model may look for gaming, but deterministic
|
||||
tests remain the authority.
|
||||
4. Present the verified winner for review. Applying or merging is a separate,
|
||||
explicit user action; `NONE` is valid when every candidate fails.
|
||||
5. Preserve losing and failed candidate receipts as useful negative results.
|
||||
|
||||
The checked-in `operate_best_of_n.workflow.js` recipe supports
|
||||
`strategy: "search"` for structured 2–16 candidate generation and review. It
|
||||
does **not** yet turn prompt-listed commands into hidden runtime gates. Do not
|
||||
advertise those gates until the runtime evaluator host consumes a frozen
|
||||
`WorkflowSearchSpec`.
|
||||
|
||||
Stop early when one candidate reveals a hard constraint that invalidates the
|
||||
tournament. Report the negative result rather than spending the remaining
|
||||
|
||||
@@ -57,6 +57,113 @@ async fn args_global_is_the_invocation_input() {
|
||||
assert_eq!(value, json!({"sum": 42, "tag": "release"}));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn checked_in_best_of_n_search_recipe_runs_with_structured_receipts() {
|
||||
let driver = Arc::new(FakeDriver::new());
|
||||
for index in 1..=2 {
|
||||
driver.on(
|
||||
// Rules match the driver-visible `TaskRequest.description`, which
|
||||
// is the full instruction text (the VM's `prompt` alias wins over
|
||||
// a short label). Match the unique per-candidate suffix line.
|
||||
&format!("candidate_id=cand_{index:03} of 2."),
|
||||
FakeReply::Complete(
|
||||
json!({
|
||||
"candidate_id": format!("cand_{index:03}"),
|
||||
"hypothesis": "bounded fixture",
|
||||
"modified_paths": ["src/lib.rs"],
|
||||
"commands_run": ["cargo test --locked"],
|
||||
"self_verdict": "pass",
|
||||
"known_risks": [],
|
||||
"artifact_refs": [format!("patch:cand_{index:03}")]
|
||||
})
|
||||
.to_string(),
|
||||
),
|
||||
);
|
||||
}
|
||||
driver.on(
|
||||
"read-only tournament judge",
|
||||
FakeReply::Complete(
|
||||
json!({
|
||||
"winner_id": "cand_001",
|
||||
"ranking": ["cand_001", "cand_002"],
|
||||
"verification_required": true,
|
||||
"reasons": ["fixture score"]
|
||||
})
|
||||
.to_string(),
|
||||
),
|
||||
);
|
||||
|
||||
let value = run(
|
||||
&driver,
|
||||
include_str!("../../../workflows/operate_best_of_n.workflow.js"),
|
||||
json!({
|
||||
"brief": "Implement the fixture",
|
||||
"strategy": "search",
|
||||
"n": 2,
|
||||
"writeRoots": ["src"],
|
||||
"model": "deepseek-v4-flash",
|
||||
"thinking": "max"
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.expect("checked-in search recipe should execute");
|
||||
|
||||
assert_eq!(value["scenario"], "operate-search");
|
||||
assert_eq!(value["review"]["winner_id"], "cand_001");
|
||||
assert_eq!(driver.spawn_count(), 3);
|
||||
let requests = driver.requests();
|
||||
assert_eq!(requests[0].model.as_deref(), Some("deepseek-v4-flash"));
|
||||
assert_eq!(requests[0].thinking.as_deref(), Some("max"));
|
||||
assert_eq!(requests[0].write_roots, ["src"]);
|
||||
assert_eq!(requests[2].write_authority.as_deref(), Some("read_only"));
|
||||
// Regression: the driver-visible description is the full instruction text,
|
||||
// so reply rules must target text that actually reaches the driver. If a
|
||||
// future recipe reintroduces a separate short `description` next to a long
|
||||
// `prompt`, these needles stop matching, the FakeDriver falls back to its
|
||||
// non-JSON "done:..." reply, and the structured receipts fail loudly.
|
||||
assert!(
|
||||
requests[0]
|
||||
.description
|
||||
.starts_with("You are one independent candidate")
|
||||
);
|
||||
assert!(
|
||||
requests[0]
|
||||
.description
|
||||
.contains("CANDIDATE-SPECIFIC INSTRUCTION: candidate_id=cand_001 of 2.")
|
||||
);
|
||||
assert!(
|
||||
requests[2]
|
||||
.description
|
||||
.starts_with("You are the read-only tournament judge")
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn task_prompt_wins_over_description_as_driver_visible_text() {
|
||||
let driver = Arc::new(FakeDriver::new());
|
||||
let value = run(
|
||||
&driver,
|
||||
r#"
|
||||
return await task({
|
||||
description: "short progress label",
|
||||
prompt: "the real instruction",
|
||||
});
|
||||
"#,
|
||||
json!(null),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
// No rules were registered, so the FakeDriver fallback echoes the
|
||||
// driver-visible description. The reply text proves the driver received
|
||||
// the prompt, not the short label.
|
||||
assert_eq!(value, json!("done:the real instruction"));
|
||||
let requests = driver.requests();
|
||||
assert_eq!(requests.len(), 1);
|
||||
assert_eq!(requests[0].description, "the real instruction");
|
||||
assert_ne!(requests[0].description, "short progress label");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn task_round_trip_carries_all_options_and_normalizes_profile() {
|
||||
let driver = Arc::new(FakeDriver::new());
|
||||
|
||||
@@ -0,0 +1,557 @@
|
||||
//! Provider-neutral experimental search authoring for Workflow.
|
||||
//!
|
||||
//! This module is an authoring and freeze boundary, not a new runtime or
|
||||
//! scheduler. A validated search still has to be lowered by the Workflow host
|
||||
//! into Fleet workers plus a runtime-owned evaluator. In particular, worker
|
||||
//! self-reports are never promoted to hard-gate evidence here.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sha2::{Digest, Sha256};
|
||||
use thiserror::Error;
|
||||
|
||||
use crate::{DEFAULT_FLEET_WORKFLOW_MAX_AGENTS, experimental_search::SearchSpecError::*};
|
||||
|
||||
pub const WORKFLOW_SEARCH_SCHEMA_VERSION: u32 = 1;
|
||||
/// Live-worker ceiling for one search admission batch.
|
||||
///
|
||||
/// 16 matches the Workflow host's live-child ceiling today
|
||||
/// (`codewhale_workflow_js::WORKFLOW_MAX_CONCURRENT`, from which the tui
|
||||
/// driver sizes its per-run admission semaphore). This crate cannot import
|
||||
/// that constant directly because `codewhale-workflow-js` depends on
|
||||
/// `codewhale-workflow`, so 16 is documented here as today's default — not a
|
||||
/// new configuration knob. Keep it in sync with the host constant.
|
||||
pub const WORKFLOW_SEARCH_MAX_CONCURRENT: u16 = 16;
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct WorkflowSearchSpec {
|
||||
#[serde(default = "default_schema_version")]
|
||||
pub schema_version: u32,
|
||||
pub name: String,
|
||||
pub objective: String,
|
||||
pub population: u16,
|
||||
pub rounds: Vec<u16>,
|
||||
pub concurrency: u16,
|
||||
pub worker: SearchWorkerSpec,
|
||||
#[serde(default)]
|
||||
pub budget: SearchBudgetSpec,
|
||||
pub hard_gates: SearchHardGateSpec,
|
||||
pub score: SearchScoreSpec,
|
||||
#[serde(default)]
|
||||
pub selection: SearchSelectionSpec,
|
||||
#[serde(default)]
|
||||
pub integration_policy: SearchIntegrationPolicy,
|
||||
}
|
||||
|
||||
impl WorkflowSearchSpec {
|
||||
pub fn from_toml(source: &str) -> Result<Self, SearchSpecError> {
|
||||
let spec: Self = toml::from_str(source).map_err(|error| Parse(error.to_string()))?;
|
||||
spec.validate()?;
|
||||
Ok(spec)
|
||||
}
|
||||
|
||||
pub fn validate(&self) -> Result<(), SearchSpecError> {
|
||||
if self.schema_version != WORKFLOW_SEARCH_SCHEMA_VERSION {
|
||||
return Err(UnsupportedSchemaVersion(self.schema_version));
|
||||
}
|
||||
validate_name(&self.name)?;
|
||||
validate_text("objective", &self.objective, 32_768)?;
|
||||
if !(2..=DEFAULT_FLEET_WORKFLOW_MAX_AGENTS as u16).contains(&self.population) {
|
||||
return Err(InvalidPopulation(self.population));
|
||||
}
|
||||
if self.concurrency == 0
|
||||
|| self.concurrency > WORKFLOW_SEARCH_MAX_CONCURRENT
|
||||
|| self.concurrency > self.population
|
||||
{
|
||||
return Err(InvalidConcurrency {
|
||||
concurrency: self.concurrency,
|
||||
population: self.population,
|
||||
});
|
||||
}
|
||||
validate_rounds(self.population, &self.rounds)?;
|
||||
validate_text("worker.model", &self.worker.model, 256)?;
|
||||
if self.worker.write_roots.is_empty() && self.worker.exact_files.is_empty() {
|
||||
return Err(UnboundedWriteScope);
|
||||
}
|
||||
validate_string_list("worker.write_roots", &self.worker.write_roots, 128, 1_024)?;
|
||||
validate_string_list("worker.exact_files", &self.worker.exact_files, 256, 1_024)?;
|
||||
if self.budget.max_cost_microusd == Some(0) {
|
||||
return Err(ZeroBudget("max_cost_microusd"));
|
||||
}
|
||||
if self.budget.max_tokens == Some(0) {
|
||||
return Err(ZeroBudget("max_tokens"));
|
||||
}
|
||||
if !self.hard_gates.forbid_test_changes {
|
||||
return Err(TestWeakeningAllowed);
|
||||
}
|
||||
if self.hard_gates.commands.is_empty() {
|
||||
return Err(MissingHardGates);
|
||||
}
|
||||
validate_string_list("hard_gates.commands", &self.hard_gates.commands, 32, 4_096)?;
|
||||
validate_string_list(
|
||||
"hard_gates.protected_paths",
|
||||
&self.hard_gates.protected_paths,
|
||||
256,
|
||||
1_024,
|
||||
)?;
|
||||
validate_text("score.command", &self.score.command, 4_096)?;
|
||||
validate_text("score.metric", &self.score.metric, 256)?;
|
||||
if !(1..=25).contains(&self.score.trials) {
|
||||
return Err(InvalidTrials(self.score.trials));
|
||||
}
|
||||
if self.score.tie_breakers.is_empty() {
|
||||
return Err(MissingTieBreakers);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Freeze the exact public inputs and evaluator identity before admission.
|
||||
/// The evaluator bytes are hashed, not exposed to generation workers.
|
||||
pub fn freeze(
|
||||
&self,
|
||||
baseline_commit: &str,
|
||||
resolved_model: &str,
|
||||
public_evidence: &[u8],
|
||||
evaluator: &[u8],
|
||||
) -> Result<FrozenWorkflowSearch, SearchSpecError> {
|
||||
self.validate()?;
|
||||
validate_commit(baseline_commit)?;
|
||||
validate_text("resolved_model", resolved_model, 256)?;
|
||||
if evaluator.is_empty() {
|
||||
return Err(EmptyEvaluator);
|
||||
}
|
||||
|
||||
let public_evidence_hash = sha256_label(public_evidence);
|
||||
let evaluator_hash = sha256_label(evaluator);
|
||||
let freeze_input = FreezeInput {
|
||||
spec: self,
|
||||
baseline_commit,
|
||||
requested_model: &self.worker.model,
|
||||
resolved_model,
|
||||
public_evidence_hash: &public_evidence_hash,
|
||||
evaluator_hash: &evaluator_hash,
|
||||
};
|
||||
let encoded =
|
||||
serde_json::to_vec(&freeze_input).map_err(|error| FreezeEncoding(error.to_string()))?;
|
||||
let preregistration_hash = sha256_label(&encoded);
|
||||
let search_id = format!("search-{}", &preregistration_hash[7..23]);
|
||||
|
||||
Ok(FrozenWorkflowSearch {
|
||||
schema_version: self.schema_version,
|
||||
search_id,
|
||||
baseline_commit: baseline_commit.to_string(),
|
||||
preregistration_hash,
|
||||
public_evidence_hash,
|
||||
evaluator_hash,
|
||||
requested_model: self.worker.model.clone(),
|
||||
resolved_model: resolved_model.to_string(),
|
||||
candidate_ids: self.candidate_ids(),
|
||||
})
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn candidate_ids(&self) -> Vec<String> {
|
||||
let width = self.population.to_string().len().max(3);
|
||||
(1..=self.population)
|
||||
.map(|index| format!("cand_{index:0width$}"))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Deterministic admission batches. Fleet owns actual scheduling and may
|
||||
/// run fewer workers when its configured pool or provider quota is lower.
|
||||
#[must_use]
|
||||
pub fn admission_batches(&self) -> Vec<Vec<String>> {
|
||||
self.candidate_ids()
|
||||
.chunks(usize::from(self.concurrency))
|
||||
.map(<[String]>::to_vec)
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct SearchWorkerSpec {
|
||||
#[serde(default)]
|
||||
pub provider: Option<String>,
|
||||
pub model: String,
|
||||
#[serde(default)]
|
||||
pub reasoning_effort: SearchReasoningEffort,
|
||||
#[serde(default)]
|
||||
pub write_authority: SearchWriteAuthority,
|
||||
#[serde(default)]
|
||||
pub write_roots: Vec<String>,
|
||||
#[serde(default)]
|
||||
pub exact_files: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum SearchReasoningEffort {
|
||||
Off,
|
||||
Low,
|
||||
Medium,
|
||||
#[default]
|
||||
High,
|
||||
Max,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum SearchWriteAuthority {
|
||||
#[default]
|
||||
WorktreeWrite,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Default)]
|
||||
pub struct SearchBudgetSpec {
|
||||
#[serde(default)]
|
||||
pub max_cost_microusd: Option<u64>,
|
||||
#[serde(default)]
|
||||
pub max_tokens: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct SearchHardGateSpec {
|
||||
pub commands: Vec<String>,
|
||||
#[serde(default = "default_true")]
|
||||
pub forbid_test_changes: bool,
|
||||
#[serde(default)]
|
||||
pub protected_paths: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct SearchScoreSpec {
|
||||
pub command: String,
|
||||
pub metric: String,
|
||||
#[serde(default)]
|
||||
pub direction: SearchDirection,
|
||||
#[serde(default = "default_trials")]
|
||||
pub trials: u16,
|
||||
#[serde(default)]
|
||||
pub tie_breakers: Vec<SearchTieBreaker>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum SearchDirection {
|
||||
#[default]
|
||||
Minimize,
|
||||
Maximize,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum SearchTieBreaker {
|
||||
DiffLines,
|
||||
CostMicrousd,
|
||||
Score,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct SearchSelectionSpec {
|
||||
#[serde(default)]
|
||||
pub policy: SearchSelectionPolicy,
|
||||
#[serde(default = "default_true")]
|
||||
pub retain_diversity: bool,
|
||||
#[serde(default)]
|
||||
pub ordering: Vec<SearchSelectionMetric>,
|
||||
}
|
||||
|
||||
impl Default for SearchSelectionSpec {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
policy: SearchSelectionPolicy::Pareto,
|
||||
retain_diversity: true,
|
||||
ordering: Vec::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum SearchSelectionPolicy {
|
||||
#[default]
|
||||
Pareto,
|
||||
Ordered,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum SearchSelectionMetric {
|
||||
Score,
|
||||
Runtime,
|
||||
DiffLines,
|
||||
CostMicrousd,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum SearchIntegrationPolicy {
|
||||
/// Produce a verified, reviewable winner or NONE. Never apply or merge it.
|
||||
#[default]
|
||||
ReviewOnly,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct FrozenWorkflowSearch {
|
||||
pub schema_version: u32,
|
||||
pub search_id: String,
|
||||
pub baseline_commit: String,
|
||||
pub preregistration_hash: String,
|
||||
pub public_evidence_hash: String,
|
||||
pub evaluator_hash: String,
|
||||
pub requested_model: String,
|
||||
pub resolved_model: String,
|
||||
pub candidate_ids: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct FreezeInput<'a> {
|
||||
spec: &'a WorkflowSearchSpec,
|
||||
baseline_commit: &'a str,
|
||||
requested_model: &'a str,
|
||||
resolved_model: &'a str,
|
||||
public_evidence_hash: &'a str,
|
||||
evaluator_hash: &'a str,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Error)]
|
||||
pub enum SearchSpecError {
|
||||
#[error("failed to parse Workflow search TOML: {0}")]
|
||||
Parse(String),
|
||||
#[error("unsupported Workflow search schema version {0}")]
|
||||
UnsupportedSchemaVersion(u32),
|
||||
#[error("search name must be a 1-96 character lowercase token")]
|
||||
InvalidName,
|
||||
#[error("{field} must be non-empty and no longer than {max} characters")]
|
||||
InvalidText { field: &'static str, max: usize },
|
||||
#[error("population {0} must be between 2 and 1000")]
|
||||
InvalidPopulation(u16),
|
||||
#[error(
|
||||
"concurrency {concurrency} must be between 1 and 16 and no greater than population {population}"
|
||||
)]
|
||||
InvalidConcurrency { concurrency: u16, population: u16 },
|
||||
#[error("rounds must start at population, decrease strictly, and end at 1")]
|
||||
InvalidRounds,
|
||||
#[error("write-capable search workers require write_roots or exact_files")]
|
||||
UnboundedWriteScope,
|
||||
#[error("{field} contains an empty, oversized, or duplicate entry")]
|
||||
InvalidStringList { field: &'static str },
|
||||
#[error("{0} must be greater than zero when set")]
|
||||
ZeroBudget(&'static str),
|
||||
#[error("experimental search must forbid test changes")]
|
||||
TestWeakeningAllowed,
|
||||
#[error("experimental search requires at least one runtime-owned hard-gate command")]
|
||||
MissingHardGates,
|
||||
#[error("score.trials must be between 1 and 25, got {0}")]
|
||||
InvalidTrials(u16),
|
||||
#[error("experimental search requires at least one deterministic tie-breaker")]
|
||||
MissingTieBreakers,
|
||||
#[error("baseline_commit must be a 7-64 character hexadecimal commit id")]
|
||||
InvalidBaselineCommit,
|
||||
#[error("evaluator bytes must be non-empty")]
|
||||
EmptyEvaluator,
|
||||
#[error("failed to encode frozen Workflow search: {0}")]
|
||||
FreezeEncoding(String),
|
||||
}
|
||||
|
||||
fn default_schema_version() -> u32 {
|
||||
WORKFLOW_SEARCH_SCHEMA_VERSION
|
||||
}
|
||||
|
||||
fn default_trials() -> u16 {
|
||||
5
|
||||
}
|
||||
|
||||
fn default_true() -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
fn validate_name(value: &str) -> Result<(), SearchSpecError> {
|
||||
if value.is_empty()
|
||||
|| value.len() > 96
|
||||
|| !value
|
||||
.bytes()
|
||||
.all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || b"-_".contains(&byte))
|
||||
{
|
||||
return Err(InvalidName);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_text(field: &'static str, value: &str, max: usize) -> Result<(), SearchSpecError> {
|
||||
if value.trim().is_empty() || value.chars().count() > max {
|
||||
return Err(InvalidText { field, max });
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_rounds(population: u16, rounds: &[u16]) -> Result<(), SearchSpecError> {
|
||||
if rounds.len() < 2
|
||||
|| rounds.first() != Some(&population)
|
||||
|| rounds.last() != Some(&1)
|
||||
|| rounds.windows(2).any(|pair| pair[0] <= pair[1])
|
||||
{
|
||||
return Err(InvalidRounds);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_string_list(
|
||||
field: &'static str,
|
||||
values: &[String],
|
||||
max_items: usize,
|
||||
max_chars: usize,
|
||||
) -> Result<(), SearchSpecError> {
|
||||
if values.len() > max_items
|
||||
|| values
|
||||
.iter()
|
||||
.any(|value| value.trim().is_empty() || value.chars().count() > max_chars)
|
||||
|| values
|
||||
.iter()
|
||||
.enumerate()
|
||||
.any(|(index, value)| values[..index].contains(value))
|
||||
{
|
||||
return Err(InvalidStringList { field });
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_commit(value: &str) -> Result<(), SearchSpecError> {
|
||||
if !(7..=64).contains(&value.len()) || !value.bytes().all(|byte| byte.is_ascii_hexdigit()) {
|
||||
return Err(InvalidBaselineCommit);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn sha256_label(bytes: &[u8]) -> String {
|
||||
let digest = Sha256::digest(bytes);
|
||||
let mut output = String::with_capacity(71);
|
||||
output.push_str("sha256:");
|
||||
for byte in digest {
|
||||
use std::fmt::Write as _;
|
||||
let _ = write!(output, "{byte:02x}");
|
||||
}
|
||||
output
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
const SPEC: &str = r#"
|
||||
name = "speed-up-certificate"
|
||||
objective = "Reduce runtime without changing exact results"
|
||||
population = 32
|
||||
rounds = [32, 8, 3, 1]
|
||||
concurrency = 16
|
||||
integration_policy = "review_only"
|
||||
|
||||
[worker]
|
||||
provider = "deepseek"
|
||||
model = "deepseek-v4-flash"
|
||||
reasoning_effort = "max"
|
||||
write_authority = "worktree_write"
|
||||
write_roots = ["code"]
|
||||
|
||||
[budget]
|
||||
max_cost_microusd = 5000000
|
||||
max_tokens = 10000000
|
||||
|
||||
[hard_gates]
|
||||
commands = ["cargo test --locked", "git diff --exit-code -- expected.json"]
|
||||
forbid_test_changes = true
|
||||
protected_paths = ["tests", "expected.json"]
|
||||
|
||||
[score]
|
||||
command = "./scripts/benchmark_candidate.sh"
|
||||
direction = "minimize"
|
||||
metric = "median_runtime_ms"
|
||||
trials = 5
|
||||
tie_breakers = ["diff_lines", "cost_microusd"]
|
||||
|
||||
[selection]
|
||||
policy = "pareto"
|
||||
retain_diversity = true
|
||||
"#;
|
||||
|
||||
#[test]
|
||||
fn parses_valid_search_and_queues_through_live_cap() {
|
||||
let spec = WorkflowSearchSpec::from_toml(SPEC).expect("valid search spec");
|
||||
|
||||
assert_eq!(spec.population, 32);
|
||||
assert_eq!(spec.admission_batches().len(), 2);
|
||||
assert_eq!(spec.admission_batches()[0].len(), 16);
|
||||
assert_eq!(spec.candidate_ids()[0], "cand_001");
|
||||
assert_eq!(spec.candidate_ids()[31], "cand_032");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn freeze_is_deterministic_and_model_version_sensitive() {
|
||||
let spec = WorkflowSearchSpec::from_toml(SPEC).expect("valid search spec");
|
||||
let first = spec
|
||||
.freeze(
|
||||
"33bc6a98",
|
||||
"DeepSeek-V4-Flash-0731",
|
||||
b"public evidence",
|
||||
b"private evaluator",
|
||||
)
|
||||
.expect("freeze succeeds");
|
||||
let replay = spec
|
||||
.freeze(
|
||||
"33bc6a98",
|
||||
"DeepSeek-V4-Flash-0731",
|
||||
b"public evidence",
|
||||
b"private evaluator",
|
||||
)
|
||||
.expect("freeze succeeds");
|
||||
let drifted = spec
|
||||
.freeze(
|
||||
"33bc6a98",
|
||||
"DeepSeek-V4-Flash-next",
|
||||
b"public evidence",
|
||||
b"private evaluator",
|
||||
)
|
||||
.expect("freeze succeeds");
|
||||
|
||||
assert_eq!(first, replay);
|
||||
assert_ne!(first.search_id, drifted.search_id);
|
||||
assert_ne!(first.preregistration_hash, drifted.preregistration_hash);
|
||||
assert_eq!(first.requested_model, "deepseek-v4-flash");
|
||||
assert_eq!(first.resolved_model, "DeepSeek-V4-Flash-0731");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_unsafe_or_unbounded_searches() {
|
||||
let mut spec = WorkflowSearchSpec::from_toml(SPEC).expect("valid search spec");
|
||||
spec.hard_gates.forbid_test_changes = false;
|
||||
assert_eq!(spec.validate(), Err(SearchSpecError::TestWeakeningAllowed));
|
||||
|
||||
spec.hard_gates.forbid_test_changes = true;
|
||||
spec.worker.write_roots.clear();
|
||||
assert_eq!(spec.validate(), Err(SearchSpecError::UnboundedWriteScope));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_invalid_rounds_and_excess_live_concurrency() {
|
||||
let mut spec = WorkflowSearchSpec::from_toml(SPEC).expect("valid search spec");
|
||||
spec.rounds = vec![32, 8, 8, 1];
|
||||
assert_eq!(spec.validate(), Err(SearchSpecError::InvalidRounds));
|
||||
|
||||
spec.rounds = vec![32, 1];
|
||||
spec.concurrency = 17;
|
||||
assert_eq!(
|
||||
spec.validate(),
|
||||
Err(SearchSpecError::InvalidConcurrency {
|
||||
concurrency: 17,
|
||||
population: 32,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deserialization_refuses_auto_merge_policy() {
|
||||
let source = SPEC.replace("review_only", "auto_merge");
|
||||
let error = WorkflowSearchSpec::from_toml(&source).expect_err("must reject auto merge");
|
||||
|
||||
assert!(matches!(error, SearchSpecError::Parse(_)));
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@
|
||||
//! top only after their cancellation and evidence semantics are proven.
|
||||
|
||||
mod elevation;
|
||||
pub mod experimental_search;
|
||||
/// Setup-time Fleet composition: a suggestion schema with no runtime authority.
|
||||
///
|
||||
/// Deliberately **not** re-exported from the crate root. Nothing consumes it
|
||||
@@ -1638,18 +1639,35 @@ fn truncate_evidence(value: &str) -> String {
|
||||
pub struct BranchTournament {
|
||||
#[serde(default)]
|
||||
pub min_score: u32,
|
||||
#[serde(default)]
|
||||
pub ordering: TournamentOrdering,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum TournamentOrdering {
|
||||
/// Historical behavior: choose the cheapest passing branch, then score.
|
||||
#[default]
|
||||
CostThenScore,
|
||||
/// Quality-first behavior for explicitly configured evaluation workflows.
|
||||
ScoreThenCost,
|
||||
}
|
||||
|
||||
impl BranchTournament {
|
||||
pub fn select(&self, candidates: &[BranchCandidate]) -> Option<BranchCandidate> {
|
||||
candidates
|
||||
.iter()
|
||||
.filter(|candidate| {
|
||||
let passing = || {
|
||||
candidates.iter().filter(|candidate| {
|
||||
candidate.status == WorkflowRunStatus::Succeeded
|
||||
&& candidate.score >= self.min_score
|
||||
})
|
||||
.min_by_key(|candidate| (candidate.cost, std::cmp::Reverse(candidate.score)))
|
||||
.cloned()
|
||||
};
|
||||
match self.ordering {
|
||||
TournamentOrdering::CostThenScore => passing()
|
||||
.min_by_key(|candidate| (candidate.cost, std::cmp::Reverse(candidate.score))),
|
||||
TournamentOrdering::ScoreThenCost => passing()
|
||||
.min_by_key(|candidate| (std::cmp::Reverse(candidate.score), candidate.cost)),
|
||||
}
|
||||
.cloned()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2790,7 +2808,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fleet_validation_accepts_one_hundred_agents_and_variable_models() {
|
||||
fn fleet_validation_accepts_one_thousand_agents_and_variable_models() {
|
||||
let nodes = (0..DEFAULT_FLEET_WORKFLOW_MAX_AGENTS)
|
||||
.map(|index| {
|
||||
let mut leaf = match leaf_node(&format!("agent-{index}")) {
|
||||
@@ -2817,14 +2835,14 @@ mod tests {
|
||||
|
||||
let shape = workflow
|
||||
.validate_for_fleet()
|
||||
.expect("one hundred agents should fit the Fleet Workflow limit");
|
||||
.expect("one thousand agents should fit the Fleet Workflow limit");
|
||||
|
||||
assert_eq!(shape.total_agents, DEFAULT_FLEET_WORKFLOW_MAX_AGENTS);
|
||||
assert_eq!(shape.max_depth, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fleet_validation_rejects_more_than_one_hundred_agents() {
|
||||
fn fleet_validation_rejects_more_than_one_thousand_agents() {
|
||||
let nodes = (0..=DEFAULT_FLEET_WORKFLOW_MAX_AGENTS)
|
||||
.map(|index| leaf_node(&format!("agent-{index}")))
|
||||
.collect();
|
||||
@@ -4052,7 +4070,10 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn tournament_selects_passing_minimal_branch() {
|
||||
let tournament = BranchTournament { min_score: 60 };
|
||||
let tournament = BranchTournament {
|
||||
min_score: 60,
|
||||
ordering: TournamentOrdering::CostThenScore,
|
||||
};
|
||||
let candidates = vec![
|
||||
candidate(
|
||||
"expensive-pass",
|
||||
@@ -4079,6 +4100,24 @@ mod tests {
|
||||
assert_eq!(selected.branch_id, "cheap-pass");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tournament_can_select_score_before_cost_explicitly() {
|
||||
let tournament = BranchTournament {
|
||||
min_score: 60,
|
||||
ordering: TournamentOrdering::ScoreThenCost,
|
||||
};
|
||||
let candidates = vec![
|
||||
candidate("quality", WorkflowRunStatus::Succeeded, 95, 100, "quality"),
|
||||
candidate("minimal", WorkflowRunStatus::Succeeded, 70, 10, "minimal"),
|
||||
];
|
||||
|
||||
let selected = tournament
|
||||
.select(&candidates)
|
||||
.expect("one passing branch should be selected");
|
||||
|
||||
assert_eq!(selected.branch_id, "quality");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pareto_frontier_keeps_diverse_candidates() {
|
||||
let frontier = ParetoFrontier { max_items: 4 };
|
||||
|
||||
+6
-3
@@ -322,13 +322,16 @@ perform real work as `codewhale exec` processes.
|
||||
|
||||
Default Workflow-to-Fleet validation is intentionally bounded:
|
||||
|
||||
- 100 total worker agents per workflow run;
|
||||
- 5 recursive Fleet rings;
|
||||
- 1,000 total worker agents per Workflow run;
|
||||
- 16 live worker agents at once; larger populations queue (block) on the host's
|
||||
per-run concurrency gate until a live slot frees, then route through Fleet;
|
||||
- 5 recursive Fleet rings as the hard ceiling (default user configuration: 2);
|
||||
- bounded loops only (`max_iterations` required);
|
||||
- bounded dynamic expansion only (`max_children` plus a template required).
|
||||
|
||||
These are population limits, not a demand to launch everything at once. A
|
||||
100-agent workflow should still drain through the configured Fleet worker pool.
|
||||
1,000-agent Workflow should still drain through the configured Fleet worker
|
||||
pool.
|
||||
Recommended model layouts, such as a DeepSeek Pro orchestrator with Flash
|
||||
workers in the first ring and cheaper workers farther out, are presets only.
|
||||
Every slot can inherit the active model or carry an explicit model override.
|
||||
|
||||
@@ -123,12 +123,26 @@ Planned scope:
|
||||
|
||||
## Eval Workset
|
||||
|
||||
Implemented authoring foundation:
|
||||
|
||||
- Provider-neutral `WorkflowSearchSpec` validation and deterministic freeze
|
||||
receipts for an experimental-search option within Workflow. The freeze binds
|
||||
the baseline, requested and resolved model names, public evidence, and
|
||||
evaluator identity before candidate admission.
|
||||
- The best-of-N Workflow starter can generate 2–16 structured, independent
|
||||
worktree candidates with cache-stable shared instructions and a read-only
|
||||
review. This is generation/review evidence, not runtime-owned hard-gate proof.
|
||||
|
||||
Planned scope:
|
||||
|
||||
- Reproducible task suites for coding, review, docs, release checks, and
|
||||
long-context workflows.
|
||||
- Side-by-side route comparisons where the exact model, provider, thinking
|
||||
level, prompt, and tool policy are captured.
|
||||
- Runtime-owned hard gates and command scoring after worker write authority is
|
||||
revoked; clean-baseline replay; duplicate-patch detection; multi-round
|
||||
Pareto/diversity promotion; aggregate receipts over Fleet receipts; and a
|
||||
Workflow-panel leaderboard. No winner is applied or merged automatically.
|
||||
|
||||
## Observability Workset
|
||||
|
||||
|
||||
@@ -123,13 +123,30 @@ or filesystem authority.
|
||||
Fleet launch validation applies a conservative default shape before any
|
||||
Workflow IR is lowered to workers:
|
||||
|
||||
- up to 100 total worker agents per workflow run;
|
||||
- up to 5 recursive Fleet rings;
|
||||
- up to 1,000 total worker agents per Workflow run;
|
||||
- up to 16 live worker agents at once; larger populations queue (block) on the
|
||||
host's per-run concurrency gate until a live slot frees, then route through
|
||||
Fleet;
|
||||
- up to 5 recursive Fleet rings (the default user configuration is 2);
|
||||
- loops require `max_iterations`;
|
||||
- dynamic `expand` nodes require `max_children` and a template.
|
||||
|
||||
Those limits bound the workflow population, not instantaneous launch
|
||||
concurrency. A valid 100-agent workflow can still drain through a smaller Fleet
|
||||
Those limits distinguish population from instantaneous launch concurrency. A
|
||||
valid 1,000-agent Workflow can still drain through a smaller Fleet
|
||||
worker pool. Model selection stays per slot: a DeepSeek preset can suggest
|
||||
`deepseek-v4-pro` for the orchestrator and `deepseek-v4-flash` for nearby
|
||||
workers, but users and agents may override any slot when the task calls for it.
|
||||
|
||||
## Experimental search is a Workflow option
|
||||
|
||||
Experimental search generalizes the existing best-of-N recipe without adding a
|
||||
new product mode, scheduler, or sub-agent API. A provider-neutral
|
||||
`WorkflowSearchSpec` freezes the objective, baseline, model request and resolved
|
||||
version, public evidence, evaluator hash, hard gates, scoring rule, budgets,
|
||||
write scope, rounds, and review-only integration policy before admission.
|
||||
|
||||
The current JS starter supports structured generation and read-only review with
|
||||
`strategy: "search"`. Runtime-owned command gates, hidden evaluation, benchmark
|
||||
scoring, and clean-baseline replay are an explicit host seam still to wire; a
|
||||
candidate's self-verdict must never be promoted into evaluator truth. See
|
||||
[Workflow Experimental Search](WORKFLOW_EXPERIMENTAL_SEARCH.md).
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
# Workflow Experimental Search
|
||||
|
||||
Experimental search is an option for **Workflow**, not a fourth mode and not a
|
||||
second scheduler. Fleet still owns workers, routes, concurrency, leases, and
|
||||
receipts. Workflow owns the frozen order: independent generation, evaluation,
|
||||
selection, repair, and verification.
|
||||
|
||||
The product vocabulary is Fleet (who), Workflow (what order), Lane (one running
|
||||
Workflow), Runtime (where/how), and Operate (the mode). Do not introduce
|
||||
"WhaleFlow" as a current synonym.
|
||||
|
||||
## Current foundation
|
||||
|
||||
- The imperative Workflow VM admits at most 1,000 tasks per run and 16 live
|
||||
tasks at once. The host's per-run concurrency gate is a semaphore sized
|
||||
`WORKFLOW_MAX_CONCURRENT = 16` (`codewhale-workflow-js`); additional
|
||||
`task()` spawns block on that gate until a live slot frees, then route
|
||||
through Fleet. A larger declared population therefore queues at the gate,
|
||||
not through Fleet itself.
|
||||
- `WorkflowSearchSpec` is a provider-neutral TOML/Rust authoring boundary. It
|
||||
validates bounded worktree writes, rounds, budgets, mandatory anti-test-
|
||||
weakening posture, hard-gate commands, deterministic scoring, selection, and
|
||||
review-only integration.
|
||||
- `hard_gates.commands` and `score.command` are parsed and validated only;
|
||||
nothing in this slice executes them. Runtime-owned gate execution and
|
||||
benchmark scoring are the evaluator host seam described below.
|
||||
- Freezing a spec records one deterministic search id and preregistration hash
|
||||
over the baseline commit, requested and resolved model ids, public evidence
|
||||
hash, evaluator hash, and the complete spec.
|
||||
- `operate_best_of_n.workflow.js` supports `strategy: "search"` for 2–16
|
||||
structured independent candidates and one read-only reviewer. The stable
|
||||
shared instructions precede the candidate-specific suffix to favor provider
|
||||
prefix caching.
|
||||
- `BranchTournament` preserves its historical cost-first default but now
|
||||
supports explicit score-first ordering. Pareto selection remains available
|
||||
in the typed Workflow core.
|
||||
|
||||
## Security and truth boundary
|
||||
|
||||
The JS starter does not own the shell or evaluator. It therefore cannot turn a
|
||||
command mentioned in a prompt into a hidden, runtime-owned gate. Candidate
|
||||
self-verdicts and claimed commands are untrusted. Until the evaluator host seam
|
||||
lands, the starter produces generation and review evidence only.
|
||||
|
||||
The evaluator host must:
|
||||
|
||||
1. freeze a real Git baseline and evaluator before admitting candidates;
|
||||
2. give every writer its own worktree and the same public evidence;
|
||||
3. revoke writer authority before injecting hidden tests or scorer details;
|
||||
4. apply each patch to a clean baseline, reject forbidden/test changes, then
|
||||
run hard gates before performance scoring;
|
||||
5. record commands, exit codes, environment, token/cache/cost usage, artifacts,
|
||||
promotion reasons, and failures on top of Fleet receipts;
|
||||
6. replay the provisional winner cleanly and run an independent read-only
|
||||
adversarial review; and
|
||||
7. return `NONE` when all candidates fail and never apply or merge a winner
|
||||
without a later explicit user action.
|
||||
|
||||
## Provider presets
|
||||
|
||||
The abstraction remains provider-neutral. A DeepSeek Flash preset can exploit
|
||||
its automatically managed prefix cache by keeping shared instructions,
|
||||
experiment rules, repository evidence, and the response contract stable, with
|
||||
the candidate id last. Preliminary scouts can use lower effort while promoted
|
||||
implementers/finalists use high or max effort.
|
||||
|
||||
Record both the requested API model id and the resolved provider version.
|
||||
Provider account concurrency is not Fleet concurrency: the runtime keeps its
|
||||
16-live-worker ceiling, handles 429 responses and keep-alives outside the
|
||||
deterministic VM, and stops new admissions when the shared budget is exhausted.
|
||||
|
||||
## Example authoring shape
|
||||
|
||||
```toml
|
||||
name = "speed-up-certificate"
|
||||
objective = "Reduce runtime without changing exact results"
|
||||
population = 32
|
||||
rounds = [32, 8, 3, 1]
|
||||
concurrency = 16
|
||||
integration_policy = "review_only"
|
||||
|
||||
[worker]
|
||||
provider = "deepseek"
|
||||
model = "deepseek-v4-flash"
|
||||
reasoning_effort = "high"
|
||||
write_authority = "worktree_write"
|
||||
write_roots = ["code"]
|
||||
|
||||
[budget]
|
||||
max_cost_microusd = 5000000
|
||||
max_tokens = 10000000
|
||||
|
||||
[hard_gates]
|
||||
commands = [
|
||||
"PYTHONWARNINGS=error python certificate.py",
|
||||
"git diff --exit-code -- expected_result.json",
|
||||
]
|
||||
forbid_test_changes = true
|
||||
protected_paths = ["tests", "expected_result.json"]
|
||||
|
||||
[score]
|
||||
command = "./scripts/benchmark_candidate.sh"
|
||||
direction = "minimize"
|
||||
metric = "median_runtime_ms"
|
||||
trials = 5
|
||||
tie_breakers = ["diff_lines", "cost_microusd"]
|
||||
|
||||
[selection]
|
||||
policy = "pareto"
|
||||
retain_diversity = true
|
||||
```
|
||||
|
||||
This file is authoring input, not yet a runnable CLI promise. The next runtime
|
||||
slice is the evaluator host and aggregate receipt; after that, the natural-
|
||||
language authoring layer can safely compile a user's request into this shape.
|
||||
@@ -1,21 +1,56 @@
|
||||
/**
|
||||
* Operate starter — best-of-N implementers in worktrees, then reviewer.
|
||||
* Operate starter — independent worktree candidates, then one reviewer.
|
||||
*
|
||||
* Spawns N worktree implementers with the same brief, then a read-only
|
||||
* reviewer that picks a winner. The parent must apply the winner only after
|
||||
* PASS evidence (skill: best-of-n).
|
||||
* Set strategy="search" for a bounded 2-16 candidate search. This remains a
|
||||
* Workflow recipe, not a new mode or scheduler. Runtime-owned command gates
|
||||
* and clean-baseline scoring require the typed search/evaluator host seam.
|
||||
*
|
||||
* Run: /workflow run workflows/operate_best_of_n.workflow.js
|
||||
* Args: { brief, n?, targetFiles?, writeRoots? }
|
||||
* Args: { brief, n?, strategy?, rubric?, model?, thinking?, targetFiles?, writeRoots? }
|
||||
*/
|
||||
export default async function (args) {
|
||||
const brief =
|
||||
args?.brief ??
|
||||
args?.task ??
|
||||
"Propose and implement the smallest correct fix for the open failure.";
|
||||
const n = Math.min(4, Math.max(2, Number(args?.n ?? 3) || 3));
|
||||
const strategy = args?.strategy === "search" ? "search" : "best_of_n";
|
||||
const maxCandidates = strategy === "search" ? 16 : 4;
|
||||
const defaultCandidates = strategy === "search" ? 8 : 3;
|
||||
const n = Math.min(
|
||||
maxCandidates,
|
||||
Math.max(2, Number(args?.n ?? defaultCandidates) || defaultCandidates)
|
||||
);
|
||||
const exactFiles = Array.isArray(args?.targetFiles) ? args.targetFiles : [];
|
||||
const writeRoots = Array.isArray(args?.writeRoots) ? args.writeRoots : [];
|
||||
const rubric =
|
||||
args?.rubric ??
|
||||
"Correctness first; then fit, measured quality, simplicity, risk, and verification evidence.";
|
||||
const model = typeof args?.model === "string" ? args.model : undefined;
|
||||
const thinking =
|
||||
typeof args?.thinking === "string" ? args.thinking : undefined;
|
||||
|
||||
const candidateSchema = {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
required: [
|
||||
"candidate_id",
|
||||
"hypothesis",
|
||||
"modified_paths",
|
||||
"commands_run",
|
||||
"self_verdict",
|
||||
"known_risks",
|
||||
"artifact_refs",
|
||||
],
|
||||
properties: {
|
||||
candidate_id: { type: "string" },
|
||||
hypothesis: { type: "string" },
|
||||
modified_paths: { type: "array", items: { type: "string" } },
|
||||
commands_run: { type: "array", items: { type: "string" } },
|
||||
self_verdict: { type: "string", enum: ["pass", "fail"] },
|
||||
known_risks: { type: "array", items: { type: "string" } },
|
||||
artifact_refs: { type: "array", items: { type: "string" } },
|
||||
},
|
||||
};
|
||||
|
||||
phase("Candidates");
|
||||
const candidateFns = [];
|
||||
@@ -23,9 +58,30 @@ export default async function (args) {
|
||||
const index = i;
|
||||
candidateFns.push(() =>
|
||||
task({
|
||||
description: `Best-of-N candidate ${index}/${n}`,
|
||||
// The VM delivers one text to the driver: `prompt` (alias) wins over
|
||||
// `description`, so the full instruction lives in `description` and
|
||||
// `label` carries the short progress name. A separate short
|
||||
// `description` would never reach the driver.
|
||||
description: [
|
||||
"You are one independent candidate in a Codewhale Workflow search.",
|
||||
"Implement the same frozen brief and rubric in this isolated worktree only.",
|
||||
"Do not inspect other candidates, rankings, hidden tests, or evaluator internals.",
|
||||
"Do not push. Do not merge. Do not touch the parent checkout.",
|
||||
"Your self_verdict is informational; only runtime-owned evaluation can pass a hard gate.",
|
||||
"Return only the required structured response.",
|
||||
"",
|
||||
"BRIEF:",
|
||||
String(brief),
|
||||
"",
|
||||
"RUBRIC:",
|
||||
String(rubric),
|
||||
"",
|
||||
`CANDIDATE-SPECIFIC INSTRUCTION: candidate_id=cand_${String(index).padStart(3, "0")} of ${n}.`,
|
||||
].join("\n"),
|
||||
label: `candidate_${index}`,
|
||||
type: "implementer",
|
||||
...(model ? { model } : {}),
|
||||
...(thinking ? { thinking } : {}),
|
||||
worktree: true,
|
||||
writeAuthority: "worktree_write",
|
||||
...(exactFiles.length ? { exactFiles } : {}),
|
||||
@@ -35,19 +91,8 @@ export default async function (args) {
|
||||
"Do not share other candidates' answers.",
|
||||
"Parent checkout must remain unchanged until apply.",
|
||||
],
|
||||
acceptance: [
|
||||
"Return VERDICT PASS/FAIL with command evidence.",
|
||||
"List every modified path with a one-line why.",
|
||||
],
|
||||
prompt: [
|
||||
`You are candidate ${index} of ${n} in a best-of-N tournament.`,
|
||||
"Implement the brief below in this isolated worktree only.",
|
||||
"Run relevant checks. End with VERDICT: PASS|FAIL, COMMANDS, EVIDENCE.",
|
||||
"Do not push. Do not merge. Do not touch the parent checkout.",
|
||||
"",
|
||||
"BRIEF:",
|
||||
String(brief),
|
||||
].join("\n"),
|
||||
acceptance: ["Return the exact structured candidate contract."],
|
||||
responseSchema: candidateSchema,
|
||||
})
|
||||
);
|
||||
}
|
||||
@@ -55,30 +100,52 @@ export default async function (args) {
|
||||
|
||||
phase("Review");
|
||||
const review = await task({
|
||||
description:
|
||||
"Score candidates against one rubric; name a winner only with PASS evidence.",
|
||||
label: "reviewer",
|
||||
type: "review",
|
||||
worktree: false,
|
||||
prompt: [
|
||||
"You are the tournament judge. Score every candidate against: correctness,",
|
||||
"fit to the brief, simplicity, risk, and verification evidence.",
|
||||
"Reject candidates without PASS command evidence for code work.",
|
||||
"Name exactly one winner (or NONE if all fail), with decisive reasons.",
|
||||
// Single driver-visible text; `label` carries the short progress name.
|
||||
description: [
|
||||
"You are the read-only tournament judge. Score every candidate against the frozen rubric.",
|
||||
"Treat self_verdict and claimed commands as untrusted candidate statements.",
|
||||
"Name one provisional winner_id, or NONE if all fail, with decisive reasons.",
|
||||
"Do not merge or apply changes. Do not invent missing evidence.",
|
||||
"Set verification_required=true for every code winner.",
|
||||
"Return only the required structured response.",
|
||||
"",
|
||||
"candidates:",
|
||||
"BRIEF:",
|
||||
String(brief),
|
||||
"",
|
||||
"RUBRIC:",
|
||||
String(rubric),
|
||||
"",
|
||||
"CANDIDATES:",
|
||||
String(JSON.stringify(candidates, null, 2) ?? "(missing)"),
|
||||
].join("\n"),
|
||||
label: "reviewer",
|
||||
type: "review",
|
||||
writeAuthority: "read_only",
|
||||
worktree: false,
|
||||
responseSchema: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
required: ["winner_id", "ranking", "verification_required", "reasons"],
|
||||
properties: {
|
||||
winner_id: { type: "string" },
|
||||
ranking: { type: "array", items: { type: "string" } },
|
||||
verification_required: { type: "boolean" },
|
||||
reasons: { type: "array", items: { type: "string" } },
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
return {
|
||||
scenario: "operate-best-of-n",
|
||||
scenario: strategy === "search" ? "operate-search" : "operate-best-of-n",
|
||||
strategy,
|
||||
n,
|
||||
brief,
|
||||
rubric,
|
||||
candidates,
|
||||
review,
|
||||
apply_policy:
|
||||
"Parent applies winner only after reviewer PASS + parent re-verify.",
|
||||
"Parent applies a winner only after independent clean replay and explicit user approval.",
|
||||
execution_boundary:
|
||||
"This recipe generates and reviews candidates. It does not claim runtime-owned hidden gates, benchmark scoring, or clean-baseline replay; use a frozen WorkflowSearchSpec once the evaluator host is wired.",
|
||||
};
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user