* Address ArgoCD overlay review follow-ups (#4977) and PR #4744 comments Add CI validation of kustomize overlays, clarify the ignoreDifferences /data vs /stringData ArgoCD normalization, separate sync-completes from app-healthy in the Ingress wave comment, add TODO(v0.29) to the bare-Pod fallback in terminate(), and update the sandbox-runners README to reflect the bare-Pod → Job migration. Signed-off-by: Yuan Tang <terrytangyuan@gmail.com> * fix(ci): install kustomize via official script instead of third-party action The pinned SHA for imranismail/setup-kustomize was unresolvable. Signed-off-by: Yuan Tang <terrytangyuan@gmail.com> * fix(docs): correct backoffLimit value in sandbox-runners README The README stated `backoffLimit: 0` but the actual code uses `_JOB_BACKOFF_LIMIT = 6` — fix the doc to match. --------- Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
This commit is contained in:
Yuan Tang
@@ -0,0 +1,55 @@
|
||||
name: Kustomize validate
|
||||
|
||||
# Renders every deploy/kubernetes overlay with `kustomize build` so manifest
|
||||
# drift (duplicate bases, missing patches, invalid YAML) is caught in CI
|
||||
# rather than at deploy time. Only runs when overlay or base files change.
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
paths:
|
||||
- 'deploy/kubernetes/**'
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- 'release/v[0-9]*'
|
||||
paths:
|
||||
- 'deploy/kubernetes/**'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: kustomize-${{ github.event.pull_request.number || github.sha }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
validate:
|
||||
name: Kustomize build (overlays)
|
||||
if: ${{ !github.event.pull_request.draft }}
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Check out repo
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
|
||||
|
||||
- name: Install kustomize
|
||||
run: |
|
||||
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
|
||||
sudo mv kustomize /usr/local/bin/
|
||||
|
||||
- name: Render all overlays
|
||||
run: |
|
||||
failed=0
|
||||
for overlay in deploy/kubernetes/overlays/*/; do
|
||||
name="$(basename "$overlay")"
|
||||
echo "::group::$name"
|
||||
if kustomize build "$overlay"; then
|
||||
echo "::endgroup::"
|
||||
else
|
||||
echo "::endgroup::"
|
||||
echo "::error::kustomize build failed for overlay '$name'"
|
||||
failed=1
|
||||
fi
|
||||
done
|
||||
exit "$failed"
|
||||
@@ -45,6 +45,9 @@ spec:
|
||||
# them out of band (kubectl edit, sealed-secrets, external-secrets), and
|
||||
# selfHeal must not revert those edits. Without this, selfHeal
|
||||
# continuously overwrites live credentials with the placeholder.
|
||||
#
|
||||
# The pointer targets /data, not /stringData, because ArgoCD normalizes
|
||||
# stringData into base64 /data on the live object before diffing.
|
||||
- group: ""
|
||||
kind: Secret
|
||||
name: omnigent-secrets
|
||||
|
||||
@@ -32,10 +32,12 @@ patches:
|
||||
|
||||
# The Ingress depends on an ingress controller (nginx by default) and
|
||||
# cert-manager. ArgoCD scores an Ingress without status.loadBalancer as
|
||||
# Progressing, so on a cluster with no controller the sync stalls forever.
|
||||
# Putting it in wave 1 (everything else is implicit wave 0) means nothing
|
||||
# gates on its health. ArgoCD's kind ordering already applies it last within
|
||||
# a wave, so this only affects the health gate.
|
||||
# Progressing. Wave 1 (everything else is implicit wave 0) means no later
|
||||
# sync wave gates on its health, so the *sync* completes. The Application
|
||||
# itself may still report Progressing indefinitely on a cluster without a
|
||||
# controller — with automated + selfHeal, that keeps the Application in a
|
||||
# perpetual reconcile loop (harmless but noisy). Install the controller or
|
||||
# add a custom health check that treats the Ingress as healthy.
|
||||
- target:
|
||||
kind: Ingress
|
||||
patch: |
|
||||
|
||||
@@ -1,42 +1,43 @@
|
||||
# Kubernetes sandbox runners (on-demand host Pods)
|
||||
|
||||
This Kustomize overlay turns on the **`kubernetes`** managed-sandbox provider: a
|
||||
`host_type: managed` session spawns one **runner Pod** that runs `omnigent host`
|
||||
as its container entrypoint and dials back to the server over the existing
|
||||
launch-token tunnel. It layers the RBAC + config the provider needs onto the
|
||||
base server deployment.
|
||||
`host_type: managed` session spawns a **batch/v1 Job** whose child Pod runs
|
||||
`omnigent host` as its container entrypoint and dials back to the server over the
|
||||
existing launch-token tunnel. It layers the RBAC + config the provider needs onto
|
||||
the base server deployment.
|
||||
|
||||
## Launch model: entrypoint-as-host
|
||||
|
||||
The runner Pod's container command **is** the host. An **init container**
|
||||
prepares the workspace (`mkdir` + optional `git clone`); the **main container**
|
||||
then runs `omnigent host` under a tiny PID-1 reaper. The host re-parents runner
|
||||
processes to PID 1, which the reaper reaps; SIGTERM is forwarded for graceful
|
||||
shutdown.
|
||||
The runner is launched as a **batch/v1 Job** (one Pod, `backoffLimit: 6`). The
|
||||
Job's child Pod runs `omnigent host` as its container command. An **init
|
||||
container** prepares the workspace (`mkdir` + optional `git clone`); the **main
|
||||
container** then runs `omnigent host` under a tiny PID-1 reaper. The host
|
||||
re-parents runner processes to PID 1, which the reaper reaps; SIGTERM is
|
||||
forwarded for graceful shutdown.
|
||||
|
||||
The launch token is delivered through a **per-Pod Kubernetes Secret** referenced
|
||||
The launch token is delivered through a **per-Job Kubernetes Secret** referenced
|
||||
by the Pod's `secretKeyRef` — it never enters the Pod spec, a command line, or
|
||||
an audit log. The launcher creates that Secret at provision and deletes it
|
||||
alongside the Pod at terminate.
|
||||
alongside the Job at terminate.
|
||||
|
||||
Because the host is **never started by `exec`-ing into an already-running
|
||||
container**, this provider needs **no `pods/exec` grant** — and avoids the
|
||||
exec-into-running-container class of runtime issues entirely. The server SA's
|
||||
rights are the minimum the launcher calls: create/get/delete Pods, get
|
||||
`pods/log` (start-failure diagnostics only), create/delete Secrets (the per-Pod
|
||||
token), and list events.
|
||||
rights are the minimum the launcher calls: create/get/delete Jobs,
|
||||
list/get Pods (to poll the Job's child), get `pods/log` (start-failure
|
||||
diagnostics only), create/delete Secrets (the per-Job token), and list events.
|
||||
|
||||
## Two-namespace, least-blast-radius design
|
||||
|
||||
| Namespace | Holds |
|
||||
|---|---|
|
||||
| `omnigent` | the server, its DB/PVC, its Secrets, the `omnigent-server` SA |
|
||||
| `omnigent-sandboxes` | runner Pods, the per-Pod token Secrets, the harness-creds Secret, the powerless `omnigent-runner` SA, the scoped Role + RoleBinding |
|
||||
| `omnigent-sandboxes` | runner Jobs (and their child Pods), the per-Job token Secrets, the harness-creds Secret, the powerless `omnigent-runner` SA, the scoped Role + RoleBinding |
|
||||
|
||||
The server SA's Pod/Secret rights are a **namespaced Role** bound (cross-namespace)
|
||||
to `omnigent-sandboxes` only — so a compromised server can manage runner Pods but
|
||||
**cannot** delete the server/DB Pods, read the server's Secrets, or execute
|
||||
commands inside any Pod. The runner namespace enforces Pod Security `restricted`;
|
||||
The server SA's Job/Pod/Secret rights are a **namespaced Role** bound
|
||||
(cross-namespace) to `omnigent-sandboxes` only — so a compromised server can
|
||||
manage runner Jobs but **cannot** delete the server/DB Pods, read the server's
|
||||
Secrets, or execute commands inside any Pod. The runner namespace enforces Pod Security `restricted`;
|
||||
the generated runner Pod is already restricted-compliant (non-root uid 1000, drop
|
||||
`ALL` caps, `seccompProfile: RuntimeDefault`, no privilege escalation).
|
||||
|
||||
@@ -83,7 +84,7 @@ runner Pod unexpectedly carries no credential:
|
||||
(`omnigent/server/managed_hosts.py`), e.g. "agent … is not a genuine built-in;
|
||||
omitting agent label".
|
||||
- A name that is not a valid label value logs a `WARNING` from
|
||||
`build_pod_manifest` (`omnigent/onboarding/sandboxes/kubernetes.py`), e.g.
|
||||
`build_job_manifest` (`omnigent/onboarding/sandboxes/kubernetes.py`), e.g.
|
||||
"agent … is not a valid omnigent.ai/agent value; runner Pod … stays
|
||||
unclassified". Note the gate upstream will already have logged this agent as
|
||||
classified, so this is the line that explains the missing label.
|
||||
|
||||
@@ -1704,6 +1704,9 @@ class KubernetesSandboxLauncher(SandboxHostLauncher):
|
||||
_request_timeout=_POD_READY_REQUEST_TIMEOUT_S,
|
||||
),
|
||||
),
|
||||
# TODO(v0.29): remove this entry once all runners have rolled
|
||||
# past v0.28 — bare Pods are no longer created. Keep in sync
|
||||
# with the pods:create/delete TODO in role.yaml.
|
||||
# Fall back to deleting a bare Pod left by the pre-Job
|
||||
# launcher. Child Pods are named <job>-<rand5> so this only
|
||||
# targets pre-migration bare Pods whose name IS sandbox_id.
|
||||
|
||||
Reference in New Issue
Block a user