Cluster environments¶
Three ways to get a cluster, one identical workshop on top of them.
1 · The idea that makes this work¶
The namespace is the unit of work, not the cluster
Every lab, every guardrail, every eval and every quest operates on one namespace with a namespace-scoped service account. Nothing in the participant path needs cluster-scope.
Once that's true, "which cluster" stops being a fork in the curriculum and becomes a twenty-line shell script per environment.
This single decision buys us four things at once:
| It gives us | Because |
|---|---|
| Three environments for the price of one curriculum | Only getting a kubeconfig differs |
| Safe "bring your own cluster" | A namespace-scoped SA can't touch the rest of your cluster |
| Fast evals | Namespace reset is seconds; a cluster rebuild is ~2 minutes (P1) |
| Portable quests | Grading reads namespace state, which looks the same everywhere |
2 · The environment contract¶
After make env-up ENV=<env>, all five of these are true, regardless of environment:
kubectlhas a working context, andmake verifynames it back to you- A namespace
agentic-<handle>exists and is the default for the session - The RBAC ladder exists in that namespace:
agent-ro,agent-ns,agent-harden - The deliberately insecure workload is deployed there
make verifyexits 0
Everything downstream — make agent, make health, make chaos, make eval,
make prove — targets that namespace and never asks how it got there.
bootstrap/
├── base/ # ONE kustomization, applied identically everywhere
│ ├── namespace.yaml
│ ├── rbac-ladder.yaml
│ ├── workload-insecure.yaml
│ └── kustomization.yaml
└── env/
├── codespaces.sh # ~20 lines: k3d inside the codespace
├── k3d.sh # ~20 lines: k3d locally
└── byo.sh # ~30 lines: validate an existing context, then guard
env-up:
@bootstrap/env/$(ENV).sh # get a kubeconfig — the only difference
@kubectl apply -k bootstrap/base # identical from here down
@$(MAKE) verify
The line worth defending
If a lab instruction ever has to say "if you're on Codespaces, do X instead", the abstraction has leaked and we should fix the bootstrap rather than the docs.
3 · The three environments¶
k3d running inside a Codespace, via docker-in-docker.
Why this is the headline option: it retires the two highest risks in the premise review outright.
| Risk | Status in Codespaces |
|---|---|
| P1 — conference Wi-Fi can't carry 35 image pulls | Gone. Images pull inside GitHub's network |
| P2 — corporate laptop blocks Docker or the API | Gone. Nothing runs locally but a browser |
A prebuilt devcontainer image means the cluster is warm within a minute of the Codespace opening.
- ✅ Nothing to install; a browser is enough
- ✅ API key injected as a repo Codespaces secret — never pasted, never committed
- ✅ Identical for everyone, so facilitator debugging is one known environment
- ⚠️ Requires a GitHub account and burns Codespaces quota
- ⚠️ Default machine size matters — specify 4-core in
devcontainer.json
What the proposal originally assumed.
- ✅ No account, no quota, no dependency on the venue's network at runtime
- ✅ Fastest iteration once warm — the right choice for us while building
- ⚠️ Requires the prerequisites actually done at home
- ⚠️ arm64 and amd64 both need testing
An existing cluster the participant already has: a homelab, a dev EKS/AKS/GKE, Docker Desktop's Kubernetes, minikube.
- ✅ The most realistic — the agent is acting on a cluster that has other things in it
- ✅ Best take-home value: they leave having done this on their own infrastructure
- ⚠️ Needs guardrails of its own. See §5
- ⚠️ Cluster-scoped labs don't work here. See §4
4 · Capability matrix — the honest part¶
Not every lab survives every environment, and pretending otherwise would break the room.
| Codespaces | Local k3d | BYO cluster | |
|---|---|---|---|
| Lab 1 — deploy & debug | ✅ | ✅ | ✅ |
| Lab 2 — hardening | ✅ | ✅ | ✅ |
Lab 3 — bad-rollout, secret-gone |
✅ | ✅ | ✅ |
Lab 3 — dns-mess (CoreDNS) |
✅ | ✅ | ❌ cluster-scoped |
Lab 3 — node-pressure |
✅ | ✅ | ❌ cluster-scoped |
| Prompt-injection probe | ✅ | ✅ | ✅ |
| Quests (namespace-graded) | ✅ | ✅ | ✅ |
| Quest: admission policy (legendary) | ✅ | ✅ | ⚠️ needs cluster-admin |
Action item: re-scope the chaos scenarios
Two of the four chaos scenarios in the current design are cluster-scoped, which makes them unavailable to BYO participants — and unavailable to anyone if we ever move to a shared cluster.
Re-scope them to namespace level. DNS breakage can be simulated with a NetworkPolicy that blocks egress to port 53 from the namespace. Memory pressure can be simulated with a resource quota plus a greedy pod. Both are namespace-local, both produce the same diagnostic experience, and both then work everywhere.
This matters more than it looks: Lab 3 is our demo, and the demo should run on the same footing as the participants.
5 · Safety for "bring your own cluster"¶
This option is genuinely useful and genuinely the most dangerous thing in the workshop. We are handing people an agent with write access and telling them to point it at their own infrastructure.
bootstrap/env/byo.sh must refuse to proceed unless:
- The context is not production-shaped. Reject contexts matching
prod,production,prd; reject if the namespacekube-systemhas more than n non-default workloads; warn loudly on any cloud-provider context name. - An explicit acknowledgement is given. No default, no
-yflag in the docs: - The namespace is new. Refuse to adopt an existing namespace — create
agentic-<handle>or stop. - The RBAC ladder is namespace-scoped only. No ClusterRoleBinding is ever created for an agent SA. This is what actually bounds the damage; the checks above are seatbelts.
make nukeis precise. Deletes exactly the one namespace, and prints what it will delete before it does.
Put this on a slide
The BYO guard is itself a teaching artefact. It's the workshop practising what it
preaches: we don't trust a prompt to keep you safe, we make the dangerous thing
structurally hard. Show byo.sh in block 1.
6 · As built — verified 2026-09-02¶
The contract is implemented and tested end to end. Verification was run against a
kind cluster, which is deliberate: kind is not one of the three supported
environments, so the base applying cleanly to it is evidence the abstraction really is
environment-agnostic.
The ladder enforces at the API server, not in the prompt¶
Not can-i output — actual API calls, as the agent's service account:
Attempt as agent-harden |
Result |
|---|---|
patch deploy/shopfront |
✅ deployment.apps/shopfront patched |
delete deploy/shopfront |
🚫 Forbidden: cannot delete resource "deployments" |
get secrets |
🚫 Forbidden: cannot list resource "secrets" |
-n kube-system get pods |
🚫 Forbidden — blast radius holds |
create networkpolicy |
✅ Lab 2 has what it needs |
patch as agent-ro |
🚫 Forbidden: cannot patch resource "deployments" |
make ladder prints all of this for participants, side by side.
The BYO guard, validated against a real production cluster¶
The development machine's kubeconfig turned out to contain a live production AKS context. Running the guard's name check across every context on it:
allowed do-ams3-innocent-dog
allowed do-k3s
allowed hlutur
BLOCKED lz-prod-aks ← real production cluster, refused
warn lz-stage-aks (managed cloud — allowed, with acknowledgement)
This is exactly the accident the guard exists to prevent, found on the first machine we
tried. It also refuses without the acknowledgement variable, refuses to adopt a namespace
it didn't create, and make env-up ENV=byo exits non-zero so make halts rather than
continuing into the labs.
A trap for macOS participants — use braces
Found while testing. On zsh, :a inside a parameter expansion is a path modifier:
NS=agentic-ops
kubectl --as=system:serviceaccount:$NS:agent-harden ...
# zsh expands $NS:a → /Users/you/somewhere/agentic-ops, then "gent-harden"
# → a confusing Forbidden error about a user that doesn't exist
Always write ${NS}:agent-harden. Every lab instruction and quest card that
interpolates a namespace before : must use braces, or a third of the room —
everyone on a Mac — hits a baffling error.
7 · What we build, and when¶
Environments are cheap only if we build the contract before the labs. Fold into the development plan as follows:
| Week | Work |
|---|---|
| W1 | ✅ Done. bootstrap/base + apply.sh + verify.sh + nuke.sh, all three env scripts, make ladder |
| W2 | devcontainer.json written — still to do: prebuild the image and prove a full lab runs in a real Codespace |
| W3 | Test byo.sh against an actual remote cluster (name checks pass; the apply path is untested remotely) |
| W4 | Re-scope the two cluster-scoped chaos scenarios to namespace level |
Recommendation on defaults
Make Codespaces the documented default and local k3d the documented alternative. That inverts the current docs, and it's the right way round: it removes the two risks most likely to sink the room, and it means the prerequisites page shrinks to "have a GitHub account".
Keep local k3d fully supported — it's what we develop against, and some participants will prefer it — but stop making it the path of least resistance.