5 · Lab 3 — Chaos and recovery¶
10 minutes · facilitator-led demo
Format changed from the proposal
This was submitted as a 15-minute hands-on lab. We run it as a demo at the front of the room, with participants as backseat drivers. The reasoning is in P7: with a non-deterministic agent and a room that may be running behind, a hands-on block here is the one that breaks the schedule.
Goal¶
Show agent-driven incident response end to end, with nobody touching kubectl.
Release the chaos¶
The script breaks something in the cluster. The scenarios:
| Scenario | What happens |
|---|---|
secret-gone |
A Secret mounted by a Deployment is deleted |
node-pressure |
A node is pushed into memory pressure |
dns-mess |
The CoreDNS config is mangled |
bad-rollout |
A Deployment is pointed at an image that doesn't exist |
For the demo we run one scenario, chosen in advance so it's reliable. The room doesn't get told which.
The task¶
Something is wrong in the cluster. Find out what, explain what happened, and propose a fix.
The agent should:
- Find the fault — search broadly, then narrow
- Explain what happened, in plain language
- Propose or perform a fix
What to watch for¶
Narrate this out loud as it runs — it's the whole value of the block:
- Did it start broadly (
get events -A) or jump to a conclusion? - Did it separate symptom from cause? (A pod being down ≠ a Secret being deleted.)
- Did it propose anything destructive?
- When it was wrong, how was it wrong? Confidently?
Discussion¶
How long would you have taken on the same scenario? And — more importantly — would you have trusted the diagnosis without verifying it?
Running it yourself¶
Everything here works on your own machine afterwards:
The rule that makes it worthwhile: no manual kubectl. If the agent gets stuck, that's
a finding, not a problem for you to solve on its behalf. Note where it stopped.
Checkpoint¶
- The agent identified the correct root cause
-
make healthis green again