Skip to content

Incident reference

The agent ships eleven runbooks, one per member of SignalKind, plus a fallback. An alert selects one through its hephaisto_kind label.

These are prompts, not operator runbooks

They are written in the second person and addressed to the model — they tell it what to investigate first and which instinct to distrust. They read well enough for a human that they are worth publishing, but that is not who they were written for.

Each page below is the real file, transcluded. If a runbook changes, this changes.

KindWhat it tells the model
CrashLoopBackOffThe restart count is the symptom, never the cause. previous: true logs, always and first.
ImagePullBackOffThe event message is the entire diagnosis. Do not hunt for logs that will never exist.
ConfigErrorA referenced ConfigMap or Secret key is missing. The pod never starts, so there are no logs.
OomKilledexitCode: 137. The kernel killed it without warning, so usually no logs.
UnschedulableThe reason exists only in Kubernetes Events. No metric tells you why.
ReadinessFlappingThe false-positive test. The instinct to restart a flapping pod is wrong.
NodePressureEscalate the scope immediately — pods failing on that node are consequences.
PvcNearlyFullA volume above 85/90%, with the PromQL inline.
JobFailedA Job exhausted its backoffLimit.
HighErrorRateSpan metrics. There may be no Kubernetes symptom at all.
_DefaultThe fallback. Reason about the controller, not the ephemeral pod.

Why there is a false-positive test in the set

ReadinessFlapping exists to catch an agent that reaches for a restart because a restart is the tool it has. A flapping readiness probe is usually a dependency problem or a probe tuned too tightly, and restarting the pod makes it worse while looking decisive.

An agent that scores well on ten scenarios but restarts this one has not learned to diagnose. It has learned to act.

AGPL-3.0-only · Source