This page is docs/architecture.md, transcluded from the repository. It is written for somebody about to change the code, which makes it more specific than a docs-site overview would normally be — that is the reason it is published rather than rewritten.
Hephaisto architecture
The pipeline
SOURCES CORE OUTCOMES
┌──────────────┐
│ K8s informers│──┐ ┌──────────┐ fingerprint ┌──────────┐
│(pods/events/ │ ├─────▶│ Signal │────dedup───────▶│ Incident │
│ nodes/jobs) │ │ └──────────┘ └────┬─────┘
└──────────────┘ │ ▼
┌──────────────┐ │ ┌──────────────────┐
│ Alertmanager │──┤ │ 1. INVESTIGATE │ read-only tools,
│ webhook │ │ │ (LLM + tools) │ step-budgeted
└──────────────┘ │ └────────┬─────────┘
┌──────────────┐ │ ▼
│ Periodic │──┘ ┌──────────────────┐
│ PromQL sweep │ │ Finding+Evidence │ ── Grafana annotation
└──────────────┘ └────────┬─────────┘ ── Blazor timeline
▼
┌──────────────────┐
│ 2. PLAN │ NO tools,
│ (LLM, schema) │ JSON schema only
└────────┬─────────┘
▼
┌──────────────────┐
│ 3. EXECUTE │ pure C#,
│ PolicyEngine │ closed vocabulary
└────────┬─────────┘
┌─────────────────┴──────────────────┐
Allow (low risk) RequireApproval
▼ ▼
dry-run → execute await human
└───────────────┬────────────────────┘
▼
┌───────────────────────────┐
│ Verify T+1m / T+5m / T+15m│
└──────────┬────────────────┘
┌────────────────┼────────────────┐
▼ ▼ ▼
Resolved auto-rollback EscalatedThe single most important design decision
The LLM never holds a mutating tool handle.
Phase 1 gives it read-only tools. Phase 2 is a separate model call with zero tools and a JSON response schema. Phase 3 is pure C# over the typed result, against a closed ActionType enum — ActionExecutor, a switch whose every arm is a typed API call written in this repo.
Two places where that could have leaked, and does not. The executor covers only the verbs the write Role grants and refuses everything else before a call is made, so a plan naming CordonNode produces outcome=unsupported rather than a 403. And the rollback spec is free-form JSON the model wrote: it is read for typed values and never executed as written, because doing so would hand back the mutating handle on the one path nobody watches — minutes after the incident, with budgets deliberately bypassed.
A prompt injection in a log line can therefore at most produce a plan that the deterministic policy engine then rejects. It can never reach the Kubernetes API. The split also sidesteps Gemini's historical "tools XOR responseSchema" restriction, which is a convenient second reason for a decision that was already correct on security grounds.
Incident state machine
Detected ─► Triaging ─┬─► Suppressed (dedup / flap / maintenance / self-signal)
└─► Investigating ─┬─► Escalated (budget exhausted, low confidence,
│ policy deny, no plan, quarantine)
├─► AwaitingApproval ──(timeout)──► Escalated
│ │ approve
└─► Acting ─► Verifying ─┬─► Resolved
└─(rollback)─► Escalated
Any ─► Expired (24 h no signal) │ Resolved (human, or signal quiet 10 min)Transitions go only through Core/IncidentStateMachine.cs, one method per edge, each emitting an IncidentEvent row. The column answers "what state is it in"; the event log answers "how long was it awaiting approval" — both are needed and neither substitutes.
The LLM may propose Resolved; only the verifier grants it, after checking Kubernetes and PromQL. A model marking its own work complete is not evidence.
Fingerprinting, dedup and correlation are deterministic C#
Never the LLM at ingest — it would be both expensive and nondeterministic on the hot path.
- Fingerprint =
sha256(source | kind | cluster | namespace | ownerKind/ownerName | reason), keyed on the owner, never the pod name. A Deployment whose pods churn produces one fingerprint, not fifty. - Burst collapse on the fingerprint over 5 minutes.
- Flap detection: more than 3 incidents in an hour ⇒
Suppressed{Flapping}with a 4-hour cooldown, plus one meta-incident routed straight toEscalated. - Correlation by
CorrelationKey, or same namespace with an overlapping ownerRef chain within 10 minutes, or a node-level signal absorbing pod signals on that node.
Context management — why logs are never passed through raw
Pod logs are the whole cost problem. Core/LogDigester.cs:
- strip ANSI and timestamps,
- normalise UUIDs, hex, integers, IPs and durations to placeholders,
- group identical normalised lines and emit the top-K clusters as
{count, firstSeen, lastSeen, exemplar}, - always keep the last 40 lines verbatim,
- always keep every line matching
panic|fatal|exception|OOM|refused|timeout|unauthorized|deniedverbatim with ±3 lines of context, - hard cap at 8 KB with the omission marked.
The full raw log goes to evidence_blobs; the model gets an evidence://step/{id} URI it can cite and a human can click. Digest for the model, raw for the audit.
Grounding is a runtime invariant, not a prompt instruction
Every Evidence must reference a StepId from this investigation, and its Excerpt must be a substring of that step's stored result after whitespace normalisation. Verified with Contains.
Failing evidence is dropped; a finding with zero surviving evidence is dropped; a plan citing a dropped finding is rejected and the incident escalates. Counted as grounding.rejected{reason} — a rising rate is the earliest signal of prompt drift.
This is checked in code rather than asked for in the prompt because asking does not work: a model that hallucinates a plausible log line will also sincerely believe it cited it.
Safety architecture, outermost first
The outermost layer is the one that survives a compromised process.
- RBAC — read cluster-wide, write only into
hephaisto-chaos, no Secrets access at all. ASelfSubjectAccessReviewat startup asserts the agent does not hold verbs it should never have, and refuses to boot if it does. - Policy engine — pure, deterministic, default-deny.
- Self-protection —
hephaisto,hephaisto-obsandkube-systemare permanently denied. The agent may never act on itself or on the stack it depends on to see. - Budget and rate limits, Postgres-backed so they survive a restart. Exceeding a budget downgrades to RequireApproval rather than hard-denying: a human must still be able to act.
- Cooldown — 15 minutes per workload, checked inside the same transaction that inserts the action.
- Kill switch, three independent forms: env var, live-watched ConfigMap, database row. Fail-safe direction: an unreadable ConfigMap reads as
observe; unreachable Postgres means refuse to act. - Modes —
observe,dryrun(really calls the API withdryRun=All),auto. Promote per action type, never globally. - Stability gate, evaluated immediately before execution: no acting during an in-flight rollout, on pods younger than 120 s, in a maintenance window, or when the cluster-wide unhealthy fraction is high.
- Oscillation detection — the same action three times in two hours with the incident reopening ⇒ 24-hour quarantine, recorded against the workload on the row admission already locks. This is the concrete answer to "it restarts a pod that crashes again forever", and it is the only control that notices the agent is not helping: every other one caps a rate, and a workload failing every fifteen minutes sits comfortably inside all of them. Not on the incident, because a recurrence arrives as a new incident — so a quarantine held there would lapse exactly when the loop would otherwise continue.
- Verification and auto-rollback at T+60 s, T+5 m, T+15 m — deterministic predicates, never a model, and only the last attempt may conclude a failure. The three answer different questions rather than retrying one: at 60 s "did anything obviously break", at 5 m "has it converged", at 15 m "did it come back". A check that cannot run is Inconclusive, never Failed, so an API timeout does not revert a healthy cluster.
- Immutable audit trail, append-only in Postgres and mirrored to Kubernetes Events on the target object — so
kubectl describe podshows "hephaisto restarted this pod because …", which is where an on-call engineer actually looks.
Why L3 is safe enough to enable, in four sentences
RBAC bounds the worst case to delete pods and patch workloads in one namespace. The LLM never holds a mutating handle, so prompt injection from a log line can at most produce a plan the policy engine rejects. Every auto action is individually reversible and is actually reverted on failed verification. Budget, cooldown and oscillation caps mean the worst sustained case is about ten pod restarts an hour — indistinguishable from a badly tuned HPA.
Self-observability
hephaisto.incident {incident.id, correlation_key, signal.kind, k8s.namespace, workload}
└ hephaisto.investigation {investigation.id, model, budget.steps, budget.usd}
├ chat gen_ai.operation.name=chat, gen_ai.system=gemini,
│ gen_ai.request.model, gen_ai.usage.input_tokens/output_tokens
│ ← emitted free by .UseOpenTelemetry() (semconv v1.37)
├ hephaisto.tool.* {tool.name, tool.result_bytes, tool.truncated, k8s.*}
└ hephaisto.plan {plan.action_count, plan.max_risk}
hephaisto.policy.evaluate {decision, reasons}
hephaisto.action.* {action.id, action.risk, action.mode, action.dry_run}
hephaisto.verification {result, attempt}Order matters: .UseOpenTelemetry() before .UseFunctionInvocation() in the ChatClientBuilder chain, or the tool calls are not captured inside the chat span.
The observability-selfcheck rules webhook back into the agent's own ingest, with self-signals hard-coded to Escalated and never auto-actionable — otherwise the agent can act on itself in a feedback loop.
Reaching people: the outbox
Everything above happens inside one process. This is the part that does not.
The problem it solves is not "send a message". IIncidentNotifier could already send one: an in-process Channel<T> fan-out to Blazor circuits, bounded at 64 with DropOldest. It is a fine hook point and a catastrophic delivery mechanism, because it is designed to drop — right for nudging a browser that will re-read from Postgres anyway, and wrong for the one message whose whole purpose is to reach somebody who is not looking. "Escalated, and nobody was told" is the worst failure this system has, and a pod restart must not be able to cause it.
So delivery is a table.
IncidentStateMachine.Transition appends an IncidentEvent on EVERY edge
|
v
HephaistoDbContext.SaveChangesAsync interceptor scans new IncidentEvent rows,
| evaluates routing, adds notification_deliveries
v -- ALL IN THE SAME TRANSACTION
one commit
|
v
NotificationDispatcher (10s poll) due rows off (status, next_attempt_at)
|
+-- rate limit --> Suppressed (recorded, counted, never discarded)
|
+-- INotificationChannel --> Delivered
--> Retryable -> backoff, stay Pending
--> Permanent -> Failed + audit row + ERRORWhy the interceptor rather than ten call sites
The obvious design is an enqueue call at each place an incident commits a transition. The obvious failure of that design was already in this codebase: IncidentTriage reaches Escalated twice — the self-signal arm and the storm circuit breaker — and published no live event at all. Nobody noticed, because nothing asserted it.
Transition appends an IncidentEvent on every edge without exception; that is the log the audit trail is built from. Watching those rows gives the property directly: an incident cannot reach a notifiable state without a delivery row being written by the same SaveChangesAsync. There is no ordering, no second commit, and nothing for an eleventh call site to forget.
The two events that are not transitions — the mode changing and the policy reloading — are enlisted explicitly, following the same stage-don't-save discipline as an audit row.
Why the payload is frozen
An outbox row can sit for twenty minutes behind a failing endpoint. Re-reading the incident at send time — which is exactly what IIncidentNotifier correctly does for a UI nudge — would make a retry describe a later state than the event it reports, so an escalation card would quietly become a resolution card. The snapshot is taken at enqueue and never re-read.
Deep links are the exception and are built at render, because a wrong base URL should be fixable by editing a value rather than by re-queuing every pending row.
One retry authority
ServiceDefaults applies AddStandardResilienceHandler to every client the HTTP factory builds. The notification channels opt out of it. The outbox owns retry because the outbox is the only layer that survives a restart, and two schedules stacked would multiply every attempt against an endpoint that is already struggling.
The notifier must not amplify a storm
Ingest has dedup, flap suppression and a storm circuit breaker; the outbound side inherits none of them, and a storm that opens forty incidents would otherwise produce forty pages. Two controls, both pure functions in Hephaisto.Core for the reason ActionBudget is: a status page has to be able to answer "why did that not go out" with the identical arithmetic.
The first message for a workload always goes out. A cooldown that could swallow the opening message would be a worse failure than the storm it prevents. The repeats are suppressed, counted on the row, and stated on the next message that does go out.
Persistence: Postgres 17 + pgvector
Four demands that rarely co-occur, all served by one process:
- ACID across a multi-row decision. The budget check, cooldown check, kill-switch check and the action INSERT must be one transaction, or there is a TOCTOU race on the one code path where a race means an unintended
kubectl delete. This requirement alone eliminates most alternatives, and it is why the agent is a single pod. - Relational audit queries — "every action on deployment X in 30 days".
- Heterogeneous payloads — alert bodies, tool args, pre/post state →
jsonb+ GIN. - Hybrid search over incident history — pgvector HNSW for semantics, tsvector GIN for exact identifiers, fused with Reciprocal Rank Fusion (k=60).
Hybrid rather than pure vector because vector search reliably misses exact identifiers — an image tag, an error code, a workload name — which is exactly what an SRE query is often about.
Retention is asymmetric on purpose: evidence blobs (~1 MB) expire at 30 days; incident digests and their embeddings (~2 KB) are kept indefinitely. History stays searchable long after the logs behind it are gone, which is why a digest must stand on its own.