Skip to content

What it is

Hephaisto is an autonomous SRE agent that lives in your Kubernetes cluster. It receives Alertmanager webhooks, investigates what is wrong using PromQL, LogQL, traces and the Kubernetes API, and writes up a diagnosis with the evidence it used.

It is also a first-class producer of telemetry: every investigation is a trace you can open in Grafana, step through, and then ask the agent about.

The pipeline

Alertmanager ──▶ ingest ──▶ dedup + correlate ──▶ incident


                                          investigation loop
                                    (read-only tools, budget-capped)


                                  diagnosis + evidence + proposed plan


                                            policy engine
                                            (default-deny)

                              ┌─────────────────────┼─────────────────────┐
                              ▼                     ▼                     ▼
                          escalate            await a human            execute
                         to a human            (approve)         (closed action enum)
                                                    │                     │
                                                    └──────────┬──────────┘

                                              verify at T+60s / T+5m / T+15m

                                                    ┌──────────┴──────────┐
                                                    ▼                     ▼
                                                resolved          revert / escalate

An investigation is a bounded agentic loop. The model gets read-only tools — Kubernetes reads, PromQL, LogQL, trace search — and a step, token, cost and wall-clock budget. It ends by calling conclude with a root cause, the evidence it relied on, and a confidence. Everything it did is persisted as a step trace you can replay in the UI.

The fastest way to understand the shape of that is to look at ten real ones — no install, no account.

What exists today

Being precise about this matters, because the difference is the whole safety argument.

Alertmanager ingest, dedup, correlation, suppressionworks
Investigation loop with read-only tools, budgets, step traceworks
Diagnosis, evidence, semantic incident searchworks
Policy engine — default-deny, pure, exhaustively unit-testedworks
Kill switch — three independent arms, most restrictive winsworks
Audit log, budgets, cooldowns, oscillation detectionworks
Plan generation (schema-constrained, no tools)works
Executing a plan against the clusterobserved on a cluster, five action types
Verification at T+60s / T+5m / T+15m, and rollbackbuilt, never observed closing an incident
Approval workflow — UI and APIworks
Oscillation detection wired to a workload quarantineworks
SilenceAlert — always requiring approvalbuilt, needs Alertmanager configured
Outbound notifications: webhook and Teams, over a Postgres outboxworks
RollbackDeployment, PatchResourcesnot built — refused, not attempted
Runbook memory, OIDC approval identity, in-card approvalnot built
A written design language, one token set, visual regression baselinesworks

The wording of each row is chosen, not casual. Detection, investigation and diagnosis are measured against a real cluster over ten seeded scenarios. The delivery path was measured including the assertion it exists for: the receiver taken down, the agent restarted mid-flight, the receiver brought back, and the delivery arriving anyway.

The row to read carefully

The agent has been observed executing actions against a cluster, and the policy engine that gates them is exhaustively unit-tested. What has never been observed is the last step — an incident the agent acted on reaching Resolved.

How often the planner proposes an action at all turns out to be a property of the model, not of the agent. Measured on one fixture, holding everything else fixed:

ModelRunsProposed an action
deepseek-v4-flash84
gpt-oss:120b180
gemini-3.7-flash30

Fisher's exact test gives p = 0.0047. This is worth knowing before you choose a model: a cheap local model can diagnose well and still decline to ever propose a remediation. The evidence page has the detail.

What it deliberately does not do

The executor covers exactly the verbs the write Role grants: RestartPod, RolloutRestart, ScaleWorkload, DeleteStuckJob and DeleteFailedJobPods. Anything else is refused before a call is made, with outcome=unsupported and nothing attempted — which for CordonNode and DrainNode is the honest answer, because their ClusterRole ships deliberately unbound.

The chart installs the agent, its RBAC, both NetworkPolicies, the PodMonitor, the alert rules, a Grafana dashboard, and — opt-in — an evaluation Postgres. It does not install Prometheus, Alertmanager, Grafana, Loki, Tempo or a collector. You already run those.

Next

AGPL-3.0-only · Source