Skip to content

This page is src/Hephaisto.Agent/Runbooks/Unschedulable.md, transcluded from the agent's own source. It is what the model is given when an alert declares hephaisto_kind: Unschedulable — not a description of it.

Unschedulable (Pending)

The scheduler cannot place the pod. It is Pending with no node assigned.

The critical point

The reason exists only in Kubernetes Events. No metric tells you why the scheduler refused — kube_pod_status_unschedulable only tells you that it did. This is precisely why the OTel Collector's k8s_events receiver is part of the MVP.

Query it in Loki:

logql
{service_name="k8s-events"} |= "FailedScheduling" |= "<pod-name>"

Reading FailedScheduling

The message enumerates every node and why each was rejected. Read it literally:

MessageCause
insufficient memory / insufficient cpuRequests exceed remaining allocatable capacity.
node(s) had untolerated taintMissing toleration, or the node was cordoned.
node(s) didn't match Pod's node affinity/selectorAffinity rules cannot be satisfied.
pod has unbound immediate PersistentVolumeClaimsPVC problem, not a scheduling problem. Follow the PVC.
node(s) had volume node affinity conflictThe volume is pinned to a node that cannot take the pod.

On this single-node cluster, insufficient memory usually means the request is simply larger than the node — compare against kube_node_status_allocatable.

Usual correct action

Scaling down a competing workload, or lowering the request, both need approval. On a one-node cluster, an impossible request is a human fix. Name the exact shortfall — "requests 500Gi, node allocatable is 115Gi" is a far more useful escalation than "unschedulable".

AGPL-3.0-only · Source