Agent Behaviors¶

Each turn moves through a fixed LangGraph workflow before the coordinator emits its first token, and every coordinator response is shaped by five feature-flagged behaviors plus a set of always-on safety rules baked into the system prompt.

Workflow nodes (per turn)¶

START → memory_loader → context_fetcher → coordinator
                                              │
        ┌─────────────────────────────────────┼─────────────────────────────┐
        ▼ TARGETED                            ▼ RCA_REQUIRED                ▼ direct
  targeted_investigator              subagent_executor × 4 (Send fan-out)  END
  (3 parallel reads,                  pod | metrics | logs | events
   back to coordinator)                       │ (fan-in)
        │                                     ▼
        └─────────► coordinator (synthesis) → END

Node	What it does
`memory_loader`	Loads pinned context from Postgres (user prefs, failure hints, recent RCA, session notes). SQLite mode skips this silently.
`context_fetcher`	Runs `kubectl get pods --all-namespaces` + `get events --field-selector=type=Warning` in parallel. Sets `snapshot_has_issues`, `snapshot_has_warnings`, `snapshot_pod_count`, `snapshot_built_at`, `cluster_id`, and matches playbooks against the snapshot.
`coordinator`	LLM with the four tools (`run_kubectl`, `run_helm`, `query_prometheus`, `query_loki`). Decides: direct answer, `TARGETED`, or `RCA_REQUIRED`. On synthesis turns, merges subagent findings into one `RCAResult`.
`targeted_investigator`	Runs three parallel `kubectl` reads (`describe pod`, `get events`, `get deployments`) for a single failing resource and appends them to the snapshot, then routes back to the coordinator for the final answer.
`subagent_executor` (× 4)	Domain specialist subagents — pod, metrics, logs, events. Each is a ReAct loop over the same tools, capped at 3–5 tool calls, returning a typed `AgentFinding`.

route_coordinator is the only conditional edge: it returns a string, END, or list[Send] for parallel fan-out (LangGraph's native fan-out mechanism). Fan-in back to the coordinator happens automatically once all four Send branches write into the findings reducer.

Behaviors¶

The KubeIntellect coordinator implements five additive behaviors that shape how it investigates Kubernetes issues. Each is feature-flagged in Configuration → Agent behavior flags.

Behavior	Default
kubectl error interpreter	on
Snapshot sufficiency gate	`lenient`
Gather-then-conclude discipline	always on
Playbook library	on
Visible investigation plan	on

kubectl error interpreter¶

When kubectl exits non-zero, the tool layer scans stderr for known patterns (NotFound, Forbidden, connection refused, missing CRD, immutable field, …) and appends a single-line hint after the original error. The LLM sees both — the raw error is never replaced.

Why. Stops the agent from looping on errors it could have skipped. Example:

Error from server (NotFound): pods "payments-1" not found
→ Pod may have been rescheduled — re-run `kubectl get pods -n <ns>` to find the new name.

Disable: KUBECTL_ERROR_HINTS_ENABLED=false.

Snapshot sufficiency gate¶

context_fetcher runs at the start of every turn and pre-fetches a cluster snapshot (pod list + Warning events). C2 adds a soft prompt bias: when the snapshot is healthy and the user asks a list-shaped, read-only question, the coordinator is encouraged to answer from the snapshot without an extra kubectl get pods.

Always falls back to fresh data when: - The question targets a specific named pod / deployment / service. - The user asks about logs, metrics, history, "yesterday", "last N hours". - The coordinator just performed a mutation (verifies via fresh get). - The query contains now / right now / currently. - The snapshot is older than SNAPSHOT_FRESHNESS_SECONDS (default 30s).

Why a soft bias, not a hard gate? Pod state changes fast in Kubernetes; a hard gate could return stale answers. The soft bias only fires for clean snapshots and list-shaped questions, with explicit always-fetch escape hatches.

Modes: off (no bias — pre-C2 behavior), lenient (default — bias only when truly applicable), strict (aggressive bias — opt in for trusted deployments).

Set: SNAPSHOT_SUFFICIENCY_MODE=off|lenient|strict, SNAPSHOT_FRESHNESS_SECONDS=30.

Gather-then-conclude discipline¶

A prompt-only directive: when tools are needed, follow PLAN → FETCH → SYNTHESIZE. Emit all independent tool calls in a single response (parallel), then synthesize once. Never interleave partial answers with more tool calls.

Exception: sequential dependencies (e.g. find a pod's name → describe that pod) are allowed; even then, gather everything else in parallel at each step.

This is always on — it is part of the coordinator's core system prompt.

Playbook library¶

For the top recurring Kubernetes failure modes, KubeIntellect ships a YAML playbook with a deterministic investigation sequence. When context_fetcher detects a matching pattern in the snapshot, the coordinator's system prompt includes the playbook(s) inline — guiding it to follow proven steps before improvising.

Playbooks shipped (18):

Pod / container lifecycle

CrashLoopBackOff
OOMKilled
ImagePullBackOff / ErrImagePull
CreateContainerConfigError (missing ConfigMap / Secret refs)
ContainerCreatingStuck (volume / CSI)
InitContainerFailing
ReadinessProbeFailing (also covers liveness)
CommandHardcodedFailure (hardcoded exit 1 / error in container command)
Evicted (node-pressure eviction)
TerminatingStuck (finalizers)

Scheduling / capacity

PendingInsufficientResources
PendingSchedulingConstraints (taints / affinity / nodeSelector)
QuotaExceeded (ResourceQuota)
NodeNotReady

Workloads, networking & admission

JobBackoffLimitExceeded
ServiceNoEndpoints (selector / label drift)
ServiceUnreachable
WebhookAdmissionRejected

Schema (drop a YAML file into app/agent/playbooks/):

name: <unique pattern name>
triggers:
  - pod_status_regex: "<regex on STATUS column>"
  - event_reason_regex: "<regex on Warning event REASON>"
  - event_message_regex: "<regex on Warning event MESSAGE>"
investigation_steps:
  - "<imperative step 1>"
  - "<imperative step 2>"
expected_evidence:
  - "<what to look for>"
recommended_fix_template: |
  <multi-line fix template; placeholders welcome>

A playbook matches if any of its triggers matches. The coordinator still has agency — it can deviate when the situation warrants — but the playbook gives it a strong default.

Disable: PLAYBOOKS_ENABLED=false.

Visible investigation plan¶

For queries requiring three or more tool calls, the coordinator writes its plan as the first line of the response:

INVESTIGATION_PLAN:
- Check pod status in default namespace
- Describe the crashing pod
- Query Loki for errors in the last 30m
- Propose a fix

The plan block is parsed out of the message body and emitted as a structured PlanEvent on the SSE stream. UI clients (kube-q, browsers) can render it as a checklist; Langfuse traces show it for post-mortem review.

Why. Makes multi-step investigations transparent and gives the agent an anchor to stay on-track. Trivial single-call queries skip the plan (threshold: ≥ 3 steps).

Disable: INVESTIGATION_PLAN_ENABLED=false.

Routing decision¶

Every coordinator turn produces exactly one of three routing outcomes.

Outcome	When	What the coordinator emits
direct	Simple list / status / single-resource query, or a mutation.	A normal answer (with tool calls, possibly a plan, possibly a `TARGETED:` block).
TARGETED	One specific resource is failing and needs deeper inspection.	`TARGETED: namespace=<ns>, pod=<pod>, issue=<one-line>` on its own line. The `targeted_investigator` runs three parallel reads (describe, events, deployments), appends them to the snapshot, and the coordinator answers with the enriched context.
RCA_REQUIRED	Multi-pod / cross-namespace outage, unknown root cause, cascading failures.	The literal token `RCA_REQUIRED`. The router fans out to four specialist subagents in parallel; the coordinator then synthesizes their findings into one `RCAResult`.

The sentinel text is parsed out of the message stream — it never reaches the user. TARGETED should always be preferred over RCA_REQUIRED for single-resource issues; the four-subagent fan-out is reserved for genuinely ambiguous, cross-cutting failures.

Always-on safety rules (in the coordinator prompt)¶

These rules are not feature-flagged — they live in the coordinator system prompt and apply to every turn.

Mutation batching (HITL safety)¶

At most one mutation per response. Reads (get, describe, logs, top) may still be batched in parallel — only patch / apply / create / delete / scale / set / rollout are restricted. Batching multiple mutations in one response causes redundant approval prompts and re-queues unapproved calls when HITL fires mid-batch. The _fill_orphan_tool_calls helper injects "skipped" placeholders for any tool_call without a matching ToolMessage so the LLM doesn't re-propose them on the next loop.

Fix verification (after every mutation)¶

After every successful kubectl patch / apply / create / delete, the coordinator must perform one more kubectl get on the affected resource and report the actual post-fix state ("Pod is now Running (verified)"). The reflexion subsystem re-runs this check independently — fix verification at the prompt level is the agent's contract; reflexion verification is the system's gate before promoting a pattern.

Service-endpoint cross-check (namespace-level queries)¶

For any namespace-level investigation ("check ns X", "what's wrong in X", "diagnose X"), the coordinator must include kubectl get endpoints -n <ns> and kubectl get services -n <ns> in the initial parallel batch — alongside get pods and get events. A service whose ENDPOINTS column is <none> while its target pods are Running is a silent fault: no warning event fires for selector/label drift. This cross-check is the only reliable way to surface it.

Spec-before-logs for CrashLoopBackOff¶

When diagnosing a CrashLoop pod, the coordinator must read spec.containers[].command and spec.containers[].args from kubectl describe pod before inferring a root cause from log output. A log line like "DB not configured" may be hardcoded in the container's command, in which case no env or secret patch will fix it.

Tool selection by time intent¶

Phrasing	Tool
"Current / active issues", "now", "today"	`kubectl get pods --all-namespaces` + `describe` for Last State
"Last N hours/days", "yesterday", "last night"	`query_prometheus` with `range_minutes`, `query_loki` with `since=Nh`
"Pods with issues" (no qualifier)	`kubectl get pods --all-namespaces` — pods not in Running/Completed/Succeeded right now

Using range_minutes>0 for "current issues" surfaces already-resolved problems and produces false positives.

Shell-metacharacter constraints¶

The runner blocks any kubectl command containing ;, &, `, $, or \. (Pipes and redirection are excluded — | is reimplemented in Python and < / > are harmless under shell=False.) The constraint applies to the full command string, including arguments inside --patch '[...]' or -- sh -c "...".

For container command / args changes (which usually contain shell metacharacters), the only reliable path is:

kubectl get <kind> <name> -n <ns> -o yaml to fetch the current spec.
Build the corrected manifest in the response.
kubectl apply -f - with the manifest passed via stdin.

kubectl edit is rejected outright — there is no interactive terminal in the container or the pip install.

Session-history compression¶

The coordinator caps message history at the last 20 messages (about 5 prior exchanges). When the cap fires, the dropped messages are summarised deterministically (no extra LLM call) into a Earlier Session Context (compressed) block injected into the system prompt — preserving topics, commands run, and key tool results without bloating the context window.

Proactive Fix Mode (`auto_approve=true`)¶

When the request body sets auto_approve=true (used by the evaluation harness and trusted automation), HITL gates are bypassed and a Proactive Fix Mode block is appended to the system prompt: the coordinator must apply identified fixes immediately, choose the safest default for ambiguous parameters, verify with a fresh get, and stop with an explicit "cannot determine" message rather than guessing.

Always-confirm gate (overrides `auto_approve`)¶

A small set of cascading-blast actions ALWAYS prompt for confirmation, even on auto_approve=true sessions. The HITL interrupt fires with risk_level=high and always_confirm=true; there is no way to silently auto-approve them.

Verb pattern	Why it cannot auto-approve
`delete namespace\\|ns`	Cascades to every resource in the namespace; no rollback.
`delete pv\\|persistentvolume`	Releases user data; CSI drivers may delete the underlying disk.
`delete crd\\|customresourcedefinition`	Cascades to every CR of that kind cluster-wide.
`set image \\|set resources`	Live mutation of running workloads; use `rollout undo` to revert.
`drain`	Evicts every pod on the node; depends on PDB compliance.

Defined in app/tools/kubectl_tool.py (_ALWAYS_CONFIRM_DELETE_TARGETS, _ALWAYS_CONFIRM_SET_SUBCOMMANDS, _requires_always_confirm). Plain delete pod, apply, patch, scale continue to auto-approve under auto_approve=true.

The role layer runs before the always-confirm gate, so existing role permissions still apply: readonly keys can't reach the gate at all, operator keys are still blocked on high-risk verbs (delete, drain, replace, taint) before HITL runs. The always-confirm gate only matters for admin and superadmin keys (and for operator running set image| resources, which is medium-risk but always-confirm).

How they compose¶

A typical investigation of a CrashLoopBackOff pod, with all behaviors on:

Snapshot gate — context_fetcher builds the snapshot, sees the unhealthy pod, sets snapshot_has_issues=true and matches the CrashLoopBackOff playbook.
The coordinator system prompt now includes the snapshot, the playbook details (describe → previous logs → events), and the snapshot sufficiency block (which won't fire because issues are present — we always fetch when unhealthy).
Investigation plan — for a 3+ step query, the coordinator emits INVESTIGATION_PLAN: … first; UI shows the checklist.
Parallel discipline — coordinator emits all independent tool calls in one response.
Error interpreter — if any kubectl call returns a known error pattern, the hint is appended before the LLM sees it, avoiding retry loops.
Final answer references each plan step and proposes a fix from the playbook's recommended_fix_template.
Reflexion outcome write — if the answer ran a mutation, the reflexion subsystem verifies the cluster post-fix and records the outcome (cluster-scoped, with cooldown).

Each phase can be flipped independently if you need to roll one back.

The five behaviors above shape one turn. On top of them, KubeIntellect runs a reflexion subsystem that records outcomes across turns and promotes recurring, verified patterns back into future prompts. Cluster-scoped, with verification gates, cooldown, and retention.

See Reflexion Subsystem for the design and operational view.