From AI Agents to AI Controllers

Extending the Kubernetes controller model across engineering

“The art of progress is to preserve order amid change, and to preserve change amid order.”

—Alfred North Whitehead, Process and Reality (1929)

1. The task is not the objective

Task-centered agentic coding begins with a human decision: something in the world should change, and changing it is worth the effort. The human then initiates and steers the work through prompts. The agent helps carry out the change.

Explicit acceptance criteria and automated verification allow an agent to keep iterating with less human steering. This expands the agent’s autonomy inside a task without extending its responsibility beyond that task.

Engineering objectives outlive the tasks that advance them. Reliability, security, performance, efficiency, and comprehensibility all require continuing attention. That responsibility falls to software engineers, whose attention, capacity, and expertise are finite. As AI accelerates software change, the demand for that attention grows faster than the capacity available. Urgent problems take precedence, while emerging drift often remains unexamined until its consequences become visible.

Kubernetes offers a different model: declare a desired state and give a controller continuing responsibility for moving the system toward it. Advances in large language models make it possible to extend that model to engineering objectives whose reconciliation cannot be fully prescribed in advance.

Engineering objectives can now have controllers of their own.

This idea grew out of Maxwell, an AI controller framework I built at Qualified, now part of Salesforce. Maxwell began with a simple goal: use AI coding agents to help keep a rapidly changing codebase clean. The first agent regularly scanned for dead code and opened pull requests proposing its removal, a workflow similar to Google's Sensenmann. [1]

More agents followed, each focused on a different engineering concern. Then came a natural extension: an agent that could find an issue in existing code could also review incoming changes to prevent it.

As Maxwell grew, I realized that its agents were better defined by the desired conditions they worked toward than by the recurring tasks they performed. That was when the connection to Kubernetes controllers clicked.

Maxwell takes its name from Maxwell’s demon, a hypothetical gatekeeper that appears to reduce entropy by sorting molecules. The metaphor fits: reconciliation works against accumulated disorder, while admission guards the gate. In less than three months, more than 600 pull requests opened by Maxwell have been merged, and it has posted thousands of helpful review comments. This article generalizes the model that emerged from it.

2. Agentic controllers

A Kubernetes controller is a feedback loop: observe the current state, compare it with the desired state, act, and observe again. At its simplest:

action = controller(current_state, desired_state)

An action changes the system. The next observation reveals its effects, closing the control loop. In a traditional controller, however, the reconciliation policy is encoded in advance. It may handle cases its authors did not foresee, but only through logic they already supplied.

I use agentic controller to mean an AI agent that takes continuing responsibility for an objective. It retains the feedback loop while adding the ability to investigate unfamiliar conditions, remember outcomes, and adapt its approach.

The agent carries evidence and learned strategies between runs:

(action, next_memory) = agentic_controller(observations, desired_condition, memory)

The controller operates on observations because the complete state is rarely available. Its desired condition describes an acceptable state, a direction of improvement, or both. Its memory holds evidence, hypotheses, past interventions, exceptions, and outcomes. That memory informs the next action.

The controller changes the system, then uses the result to revise its understanding and strategy. With durable evidence, it can become more efficient, safer, and better adapted to the organization's needs.

An agentic controller therefore needs access to observe and act, explicit authority, triggers, and durable memory. On each invocation, it can act, wait, request approval, or escalate.

The controller follows the change through review and deployment, then observes its effects. If the change stalls or fails, the obligation persists into the next run. Because review attention is finite, the controller also considers what is already awaiting human judgment before proposing more work.

3. Reconciliation and admission

“The moral is clear: prevention is better than cure, in particular if the illness is unmastered complexity, for which no cure exists.”

—Edsger W. Dijkstra, “The Next Fifty Years”

A controller pursues its desired condition in two modes: it improves what exists and guards what comes next. Reconciliation and admission use the same desired condition, criteria, tools, and accumulated knowledge.

Reconciliation runs on a schedule or in response to an event. The controller inspects the existing system, investigates a violation or opportunity, proposes a change, and verifies the outcome. In a pull-request workflow, it opens a pull request and later checks the deployed result.

Admission evaluates a proposed change before it enters the system. In a pull-request workflow, opening or updating a pull request triggers the controller. Depending on its authority, it comments, suggests a modification, requests changes, approves, or rejects. Admission can't repair existing problems or respond to external changes; reconciliation alone catches avoidable regressions only after they land. Advisory review and enforced admission are different powers and must be explicit.

4. Engineering objectives

Software Engineering at Google describes software engineering as “programming integrated over time.” [2] Over that time, objectives such as these need sustained attention:

  • Production health: Availability and tail-latency SLOs; correctness checks; capacity headroom; early regression detection.
  • Security: Advisory-to-artifact tracing; reachability and exposure analysis; verified remediation or authorized exceptions; least privilege, secret rotation, and trust-boundary enforcement.
  • Resilience: Failure-mode analysis; bounded fault injection; verified failover, graceful degradation, and recovery under dependency failures, infrastructure outages, traffic shifts, and data growth.
  • Efficiency: Resource right-sizing and utilization; infrastructure cost; CI cost per completed run; removal of redundant computation and storage.
  • Code quality: Semantic duplication; consistency across related implementations; complexity and abstraction quality; obsolete path removal.
  • Upgrades: Dependency and runtime support windows; breaking API migrations; deprecated API removal; compatibility with changing external services.
  • CI and delivery: Time to actionable required results, including queueing and retries; reproducible builds; verified deployments and rollbacks; feature-flag retirement.
  • Testing: Meaningful branch and behavioral coverage; flaky-test diagnosis; representative fixtures; regression tests for discovered failures.
  • Observability: High-signal metrics, traces, logs, and alerts; sufficient diagnostic context; bounded noise, cost, and sensitive-data exposure.
  • Data integrity: Consistency constraints; safe schema evolution; retention enforcement; tested backup restoration.
  • System understanding: Accurate architecture and API documentation; explicit interfaces and dependencies; recorded decisions and operational knowledge discoverable by humans and agents.

Viewed as a backlog, these objectives produce an endless stream of tasks. Viewed as desired conditions, they become continuing responsibilities for controllers. Two examples make the distinction concrete.

5. Example: duplication and linked changes

Consider a desired condition for a healthy codebase: semantic duplication is minimal, and necessary duplication remains consistent. Google's LINT.IfChange and LINT.ThenChange annotations make these relationships explicit and checkable. [3]

A minimal, hypothetical AI controller for this objective looks like this:

---
name: semantic-duplication-reducer
triggers:
  schedule: weekly
  events: [pull_request]
repositories:
  - example/web
  - example/backend
permissions:
  github:
    contents: write
    pull-requests: write
    checks: read
---

## Desired condition

Behavior governed by the same requirement has a single authoritative
implementation or specification. When separate implementations are necessary,
they are linked with LINT.IfChange and LINT.ThenChange, and the stated
relationship remains valid.

In both modes, the agent must reason about meaning because similar text may represent different responsibilities, while equivalent behavior may be implemented differently across languages.

Google's markers report when one related block changes without the other. [3] Here, the AI controller uses the same annotations to check the semantic relationship rather than requiring a mechanical co-change. If two linked implementations must accept the same inputs, renaming a local variable leaves the relationship intact; adding a new input format requires both implementations to change.

6. Example: production health

Now consider a more complex desired condition, beyond the reach of linters and static analyzers because it can't be evaluated from source code alone. A production service must meet its availability and tail-latency SLOs; its workloads must remain healthy, schedulable, and sufficiently provisioned. A production-health controller can investigate deviations using read-only access to Datadog through an MCP server and to Kubernetes through an RBAC role.

---
name: production-health-stabilizer
triggers:
  schedule: every 15 minutes
  events: [datadog_alert, pull_request]
clusters:
  - production
services:
  - api
repositories:
  - example/api
  - example/infrastructure
escalation:
  issue: linear
  notify: slack
permissions:
  github:
    contents: write
    pull-requests: write
  datadog: read
  kubernetes:
    observe: production-health-observer
    intervene:
      role: production-health-operator
      approval: required
      actions: [scale, restart, adjust-resources]
---

## Desired condition

The API meets its availability and tail-latency SLOs. Its Kubernetes workloads
remain healthy, schedulable, and provisioned with sufficient capacity.

Its normal path is a pull request changing application code or infrastructure configuration. When waiting for review and deployment would endanger production, it can propose a direct Kubernetes intervention. After human approval, it receives narrowly scoped, short-lived operator access to scale a deployment, restart a stuck rollout, or adjust its resources.

Consider a hypothetical sequence across several runs. A Datadog alert signals rising tail latency. The controller finds two plausible causes: CPU throttling and elevated database lock waits. The available evidence does not distinguish between them.

The controller proposes a bounded experiment: temporarily raise the CPU limit for a canary workload. A human approves the experiment and its rollback plan. On its next run, the controller finds that throttling has disappeared but latency has not improved, while lock waits remain elevated. It restores the original resource allocation and records that the CPU hypothesis was not supported under this workload.

The controller then traces the lock waits to a transaction that holds locks while performing unrelated work. It opens a pull request that shortens the transaction, waits for review and deployment, and observes the service again. Tail latency meets its SLO again, and lock waits return to baseline.

That outcome changes later behavior. When the same combination of symptoms recurs, the controller investigates lock contention before proposing more capacity. In admission mode, it uses the earlier production evidence to identify and explain a proposed change that reintroduces the same transaction pattern.

Not every investigation ends in a safe change. An unavailable image can leave replacement pods in ImagePullBackOff. Scaling can't help: every new pod will fail. If deployment history and repository state do not reveal the intended image, the controller escalates with the production impact, supporting evidence, and reason no safe action is available.

7. Cooperating and competing controllers

“A system of local optimums is not an optimum system at all.”

—Eliyahu M. Goldratt, The Goal

The harder case is when controllers' objectives conflict. Consider three controllers: one improves meaningful unit-test coverage, one reduces the time to actionable CI results, and one reduces CI cost per completed run. Each can move the system toward its own objective while moving it away from one or both of the others.

Suppose the coverage controller finds important parser behavior without tests and opens a pull request adding a large property-based test suite. The change improves meaningful coverage but makes required CI slower and more expensive. The runtime controller proposes sharding the suite across more runners. Feedback arrives sooner, but duplicated setup and additional runners prompt the cost controller to object.

The proposed model is hierarchical: a higher-level controller mediates among controllers with narrower objectives. Here, a CI controller owns the combined desired condition: sufficient confidence within an acceptable feedback time and budget. It coordinates proposals and experiments and asks each controller to evaluate the same proposals and results. It also acts as the admission gate for their changes: it blocks changes that violate binding constraints and mediates conflicts within its delegated authority.

The CI controller first searches for a change that improves one or more objectives without degrading the others, such as reusing artifacts or partitioning tests across existing workers. Each lower-level controller evaluates the proposal; the CI controller observes the combined result.

Some conflicts cannot be resolved by measurement alone. Suppose none of these changes is sufficient and the only practical way to shorten CI feedback is to add more runners. The runtime controller can estimate the time saved; the cost controller can estimate the additional monthly spend. Neither measurement determines how much the organization should pay for developer velocity. Unless policy already expresses that preference, the CI controller must escalate the decision, with its evidence and alternatives, to an engineering leader with the authority to make it.

The CI controller records that decision as durable guidance. It can authorize the change, establish a budget constraint, or reject it, then observe whether the expected benefit and cost materialize. It can revisit the decision when workloads, prices, or priorities change. The hierarchy does not eliminate judgment; it routes each decision to a controller or human with the necessary scope and authority. It reduces churn without disguising value judgments as technical conclusions.

8. The objective persists

The agentic revolution is accelerating software change. The same capabilities can sustain the reliability, security, efficiency, and comprehensibility of the systems being changed.

An AI controller makes the objective, rather than the task, the durable unit of engineering work. Tasks still end. The controller’s responsibility for the objective does not.


[1]See the Google Testing Blog's Sensenmann: Code Deletion at Scale.
[2]See Software Engineering at Google.
[3]See ChromiumOS's guide to Gerrit IfThisThenThat lint.

Comments