Jeremy Frenay

Anyone can build an agent that suggests a fix for a Kafka incident. The hard part is letting it act on production, where one bad call can purge a topic or reset consumer offsets and take real data down with it. This is a tales-from-the-trenches talk about an SRE agent we built to operate Kafka, and mostly about the part nobody puts in the demo: how we permissioned it so that even a fully compromised agent can't do catastrophic damage. A Kafka SRE agent is close to a worst case. It reads private telemetry. It ingests content we don't control, so a single poisoned log line becomes a prompt-injection vector. And it can do things you can't take back. So we stopped trusting the model to be careful and moved the authority into the permission layer instead. The model only ever gets to recommend. The model we landed on, after a couple of incidents taught us better, tiers actions by blast radius. Read-only diagnostics like consumer lag, under-replicated partitions, and broker health run on their own. Reversible actions like restarting a connector task or nudging a quota run too, but everything gets logged. The destructive stuff (deleting topics, resetting offsets, reassigning partitions)…

MCP's new Tasks primitive makes tool calls asynchronous: a request returns a durable handle now, and the result arrives later. That's the right model for work that runs for minutes or hours, like ETL jobs, deep research, or batch reasoning, but the spec leaves the hard parts to implementers. Where...