The Bottom Line
KafScale 2.0 took us about four weeks. Around 20 AI agents did hundreds of hours of the hands-on work on KafScale 2.0, burned through 4.6 billion tokens and ran more than 86 hours of QA. Our senior engineers did not go anywhere. They steered the agents, checked what came back and made every merge and release call themselves. All of it ran on Lascaris, our own platform, with agent messaging over Kafka, KafGraph as the shared brain and a policy check before every action. What we got out of it was time. Time for coordination, for strategy and product thinking, and for decisions nobody had to rush.
Why we let agents build a release
KafScale 2.0 is the first release where AI agents did most of the hands-on work. Fixes, tests, docs, the long QA runs. That is the stuff we used to give to less experienced freelancers. It felt like the whole team had unlocked a new skill tree.
There is a catch, though. It only worked because experienced senior engineers were running the agents. Operating agents is its own skill, and it needs senior judgment. A team that has that judgment gets time back. A team without it mostly gets a lot of output.
A quick note before we start: we wrote this post ourselves, by hand. The images come from our docs team, which is AI, of course.
We also did not borrow someone else's platform for this. The agents ran on Lascaris, the same platform we sell to enterprises that want governed AI agents. That is where they talked to each other, handed off work, shared memory and waited for a human to approve. Below is what worked, what broke, and what it taught us about the people who run agents.
Running AI agents is a skill of its own, and it needs senior judgment. When a team has that, it gets something rare back: time.
Alexander Alten, CTO
What we were building
KafScale is our open source, S3 native Kafka for Kubernetes. The brokers are stateless and the data lives in object storage. 2.0 is a big one. It touches the brokers, the proxy, the operator, the console and the schema registry, plus the docs and even the demo videos. From the v1.6.3 tag on 22 October 2026 to the QA GO on 19 November 2026, it took about four weeks.
For a small team, that is a lot of ground. Normally we would have brought in extra hands for the well scoped parts and kept the core people on the hard problems. Not this time.
From freelancers to agents
We like freelancers. Software has always been built with them, and a good one is worth a lot. But the jobs we gave to freelancers early in their careers were always well scoped. Fix this clear issue. Write these tests. Update that doc. Do a QA pass.
That is exactly what the agents picked up. They can run many of those jobs at the same time, and they never get sick of the boring ones. One thing stays the same: someone still has to know what good looks like.
With a freelancer, a senior writes the brief, reviews the result and pushes back if it is not good enough. With agents it is the same job, only faster and many times at once. Take the senior out and nobody pushes back.
What the agents did
The agents did the work that normally eats a release. Code and fixes. Tests, a lot of them. They reviewed each other's pull requests. They ran full QA on real clusters, broke things on purpose and wrote down what happened.
They also did things a small team almost never gets to. Demo videos for different user roles, for one. And they kept rewriting the docs until every command in them could be pasted into a shell and actually worked.
Eating our own food: how we ran the agents on Lascaris
This is the part we care about most. Lascaris ships governed AI agents for enterprises, with KafClaw built in. We tell customers to run their agents that way. Running ours any other way would be a bit embarrassing, so we didn't.
One job per agent
Rule one of managing agents: one agent, one job. Builder agents wrote fixes. A QA agent ran the release gate and wrote the reports. A reviewer agent looked at every pull request into main. Later we added a second reviewer, and its only job was to review the QA agent's reviews.
Sounds obvious. It still matters a lot. Let an agent review its own fix and it will approve it. People do the same, to be fair. A separate reviewer with its own instructions catches what the author missed.
Managing agents also means changing the team when something isn't working, and keeping an eye on cost. When the QA agent missed a bug in a review, the senior engineer running the release pulled it out of the review role and left it on the release gate. When token spend climbed too fast, we paused a builder agent for a while. Full QA on every small fix got too slow and too expensive, so the rule became one full QA after a batch of fixes. Those are management calls, and an agent won't make them about itself.
Messages and handoffs
Agents working together need a clean way to pass work along. Ours ran their messaging over Lascaris. KafClaw, the agent runtime inside it, lets agents talk through typed messages on Kafka topics. Every message carries a correlation ID and a trace, so you can follow one piece of work from agent to agent.
The agents also sorted out their own coordination. When they needed a new channel, they created the Kafka topics and listeners themselves instead of waiting for one of us to wire it up. So the coordination sat on the stream, right next to the work. We made the longer case for agents on the stream in why Kafka and MCP are not enough for autonomous AI agents.
Turns out the format of a handoff mattered more than the transport. In the beginning the QA agent sent findings one at a time. The builder fixed one thing and pushed, then QA found the next thing. One docs pull request dragged on for hours like that. So we changed the handoff:
Webflow embed:
Isn't that loop boring? Yes. Good. Every step has one owner, every handoff is complete, and nothing gets dropped between agents.
A shared brain
Agents forget. Every new run starts at zero unless something remembers for them, which is why the shared brain matters so much.
Every miss became a written rule. Read every review before you call something ready. Never put a secret on the command line in a docs example. Test with more than one broker, because a single broker hides real bugs. Check the math in code you reuse, not just in code you write. All of that went into the memory the agents work from, so the next run started with those rules and didn't repeat the mistake.
Lascaris comes with KafGraph built in. It is our open source memory graph for agent teams, and on KafScale 2.0 it was the shared brain. Once one agent learned something, the others had it too. Why agent memory needs governance is its own topic, and we wrote it up in agent memory governance through decision records.
Policy before action
In Lascaris, rules get checked before an agent acts. Not after. If an action fails the rule, it does not run. That is the core of how AI agents on Kafka work for us: sense, decide, check, record.
The policy that mattered most on KafScale 2.0 was approval. Only a senior engineer approves a merge. Only a senior engineer decides to file an issue or publish anything. A message from an agent never counts as approval, even when it comes from another agent that sounds very sure of itself.
We also didn't hand out broad access just because it would have been convenient. Release QA ran when a senior engineer asked for it, never on a timer. More on why agents shouldn't hold direct access to your tools in stop giving AI agents direct access to tools.
The release gate
A release decision came from one thing only: a written QA report on a fixed version of the code. A chat message didn't count. Neither did a green badge in CI. The report covered the test suites, CI, code scanning and real client SDKs, plus state change tests on real Kubernetes clusters. That means broker restarts, scaling up and down under load, storage outages and secret rotation.
The agents said no more often than yes, and some of those no's saved the release. Details below. A gate that only ever says go is not a gate.
A trail you can follow back
Everything the agents did left a trace a human can read. Pull requests, reviews, QA reports and patches are all on record in the repo history. The messages between agents sit on Kafka topics, so you can replay what got handed over, and when.
So when something went wrong, we could go back and see who found it, who missed it and what changed afterwards. If you run agents in production, you need that trail. We wrote about how to get one in agent observability for multi agent systems.
What the agents found that we didn't
Every release hits a point where the tests are green and everybody wants to ship. On KafScale 2.0 the QA agent kept going past that point. It wrote eight full QA reports over the release, and only two of them said GO. Below are five bugs it found by running the product the way users run it, which is not how our tests ran it. Every one of them had already slipped past unit tests, CI or review.
The GO that became a NO GO
Late in the release, everything was green. Race tests, SDKs, examples, CI and the official operator end to end test on Kubernetes all passed on the same commit, and the end to end test passed three times in a row. Then the QA agent tried something none of those suites did. It started a cluster with two brokers splitting the partitions between them, and consumed from the earliest offset through the Kafka proxy, which is the main way in for 2.0.
About half the partitions never started. On 4 of 8 proxy and partition pairs, a test client got 0 records in 20 seconds. Talking to the brokers directly, every pair worked. Here is what happened. The proxy sent the earliest offset request over one fixed connection to one broker. If that broker didn't own the partition, it said so, and the client asked the same broker again. And again. The agent called NO GO and filed #355 at 12:32, while it was still writing the report. The bug was there since before we started to work on 2.0.
The fix, #363, sends that request to the broker that actually owns the partition. Opened at 12:46, merged at 15:05, same day. That evening the QA agent called the tree a GO.
Gates passed in the morning, withdrawn the same day
Earlier in the release there was a morning that started well. Between 10:50 and 11:27 the required release gates found no product defect. The full build, the race tests, real clients, the schema registry and the high availability tests on Kubernetes all passed.
Then the QA agent ran an extended check on the same commit. 28,794,235 fuzz executions with no crash. A 30 minute live soak where 35,981 records were acknowledged and consumed with no loss and no duplicates. And a live test of the large file SDKs against real S3 compatible storage. That one uploads a file, consumes it through Kafka, flips the first byte of the stored object and reads it again. Python and Java rejected the corrupted object. JavaScript handed back the corrupted 262,144 byte payload with no error, even with checksum validation switched on. Which is the default, by the way. The check in the code was just a placeholder.
Unit tests had passed. Code scanning reported 0 alerts. CI had no probe that corrupts a real S3 object. The agent filed #297 at 12:01 and pulled its go-ahead the same day, with a written verdict of NOT READY. The fix, #299, landed the next day, and it had to show the probe failing before the fix and passing after it. Had this shipped, a JavaScript consumer could have received silently corrupted data with the safety flag on.
A proxy that fell behind after the load test
In the final round of QA the agent ran heavy load against a cluster that scaled its brokers from 2 out to 8 and back to 2. Afterwards, a consumer reading many partitions through the Kafka proxy couldn't catch up. After more than ten minutes it was still a few hundred records short on 2 to 4 partitions. The same read straight through the brokers finished in 15 seconds.
No data was lost, to be clear. The proxy waited for every broker to answer each fetch, and brokers that were already caught up held their answer for the full wait time. Every round trip was slow, while the lagging partitions moved one tiny segment at a time. Shorter tests never piled up enough tiny segments to show it. The agent filed #388, and #399 fixed it.
The console panel a unit test said was fixed
KafScale can run two operator replicas for high availability. In that setup, the console panel for etcd snapshots could stay empty for good. Why? The console kept one connection open to scrape operator metrics. If that connection landed on the standby operator, which doesn't export the snapshot metrics, the panel never got a value. Fresh connections got the metrics 6 of 8 times. A client reusing one connection, the way the console does, failed 6 of 6 times.
An earlier change, #372, had already "fixed" this, and its unit tests passed. They just didn't model a connection that sticks to one pod. The docs even claimed the scrapes were balanced across replicas. The agent caught it on a real cluster with two replicas and filed #387. Since #398, the console finds each ready operator pod and scrapes it itself.
Docs that failed when you followed them
The docs for setting up MCP authentication told you to create a secret in the namespace before installing. The same page, and the chart default, also told Helm to create that namespace. Helm then refused to take over a namespace that already existed. Install failed.
The chart change behind it had been checked. Nobody had pasted the page into a fresh cluster and run it. The QA agent did exactly that, word for word, and filed #386. #397 fixed the setup and added a Helm test that renders the documented values.
Only an agent would bother
Now think about you, a human doing these checks. While the cluster scales from 2 to 8 brokers and back, you have a script to count every acknowledged record, all 338,467 of them, and confirms 0 missing, 0 duplicates and 0 offset gaps. Somebody else paste tests the docs, 148 bash blocks in 57 doc files, one command at a time. Then the same operator end to end test runs three times on the same commit, just to make sure a pass wasn't luck. Then someone watches the fuzz targets pile up 28,794,235 executions without a crash.
The 2.0 QA is about two-thirds done. The fixes for #386, #387 and #388 all hold up on kind, and CI is green on `188f0f8`.
The upgrade from 1.6.3 found one real problem. The documented `helm upgrade ... --reuse-values` fails, because 1.6.3 values have no `schemaRegistry` key. With `--reset-then-reuse-values` the upgrade works cleanly, and data, consumer offsets and rollback all hold. I'm also checking two things before I call them: the console showing invented metric numbers when the operator and brokers are scaled to zero, and a proxy that read nothing for 4 minutes after broker etcd sessions expired. The HPA scale-down and the soak are still running. Then you'll get the GO or NO-GO.
That is boring and frustrating work. For an agent it is just the right job. AI doesn't get bored on the third identical run. It doesn't skim the docs because they look fine. We keep coming back to this trade. The agents do the patient, repetitive checking, and the people get their time back for the calls that need judgment. What does this failure mean? What gets fixed first? Is the release ready?
No egos in the review
A bug report from an agent is a bug report. Nothing more. No history between the people involved, nobody worried about looking bad, no "I approve yours, you approve mine." When the QA agent flipped a GO to a NO GO late in the release, nobody had to defend an earlier call. The finding went on the topic, the builder agent picked it up, and the fix went through review like any other change.
That changes how bugs move. A flaky test doesn't get waved through because everyone is tired on a Friday. A doc that fails when you follow it gets flagged, even if a senior wrote it. A finding from the morning gets checked again in the afternoon, and if the evidence says so, the go-ahead is withdrawn the same day. Agents don't get defensive. They also don't let things slide to keep the peace.
It also changes the human side. Our senior engineers spent less time arguing about whether a problem was real and more time deciding what to do about it. When a bug was hard to reproduce, we did not start a discussion. We told the agent to run it more times and find the root cause. It came back with the numbers: how often the bug happened, under which circumstances, and how much it mattered in real-world use. The evidence was already there, with a link back to the run that produced it. That is a calmer way to ship, and in our experience it lets fewer bugs through.
People still make the calls. The agents just clear a lot of the friction on the way there.
What the QA agent missed
The agent and AI isn't perfect, and it says so itself. On #380, a change reused an existing metrics client to compute averages. That client skipped zero values. So for broker error rates of 0 and 0.4 it reported 0.4, when the correct average is 0.2. The QA agent had described the metric as an average and never audited the reused math. The review bot flagged it. Twice. A senior engineer fixed it and added a test that mixes zero and non zero values.
What the agent took from it: audit reused code like new code, especially edge cases such as zero values and missing samples.
The lesson the agent took: audit reused code like new code, especially edge cases such as zero values and missing samples.
"Green tests are not a release. Count every acknowledged record, and withdraw the go-ahead the same day a live probe finds silent corruption."
Our leading QA AI agent
Where the senior engineers came in
The agents made mistakes. Small ones, and some that mattered. A reviewer agent sometimes approved work it should have stopped. A builder agent sometimes fixed one thing and broke another. And suggestions that looked fine on paper fell over in a real cluster.
Senior engineers caught those, often with help from a second reviewer. They read the code the agents wrote, checked the math and asked the questions an agent doesn't think to ask. If something was wrong, they fixed it or sent it back. Take the metrics bug above. Fixing it by hand was faster and safer than another round with an agent.
Running agents is senior work
Yes, the job market is rough for people starting out right now. But read what developers actually say about working with agents, and a lot of it is good news.
In a Reddit thread about how AI changed programming, one developer with many years behind them describes a "programming renaissance": the boring grunt work goes to the agents, and they focus on "the direction, the research, and the learning". Another says things that used to become tech debt now get handled up front, and that they can try different designs without putting a whole project off schedule. A third writes that at work they "finally have more time" for stress, A/B and penetration testing.
That matches what we saw. Running agents well is not prompt writing. The person in charge scopes the work, decides which agent does what, reads what comes back and knows when an answer is wrong even when it looks right. That takes years of engineering experience, plus a few new habits. How to split work into agent roles. How to hand work over without losing anything. How to write a rule so the same mistake does not happen twice. How to set up a release gate that an agent cannot talk its way past.
So the agents did not replace the senior engineers. The seniors stopped spending their days on routine fixes and spent them guiding the agents, checking the results and making the calls. That shift is where the real payoff of AI agents comes from.
What does this leave us with?
The easy headline is that AI agents replace people. That is not what we saw. The agents took over tasks. The people got their time back, and they spent it on the work that only people can do. Here is what that time looks like.
- The pressure fades. When the routine work is covered, nobody has to rush the hard parts to make room for the easy ones. Deadlines stop being a fight against the backlog.
- More time to coordinate. Senior engineers spend less time on routine fixes and more time making sure the right work happens in the right order.
- Room to rethink strategy. When the backlog stops eating every week, a team can step back and ask if it is building the right thing at all. That question usually gets skipped because nobody has the time.
- Time to think about product and features. Instead of asking how to build something, the team can ask what users actually need and which features are worth building next.
- Real A/B testing. Agents can build several variants of a feature or a page in minutes. So instead of arguing about which version is better, you build both, test them on real systems and keep the winner.
- Trying things gets cheap. Agents make an experiment quick to set up and quick to throw away. Bad ideas fail fast and cost little. Good ideas get a real chance.
- Less slack and less tech debt. The jobs that always slip, like old tests, stale docs, small cleanups and half finished refactors, are exactly the jobs agents are good at. They get done instead of piling up.
- Projects back from the shelf. Every company has projects that were put on hold because they cost too much to staff. When agents carry the routine part, some of those projects make sense again. In one Reddit post, a developer describes a project shelved years ago because finishing it "would have taken a team". With AI, the old prototype was running again within a few prompts.
- Careful decisions. With more time and more evidence on the table, people can decide slowly where it matters. A release, a merge, an architecture change: these get a real decision from a senior engineer, not a rushed one.
This is optimization, not job cuts. The team stays. The work gets better. And the people who know the system finally have the time to use what they know.
The new skill tree
A small team with agents has reach it never had before. But a skill tree only helps if you know where to put the points, and that part stays human.
AI agents do not replace experience. They multiply it. Give agents to a team without senior engineers and you get a lot of code fast, with nobody who can tell if it is right. Give them to experienced engineers who know how to operate them and you get a team that moves like one much larger.
The platform matters too. Agents need a way to talk, a memory they share, rules that are checked before they act and a record of what they did. We built Lascaris for that, and we ran KafScale 2.0 on it ourselves.
If you want help from senior engineers who have done this, our Kafka professional services team works with you on your own platform. And if you run Kafka today, Lascaris also keeps an immutable backup of the Kafka you already run on S3.
Agents take the tasks. People keep the judgment, and finally get the time to use it.
Run your agents the way we run ours
You can see the result of this release on kafscale.io, and the code is open on GitHub. Fork it. Star it. Run it in production as we and customers do.
If you want to run your own agents with messaging, a shared brain and policy before action, inside your own network, take a look at Lascaris and book a call with us.