Data Engineering Blog | Kafka & Apache Wayang

Mounting a Graph: What I Learned Building Shared Memory for AI Agents

Multi-agent LLM systems have a shared memory problem that vector databases cannot solve: stale embeddings, no expiry semantics, and no way to traverse a knowledge graph without a query engine. This article covers how we built KafGraph, a binary-packed knowledge graph on Kafka with tombstone expiry and lease semantics, and why binary grep over memory-mapped partitions outperforms RAG for agentic workloads. Includes a transcript of a session with KafClaw, our agent runtime, querying its own two-year knowledge corpus as node://.

IBM Acquires Confluent for $11B: What Enterprise Architects Need to Evaluate Now

IBM announced the acquisition of Confluent for $11 billion in December 2025, paying $31 per share with a close expected by mid-2026. More than 6,500 enterprise clients - over 40% of the Fortune 500 - now face concrete questions about pricing trajectory, roadmap continuity, and support model changes as Confluent moves into IBM's enterprise software structure. Apache Kafka itself is unaffected: it remains an Apache Software Foundation project under Apache 2.0, and the Kafka wire protocol means every Confluent deployment is portable to any compatible broker without client code changes. This article maps what enterprise architects need to evaluate before the deal closes, and why AI agent infrastructure should be decoupled from broker choice regardless of who owns the managed service.

KafSIEM: Kafka Open-Source Intelligence for Real-Time Threat Monitoring

KafSIEM provides open-source intelligence gathering for real-time threat detection using Kafka as the streaming backbone. Organizations replace expensive SIEM platforms with custom detection stacks that monitor regulatory announcements, threat feeds, and internal signals. KafScale enables sub-second latency across multiple data sources. Detection rules are human-readable, auditable, and adaptable to sector-specific threats (financial services, critical infrastructure, e-commerce). Implementation replaces SIEM licensing costs ($500K-2M annually) with streaming infrastructure. Real-time alerts and complete audit trails satisfy DORA and compliance requirements.

Tracing Multi-Agent Systems in Production

Traditional observability tools struggle with multi-agent systems due to long-running sessions, sub-agent execution chains, and external context. To address this, a Kafka-first architecture was implemented, leveraging Kafka’s event streaming capabilities for ordered, replayable session data. This approach enables efficient debugging, pattern detection, and memory persistence, transforming Kafka into a memory persistence layer for agent workflows. A production-grade observability pipeline for AI agents streams events to Kafka, enabling real-time dashboards and batch analytics. The pipeline uses Kafka for event streaming, Redis for fast dashboard queries, and Spark for batch pattern detection. This approach has significantly reduced debugging time, improved cost visibility, and enabled daily deployments.

When Open-Source Infrastructure Meets Enterprise Constraints: A Production Reality Check

Engineering organizations face challenges transitioning multi-agent AI systems from prototypes to production due to operational gaps. These gaps, often architectural, lead to issues like duplicate work and operational chaos. While open-source components like Kafka provide a foundation, successful implementation requires expertise in areas like cluster operations, observability, and cost optimization. Scalytics provides strategic consulting and operational support for production AI deployments, addressing the gap between controlled environments and production load. Their services include Kafka lifecycle management, agent observability platforms, and cost optimization, enabling organizations to focus on agent logic and business outcomes while Scalytics handles infrastructure operations and optimization. This approach accelerates AI agent deployment timelines and reduces operational risk.

How KafScale solves backpressure, retries, and coordination without a central orchestrator

Part 3 closes the series with the stuff that actually breaks multi-agent systems in production: backpressure, retries, duplicate work, and brittle “orchestrator brains” that become a single point of failure. KafScale flips the model by leaning on Kafka’s native primitives instead of layering custom coordination logic on top. Consumer groups give you horizontal scaling without manual load balancing. Partition lag becomes real, observable backpressure instead of hidden queues that explode memory. Retries and dead-letter topics turn failures into routable events, not cascading outages. And because the log is durable, you get replay and “time travel” debugging for free, which means you can validate new agents and models against historical traffic without touching live flows.

The Agent Abstraction Problem: Building portable components that survive framework churn

Part 2 - The Agent Abstraction Problem arises when tightly coupled agent logic to framework primitives makes agents fragile and expensive to maintain. KafScale solves this by using Kafka as a portable substrate, allowing agents to be framework-agnostic components that communicate through durable event streams. This approach enables portable components, durable contracts, and framework-agnostic patterns, making agents more resilient to framework changes and easier to test and deploy.

How KafScale transforms raw Kafka data into agent-ready context

Part 1 - KafScale’s Cognitive Lens pattern transforms raw Kafka data into agent-ready context through three complementary agent roles: Interpreter agents translate events into meaningful signals, Exposer agents detect patterns across events, and Enhancer agents enrich event streams with external data. This layered approach allows for independent deployment, testing, and reuse of agents, enabling scalable and flexible multi-agent systems. KafScale ensures agents remain portable across frameworks while maintaining strong operational guarantees.

Unlock Faster ML & AI

Our Research.
Learn how Scalytics streamlines data pipelines, empowering businesses to achieve rapid AI success.

Download

The Scalytics Blog