Dr. Mirko Kämpf

CEO & co-founder

Dr. Mirko Kämpf combines a background in big-data streaming (Cloudera, Confluent) and sustainable fintech infrastructure (ecolytiq) with open-source leadership for Apache Wayang. As part of the Wayang PMC and contributor to its community-driven growth, he brings vision and credibility to scalable federated data processing. At Scalytics he oversees solutions that deliver private AI and real-time analytics directly on existing enterprise data infrastructure.

Enterprise Agent Runbooks: Operationalizing Agentic Systems at Scale

Enterprise agent runbooks address asynchronous failure modes in distributed Kafka systems. Traditional incident procedures assume synchronous failures; agentic systems fail gradually over hours. Structured runbooks targeting four characteristic failure patterns (downstream backlog, state corruption, resource exhaustion, partition rebalancing) reduce mean-time-to-recovery from 240 minutes to 30-60 minutes. Implementation requires baseline metrics, connection resilience, observability instrumentation, and Kafka-specific recovery procedures. Teams establish clear escalation paths and quarterly validation.

Dr. Mirko Kämpf

Kafka DevOps Tooling: JulieOps vs. Alternatives (Vendor-Agnostic Analysis)

Kafka DevOps tooling divides into vendor-supported imperative options (Confluent Control Center, CLI) and unsupported declarative alternatives (JulieOps, Strimzi, Terraform). JulieOps leads in multi-distro flexibility but lacks enterprise support; Confluent excels in observability but fails at infrastructure-as-code. Vendor-agnostic analysis evaluates trade-offs for 50+ cluster environments. Platform architects gain selection criteria, benchmarks, and hybrid implementation patterns for reducing lock-in and config drift without operational debt.

Dr. Mirko Kämpf

Why Kafka DevOps Needs Vendor-Agnostic Tooling

Kafka DevOps faces a paradox: vendor tools like Confluent Control Center and CLI lack GitOps support for declarative infrastructure, while community tools like JulieOps provide GitOps but no enterprise backing. Vendor-agnostic approaches integrate JulieOps for configuration management with Control Center for observability, enabling multi-cluster operations without lock-in. Engineering leaders managing self-managed Kafka deployments gain strategies for reducing configuration drift, improving operational consistency, and avoiding vendor constraints. Implementation requires hybrid tooling, professional advisory, and clear role separation between declarative config and real-time monitoring.

Dr. Mirko Kämpf

Apache Wayang Proposals Mapped as a Knowledge Graph

Apache Wayang received 37 GSoC proposals, with 17 focused on a JDBC driver. Analyzing the proposals using a knowledge graph revealed distinct approaches to the same problem, highlighting the need for a JDBC driver. The analysis also identified other key areas for improvement, including a DataFrame API and Datalake-Friendly Backends, emphasizing the importance of interface gaps over core engine gaps.

Dr. Mirko Kämpf

Mounting a Graph: What I Learned Building Shared Memory for AI Agents

Multi-agent LLM systems have a shared memory problem that vector databases cannot solve: stale embeddings, no expiry semantics, and no way to traverse a knowledge graph without a query engine. This article covers how we built KafGraph, a binary-packed knowledge graph on Kafka with tombstone expiry and lease semantics, and why binary grep over memory-mapped partitions outperforms RAG for agentic workloads. Includes a transcript of a session with KafClaw, our agent runtime, querying its own two-year knowledge corpus as node://.

Dr. Mirko Kämpf

Tracing Multi-Agent Systems in Production

Traditional observability tools struggle with multi-agent systems due to long-running sessions, sub-agent execution chains, and external context. To address this, a Kafka-first architecture was implemented, leveraging Kafka’s event streaming capabilities for ordered, replayable session data. This approach enables efficient debugging, pattern detection, and memory persistence, transforming Kafka into a memory persistence layer for agent workflows. A production-grade observability pipeline for AI agents streams events to Kafka, enabling real-time dashboards and batch analytics. The pipeline uses Kafka for event streaming, Redis for fast dashboard queries, and Spark for batch pattern detection. This approach has significantly reduced debugging time, improved cost visibility, and enabled daily deployments.

Dr. Mirko Kämpf

How KafScale solves backpressure, retries, and coordination without a central orchestrator

Part 3 closes the series with the stuff that actually breaks multi-agent systems in production: backpressure, retries, duplicate work, and brittle “orchestrator brains” that become a single point of failure. KafScale flips the model by leaning on Kafka’s native primitives instead of layering custom coordination logic on top. Consumer groups give you horizontal scaling without manual load balancing. Partition lag becomes real, observable backpressure instead of hidden queues that explode memory. Retries and dead-letter topics turn failures into routable events, not cascading outages. And because the log is durable, you get replay and “time travel” debugging for free, which means you can validate new agents and models against historical traffic without touching live flows.

Dr. Mirko Kämpf

The Agent Abstraction Problem: Building portable components that survive framework churn

Part 2 - The Agent Abstraction Problem arises when tightly coupled agent logic to framework primitives makes agents fragile and expensive to maintain. KafScale solves this by using Kafka as a portable substrate, allowing agents to be framework-agnostic components that communicate through durable event streams. This approach enables portable components, durable contracts, and framework-agnostic patterns, making agents more resilient to framework changes and easier to test and deploy.

Dr. Mirko Kämpf

How KafScale transforms raw Kafka data into agent-ready context

Part 1 - KafScale’s Cognitive Lens pattern transforms raw Kafka data into agent-ready context through three complementary agent roles: Interpreter agents translate events into meaningful signals, Exposer agents detect patterns across events, and Enhancer agents enrich event streams with external data. This layered approach allows for independent deployment, testing, and reuse of agents, enabling scalable and flexible multi-agent systems. KafScale ensures agents remain portable across frameworks while maintaining strong operational guarantees.

Dr. Mirko Kämpf

Breaking the SAP IDoc monolith: from file payloads to Enterprise Object Streaming

How we accidentally invented Enterprise Object Streaming with KafScale. We spent two years pushing full SAP IDoc XML files into Kafka as single messages. It worked until peak season tripled volume. Consumers lagged two hours, billing stalled, and inventory updates broke. The issue was architectural: we were shipping files, not events. We rebuilt ingestion to explode each IDoc into segment-level topics, partitioned by document number and versioned with Schema Registry. Consumers subscribe only to what they need. Message sizes dropped from megabytes to kilobytes, lag fell below 30 seconds, and schema changes no longer require cross-team coordination.

Dr. Mirko Kämpf

Streaming Intelligence: Real-Time Analytics Beyond BI

Data warehouses deliver yesterday's insights. Streaming Intelligence processes ERP events, IoT telemetry, and business transactions in real-time—enabling fraud detection in milliseconds, dynamic pricing, and predictive maintenance without batch delays. Production-proven on SAP, Oracle, and Dynamics.

Dr. Mirko Kämpf

Enterprise Data Fragmentation: The Scale of the Challenge

Enterprise data environments are fragmented across hundreds of systems, making access, trust and orchestration difficult. Scalytics Federated unifies Kafka, Flink and Apache Wayang to provide real time execution, federated access and an assistive data concierge for faster insights. This architecture reduces integration effort and enables immediate value without centralizing data.

Dr. Mirko Kämpf

DLQ AI Agent: Intelligent Dead Letter Queue Routing

Dead Letter Queues accumulate failed messages that require manual triage. Our DLQ Agent uses AI to automatically classify failure reasons, route to appropriate handlers, and suggest remediations—turning a compliance headache into an intelligent self-healing system with complete audit trails.

Dr. Mirko Kämpf

Strategic Proposal Assistant: Governed, Data-Local AI for Enterprise Bid Management

This article explains how the Strategic Proposal Assistant uses ACP governance and data-local execution in Scalytics Federated to analyze project briefs securely and accurately. It shows how the system verifies claims, evaluates capabilities, and produces defensible recommendations grounded in distributed internal knowledge.

Dr. Mirko Kämpf

Kafka AI Integration: Smart Topics with Scalytics Connect

This article shows how Scalytics Federated extends Kafka with Streaming Intelligence, ACP governance, and contextual retrieval. It explains how enriched streams, continuous learning, and secure agent integration deliver real-time insights while keeping sensitive data at the source.

Dr. Mirko Kämpf

KafScale MCP server: connect AI agents to Kafka streaming operations

This article explains how Scalytics Federated uses MCP, Kafka, ACP governance, and Wayang based execution plans to deliver secure, real-time agentic RAG on sensitive data. It shows how the architecture keeps data at the source, enforces compliance, and scales across complex enterprise environments.

Dr. Mirko Kämpf

Building Governed Multi Agent Systems with MCP and the Agent Context Protocol

This article explains how Scalytics Federated extends the Model Context Protocol with the Agent Context Protocol to govern multi agent systems. It shows how ACP adds policy based access control, auditability, and secure communication, enabling agents to collaborate across environments without losing compliance or control.

Dr. Mirko Kämpf

Scalytics Partner Program for ISVs and MSPs

The Scalytics Partner Program enables ISVs and MSPs to deliver compliant and scalable data processing and AI solutions through federated execution. Partners can work across distributed data sources without moving or replicating information, which removes silos, reduces cloud spending, and supports strict regulatory requirements such as GDPR and HIPAA. Built on technology created by the original team behind Apache Wayang, Scalytics Federated gives partners a practical path to enable enterprise level analytics and AI while expanding their service portfolio and revenue potential.

Dr. Mirko Kämpf

Part 2: Shift Left Architecture for Secure Enterprise AI

Data quality problems discovered in production cost 10x more to fix than those caught at ingestion. Shift-left architecture embeds validation, governance, and transformation at data sources—reducing pipeline failures by 60% while enabling real-time analytics on clean data.

Dr. Mirko Kämpf

Part 1: Data Firewalls, Federated Zones, and the End of Centralized Data Architectures

Traditional data lake and warehouse strategies failed to eliminate silos because they removed data from its governance context. Federated architectures respect the boundaries created by data firewalls and allow computation to enter each zone securely. Scalytics Federated brings this model to enterprise scale. It enables analytics and AI without data movement, strengthens sovereignty, and aligns with regulatory demands in finance, energy, healthcare, and public sector environments.

Dr. Mirko Kämpf

Beyond the Data Platform: Why AI Requires a Federated Execution Layer

AI adoption is limited by data movement, governance constraints, and fragmented systems, not by model capability. This article explains why centralized platforms no longer meet the requirements of distributed, regulated data environments. It introduces federated execution as the architectural layer that enables training, inference, and analytics to run where the data resides, unifying existing systems without migration. This approach provides a practical path for enterprise AI that respects locality, reduces cost, and accelerates deployment.

Dr. Mirko Kämpf

ETL vs ELT: Developer's Guide

ETL transforms before loading; ELT transforms after. But the right choice depends on data volumes, transformation complexity, and target platform capabilities. This developer guide includes decision trees and code examples for Spark, dbt, and Wayang implementations.

Dr. Mirko Kämpf

Scalytics Partner Program | Simplify Distributed Data Projects and Eliminate Integration Problems

Scalytics Partners deliver distributed AI and data solutions without consolidating data or maintaining complex infrastructure. Our program provides technical enablement and go to market support for firms that need to operate across heterogeneous and regulated environments where traditional platforms cannot.

Dr. Mirko Kämpf

The Energy Cost of AI Training and the Role of Federated Learning

Energy is the real currency of AI. Centralized training burns energy in one place. Federated execution spreads energy use across existing infrastructure, eliminates unnecessary data movement, and lowers total power demand. Scalytics Federated enables efficient, distributed model training without expanding data center footprint.

Dr. Mirko Kämpf

Membership Inference Attacks: AI Privacy Risks & Solutions

Can attackers tell if your customer data trained a model? Membership inference attacks expose serious privacy risks in centralized AI. Learn how federated learning with differential privacy prevents data leakage while maintaining model accuracy for sensitive healthcare and financial applications.

Dr. Mirko Kämpf

back to all articles

Categories

Product Updates

Industry Solutions

Data Architecture

Streaming Intelligence

Federated Learning