Open infrastructure · No control plane · No vendor toll booth
The streaming spine for critical infrastructure and agentic AI. KafScale is the only Apache 2.0, S3-native, Kubernetes-first Apache Kafka compatible stack that runs where your platform runs and stays where your data is.
KafScale is purpose built on real-world experience and architectural decisions that make Kafka-compatible streaming usable for engineering teams who own the consequences of their infrastructure choices. These commitments hold across every release, every API endpoint, and every deployment. They are documented in the source, enforced in the build, and visible at the protocol layer.
Use it, modify it, redistribute it, sell services on top of it. No BSL clauses that convert in four years. No usage fees per GiB. No vendor control plane that holds your cluster hostage. The license is the same one Apache Kafka ships under, and KafScale is the only stateless, S3-native streaming platform that ships under it without an asterisk.
Brokers hold zero durable state. Immutable .kfs segments live in object storage with eleven-nines durability. Add a broker, the cluster scales. Remove a broker, the cluster shrugs. There is no partition rebalancing because there is no broker-local data to rebalance. Failure becomes a scheduling event, not an operational incident.
A proxy rewrites Kafka metadata responses so every client sees one DNS name. Brokers scale to N behind it. Clients never see a topology change, never trigger a reconnection storm, never need a custom SDK for most Kafka operations. KafScale scales with you, transparent for your producers and consumers, and S3 as storage layer.
The .kfs format is documented as part of the public specification. Processors for Iceberg sinks, SQL engines, AI agent retrievers read segments directly from S3, bypassing brokers entirely. Streaming traffic and analytical traffic read from S3 and never compete for the same compute. No other Kafka-compatible platform exposes its storage format this way.
A real operator with custom resource definitions for clusters and topics. HPA-driven scaling that does not require partition reassignment. Helm charts that deploy a working cluster in minutes. The deployment unit is a pod set on your cluster; not an agent in a vendor VPC, not a hosted service, not a binary you babysit.
Kafka wire protocol inbound and outbound. S3 as proven durability storage layer, etcd for metadata. No proprietary RPC, no vendor SDK, no closed segment format sitting between your producers and your data. If we lose the deal, your data leaves on the same standards it arrived on.
Each commitment is a red line in the build. They are documented in the source, enforced in the integration pipeline, and visible at the API. A buyer who wants to verify them does not need a sales call. The repository is at github.com/KafScale/platform.
KafScale accepts Kafka-protocol traffic from any standard client through a single proxy endpoint, flushes immutable .kfs segments to S3, and serves both real-time consumers through brokers and analytical workloads through direct-from-S3 processors. Streaming and analytics share the data; they never compete for the same compute.
For teams running Kafka with multi-terabyte partition disks, daily on-call pages from rebalance storms, and ETL pipelines that don't need sub-10ms latency. KafScale takes the load off broker disks and puts retention on S3 economics. Producers and consumers connect with their existing libraries.
▸ quickstart guideFor teams who run Confluent or Apache Kafka in production and need a parallel cluster for disaster recovery, agentic AI workloads, replay testing, or regulated workload separation. kaf-mirror live-syncs your production cluster to KafScale. Production stays untouched.
▸ see the pattern belowFor teams building agentic AI where decisions, tool calls, and reasoning chains need to be replayable, auditable, and storable for years without a per-GiB tax. The immutable S3 log becomes the substrate that agents query, reconcile, and reason over — bypassing the broker path entirely.
▸ KafClaw runtimeMost engineering teams running Confluent or Apache Kafka in production cannot put agentic AI workloads on the same cluster. Replay testing, prompt-history retention, batch reprocessing, and disaster-recovery drills all compete with the same brokers that serve real-time. KafScale plus kaf-mirror separates the two without changing a single line of producer code.
AI agents, batch reprocessors, and replay tests read from KafScale. The brokers serving your real-time traffic do not contend with reasoning chains that pull months of history.
If your primary cluster goes down or your renewal terms shift, KafScale is already a warm standby on S3. Cluster Linking and MirrorMaker 2 are options — KafScale is one with no per-GiB licensing and no proprietary control plane.
Standard broker-resident retention for prompt histories, agent decisions, and tool call logs gets expensive at multi-month scale. S3 economics let you keep years of agent state for the price of object storage, which compounds in your favour as agent workloads grow.
Mirror only the topics that need to leave production. Apply different retention, different ACLs, different encryption keys to the agent spine. The kaf-mirror layer is the policy boundary, not a downstream afterthought.
Scalytics is a Confluent partner. KafScale is positioned as a complementary spine for workloads where production isolation is required — not as a wholesale Confluent replacement. The kaf-mirror project is open source: github.com/scalytics/kaf-mirror.
KafScale is not a drop-in for every Kafka workload. The table below summarises the architectural and licensing tradeoffs that actually matter when you're choosing what runs your streaming spine for the next five years. The full comparison, including cost snapshots and feature parity, lives at kafscale.io/comparison.
| Platform | License | Storage model | Self-hostable | Best for |
|---|---|---|---|---|
| KafScale | Apache 2.0 | S3 only, etcd metadata | Yes, no control plane | BDR, agentic AI, ETL, regulated workloads |
| Apache Kafka | Apache 2.0 | Local disk, replicated | Yes | Sub-10ms latency, transactional workloads |
| Confluent Platform / Cloud | Proprietary | Local disk + tiered S3 | Self-managed Platform | Full Kafka ecosystem, ksqlDB, Connect |
| WarpStream | Proprietary | S3 + Confluent control plane | BYOC only | BYOC logging, observability |
| Redpanda | BSL 1.1 | Local disk + S3 tiering | Yes, with BSL terms | Low-latency Kafka replacement |
| AutoMQ | Apache 2.0* | S3 + EBS write-ahead log | Yes | Kafka migration with low latency |
| Bufstream | Proprietary | S3 + PostgreSQL metadata | Yes, with usage fees | Lakehouse, Protobuf-first pipelines |
WarpStream was acquired by Confluent in September 2024 and is now closed source. IBM announced acquisition of Confluent in December 2025, expected to close mid-2026. AutoMQ converted from BSL to Apache 2.0 in May 2025. Among Kafka-compatible streaming platforms shipping in April 2026, KafScale is the only one that is S3-native, stateless, fully self-hostable, and Apache 2.0 licensed without an asterisk for the eighty percent of workloads that do not require sub-10ms latency or exactly-once transactions.
A buyer running streaming for a regulated process, a BDR strategy, or an agentic platform reads architectural commitments more carefully than they read benchmarks. Each line below is enforced in the build pipeline, documented in the source, and visible at the protocol layer.
KafScale is designed for environments where a vendor cloud dependency is a dealbreaker. No external service calls in the data path. No telemetry back to the maintainers. The deployment unit is a pod set on your Kubernetes cluster: sovereignty and network boundaries stay your decision.
A Kubernetes cluster you already run, an S3 bucket, and an etcd ensemble. That is the entire dependency list. Zero phone-home, zero external auth, zero hosted control plane.
The KafScale operator manages clusters and topics through standard CRDs. kubectl apply -f topic.yaml creates a topic. HPA handles broker scaling without partition reassignment.
Any S3-compatible object store: AWS S3, MinIO, GCS via S3 API, Azure Blob via S3 API, on-prem Ceph. Lifecycle policies enforce retention. There is no second tier to manage.
S3 versioning, replication, and snapshots are the backup story. There is no broker-local state to dump, replay, or reconcile after a node failure. Recovery is a pod restart.
The data path has no external dependencies. KafScale runs in environments without internet egress, behind VPN, or in classified networks. Container images mirror to your registry; the cluster never reaches out.
The repository is at github.com/KafScale/platform. Inspect, fork, audit before purchase. There is no enterprise-only branch or hidden module.
The seven questions platform leads, OT security architects, and CTOs raise in every early call. Answered here so the briefing time can go to your actual problem.
The technical buyer reads this section more carefully than the case studies. A platform that admits what is shipped and what is in flight is one the buyer can plan against. A platform that claims everything is ready is one the buyer assumes is hiding the integration cost.
Stateless brokers, S3-native segment storage, etcd metadata, single-endpoint proxy, Kubernetes operator with cluster and topic CRDs, Kafka APIs covering produce, fetch, consumer groups, and topic administration. Helm chart and quickstart for kind clusters.
The Iceberg processor is shipping incrementally with Unity Catalog, Polaris, and AWS Glue support. The KAFSQL processor (Postgres-compatible SQL over .kfs segments) is also available and in use.
We are working with platform teams running Confluent or Apache Kafka in production who want to add a KafScale spine for BDR, agentic workloads, or long-retention analytics. If your gap is the gap described on this page, request a briefing.
KafScale is developed and maintained by Scalytics, the company founded by the original inventors of Apache Wayang (now an Apache Top-Level Project for federated data processing). The same architectural philosophy - compute where the data lives, on standards you can audit - runs through every part of KafScale.
The team includes Apache committers, former Confluent and Cloudera engineers, and operators from regulated enterprise environments. Scalytics is an active Confluent partner; KafScale is positioned as a complementary spine for workloads where production isolation, Apache 2.0 licensing, or agentic AI infrastructure are the deciding constraints.
45 minutes. Architecture, BDR pattern, kaf-mirror integration, agentic workload separation, design partner terms. Bring your hardest streaming question. Or read the source first.