Open Source: Apache Wayang, KafScale, KafClaw, and Scalytics Engineering

Follow Scalytics open-source engineering across Apache Wayang, KafScale, KafClaw, federated execution, Kafka-compatible systems, and agent infrastructure.

Apache Wayang Proposals Mapped as a Knowledge Graph

Apache Wayang received 37 GSoC proposals, with 17 focused on a JDBC driver. Analyzing the proposals using a knowledge graph revealed distinct approaches to the same problem, highlighting the need for a JDBC driver. The analysis also identified other key areas for improvement, including a DataFrame API and Datalake-Friendly Backends, emphasizing the importance of interface gaps over core engine gaps.

Author:

Dr. Mirko Kämpf

Apache Hadoop vs Apache Spark vs Apache Wayang: The Evolution of Big Data Processing

Apache Wayang became an Apache Top-Level Project in December 2025, reflecting a shift in how modern data architectures are designed. This article explains how big data processing evolved from Hadoop’s batch-oriented MapReduce model to Spark’s in-memory execution, and why cross-platform optimization is now necessary as analytics spans databases, streaming systems, and distributed engines. Drawing on peer-reviewed research and benchmark results, it outlines how Apache Wayang addresses the growing fragmentation of enterprise data platforms.

Author:

Alexander Alten

Distributed Processing for LLM Training: Unify Your Data Lakes

Large language models require diverse training data scattered across your organization. Distributed data processing with Apache Wayang lets you unify data lakes for LLM fine-tuning without moving sensitive data—enabling compliant AI training across healthcare, finance, and government sectors.

Author:

Alexander Alten

Cross-Platform Analytics Benchmark: 10x Performance Gains

We benchmarked Scalytics Connect against standalone Spark, Flink, and Java across TPC-H and ML workloads. Results: cross-platform optimization delivered 7-10x faster execution on complex joins and 40% lower infrastructure costs through intelligent workload distribution.

Author:

Dr. Zoi Kaoudi

AI Query Optimizer: ML-Powered Data Processing Speed

Traditional query optimizers fail with cross-platform workloads. Our learning-based query optimizer uses machine learning to automatically select optimal execution plans across Spark, Flink, and Postgres and reduce query times by up to 10x on complex analytical workloads.

Author:

Dr. Zoi Kaoudi

Apache Wayang: The Complete Guide

Apache Wayang enables unified cross-platform data processing across Spark, Flink, and Java in a single pipeline. Created by Scalytics founders, now an Apache TLP. Learn how to reduce processing costs by 40% while eliminating vendor lock-in with this comprehensive implementation guide.

Author:

Alexander Alten

Scalytics Architecture Deep Dive: Wayang & Federated AI

A technical deep dive into the Scalytics architecture: how Apache Wayang powers our federated data layer, the ML-based cost optimizer, and our approach to privacy-preserving AI. Essential reading for data architects evaluating cross-platform processing solutions.

Author:

Dr. Jorge Quiané