Apache Wayang · Apache 2.0 · Lascaris add-on

Apache Wayang

Cross-platform data processing, in production.

One pipeline across Postgres, Spark, Flink and your ML stack. Wayang runs each step on the engine that fits, close to where the data lives, and moves only what the next step needs. The Wayang add-on for Lascaris runs it inside your own environment.

Write the pipeline once in Java, Scala, Python or SQL. Wayang picks the engine per step.
The optimizer prices every move of data between engines, not only each single step.
Apache top-level project since November 2025. Two Scalytics co-founders sit on the PMC.

Talk to us about Lascaris Plan a Spark/Flink migration

wayang plan · your enginesplanned

source

PostgreSQL

filters where the rows are

in placeno copy

prepare

Apache Spark

large join across the cluster

clusterdistributed

train

Java process

gradient steps on small data

single nodeno cluster

run

Lascaris

tested, hardened, under SLA

supportedyour network

inside your networkonly the next step's data moves

Engines

Spark, Flink, Postgres +

APIs

Java, Scala, Python, SQL

License

Apache 2.0

Apache Wayang. In production.

One pipeline. Every engine.

Apache Wayang runs each step on Postgres, Spark, Flink or your ML stack, where the data lives. Lascaris runs it inside your network.

Talk to us

In plain terms

What is Apache Wayang?

What it is

An open source framework that sits above your data engines. You write a pipeline once, and Wayang runs each step on the engine that fits. The complete guide.

Who it is for

Teams whose pipelines span Postgres, a data lake, Spark or Flink and an ML stack, and who are tired of glue code and copies.

The problem it solves

Copying data into one engine before any work starts. That copy costs egress, storage and upkeep, and some data is not allowed to move.

What it runs on

Java Streams, Apache Spark, Apache Flink, Apache Giraph, PostgreSQL, SQLite, Apache Kafka and TensorFlow. More engines plug in.

How it decides

A cost-based optimizer prices each step and each move of data between engines, then runs the cheapest plan it finds.

Your data lives in many systems.

A typical AI pipeline pulls rows from Postgres, prepares them in Spark or Flink and trains in TensorFlow or PyTorch. Someone writes the glue and copies the data between every step.

Cost

Copies cost money

Egress to move the data, storage for every copy, and engineers to keep each pipeline alive.

Sovereignty

Some data can't move

Rules keep data in one country, one plant or one closed network. Copy-first pipelines break.

Lock-in

Engines change

Moving from Spark to Flink, or from a warehouse to a lake, means rewriting every pipeline.

How it works

One plan. The right engine for each step.

You describe the pipeline once in Java, Scala, Python or SQL. Wayang turns it into a plan that is not tied to any engine, then places every step.

One plan

Your code becomes a Wayang plan. Change the engines underneath and the plan stays the same.

Cost-based optimizer

It prices every step and every move between engines, then runs the cheapest plan it finds.

Pin when you must

If data cannot leave a database, pin that step there. Wayang plans the rest around it.

Published benchmarks

Measured by the Apache Wayang project.

Three scenarios with public method and data. We quote the project and add no numbers of our own.

Federated SQL · TPC-H Q5

Beat Postgres, close to Spark

Filters ran in Postgres and the big join in Spark, across HDFS, Postgres and S3.

Full results →

ML training · SGD

Over 10x faster on large data

Prep ran in Spark, then one Java process took over once the data had shrunk.

Full results →

WordCount · 1 to 800 GB

Fastest engine at every size

Wayang chose between Spark, Flink and a single Java process on its own each time.

Full results →

Spark, Trino, Wayang

Spark, Trino or Wayang? Often all three.

They solve different problems. Spark computes on a cluster, Wayang plans pipelines across the engines you already run, and Trino answers SQL across many sources.

Compute engine

Apache Spark

Runs SQL, streaming, machine learning and graph jobs on a cluster. Its executors process and hold the data.

Cross-platform optimizer

Apache Wayang

Plans whole pipelines, SQL or not, and runs each step on Spark, Flink, Postgres or Java. The engines do the work.

SQL query engine

Trino

Runs one SQL query across many sources through connectors. Its own workers do the processing, in parallel.

	Apache Spark	Apache Wayang	Trino
What it is	Analytics engine for large-scale data	Optimizer above your engines	Distributed SQL query engine
Workloads	SQL, streaming, ML and graph jobs	Pipelines that mix SQL, ML and more	SQL analytics and reports, not OLTP
Does data move	Yes, copied into the Spark cluster	No, only what the next step needs	Pulled to Trino workers at query time
Optimizer decides	How a job runs inside Spark	Which engine runs each step	Join order and join type
APIs	Java, Scala, Python, SQL	Java, Scala, Python, SQL	ANSI SQL, via clients and drivers
With the others	An engine Wayang can run steps on	Plans the steps, engines run them	The SQL layer for analysts and BI

Keep Spark and Trino. Wayang decides which engine runs each step. Map Wayang onto your stack

Open source, supported

Same engine. Run for you.

Wayang removes the glue code, not the physics. Engine switches cost data movement, and the cost model has to fit your cluster. Where the time goes.

	Apache Wayang	Lascaris add-on
Optimizer	Wayang optimizer and executor	The same optimizer and executor
Releases	Community releases	Version-locked and tested together
Security	You harden it yourself	Hardened and security-maintained
Support	Mailing list and community	Defined SLAs, one contract
Tuning	You tune the cost model	Tuned to your cluster in onboarding
Migration	You port Spark and Flink jobs	Jobs moved over in onboarding
Where it runs	Wherever you deploy it	Your environment, or air-gapped
License	Apache 2.0, free	Add-on to Lascaris

Analytics across sites that cannot share raw data? See Federated Intelligence.

Who builds it

Built by people who built Wayang.

Wayang is public: the code, the benchmarks and the research. Code on GitHub. All our open source projects.

Apache top-level project

Graduated in November 2025, announced by the ASF in December. 18 committers and 9 PMC members as of August 2026.

Peer-reviewed research

Started as Rheem at the Qatar Computing Research Institute. Published at VLDB in 2018, with later papers at ICDE and SIGMOD.

The people

Scalytics was founded by people who built Wayang. Alexander Alten and Dr. Mirko Kämpf sit on the Apache Wayang PMC.

Questions

Common questions.

What is Apache Wayang?

An open source framework and Apache top-level project. You write a pipeline once, and Wayang runs each step on the engine that fits: Postgres, Spark, Flink or a plain Java process.

Is Wayang just a research project?

No. It started as Rheem at the Qatar Computing Research Institute and graduated at Apache in November 2025. What open source lacks is a supported way to run it. The Lascaris add-on gives you that.

Does Wayang replace Spark or Flink?

No. Spark and Flink are engines. Wayang sits above them and decides which one runs each step. You keep the engines you already run and the skills your team has.

Is Wayang part of Lascaris?

It is an add-on. Apache Wayang itself is free under Apache 2.0. The Lascaris add-on brings tested releases, hardening, SLAs and onboarding.

Does my data have to move?

Less of it. Wayang runs filters where the data sits and moves only what the next step needs. Lascaris runs inside your environment, on-premise or air-gapped.

Can you move our Spark or Flink jobs?

Yes. Moving existing Spark and Flink jobs is part of Lascaris onboarding, together with tuning the Wayang cost model to your own cluster and data.

Who maintains Apache Wayang?

The Apache community and its PMC. Scalytics was founded by people who built Wayang, and Alexander Alten and Dr. Mirko Kämpf sit on the Apache Wayang PMC.

Lascaris add-on

Run Wayang in production.

Add Apache Wayang to Lascaris and run it inside your own environment, on-premise or air-gapped.

Add-on

Added to your platform

The add-on runs Wayang in Lascaris, version-locked, hardened and under the same SLA.

Onboarding

Your jobs, moved over

We tune the cost model to your cluster and move your existing Spark and Flink jobs.

Talk to us about Lascaris Plan a Spark/Flink migration