Open Source Developer Community

Join our developer community, where our engineers are the original creators of open source projects and active contributors to popular open source projects. Influence the roadmap, add features, and bring your ideas to life in a global open source project.

Read more about our open source commitment, projects, research and knowledge in our open source blogs.

Open Source projects we invented

Federated Data and AI - Apache Wayang

In contrast to traditional data processing systems that provide one dedicated execution engine, Apache Wayang transparently and seamlessly integrate multiple execution engines and use them to perform a single task as a DAG, which enables AI across any connected data platform, be it JDBC, Spark, Flink or Kafka.

Energy Prediction Model - LSTEnergy

LSTEnergy is an LSTM model that enables users to better understand energy consumption, predict energy flows more accurately, and consequently, save energy and reduce CO2 emissions. LSTEnergy performs with a high probability after approximately 20 epochs, depending on the dataset used.

Open Source Projects we support

Apache Spark

Apache Spark is an open-source distributed processing system used for big data workloads. It utilizes in-memory caching and optimized query execution for fast analytic queries against data of any size.

Apache Flink

Apache Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. It can process unbounded and bounded data streams.

Apache Hadoop

Apache Hadoop is an open-source software framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models, uniting multiple data tools.

Apache Impala

Apache Impala is an open-source, native analytic database for Apache Hadoop. It provides low latency and high concurrency for BI/analytic queries on Hadoop, which is not delivered by batch frameworks such as Apache Hive.

TensorFlow

TensorFlow is a platform for machine learning, TTF is made for federated TensorFlow. It supports distributed training, immediate model iteration and easy debugging with Keras, and much more.

PostgreSQL

PostgreSQL is a powerful, open-source object-relational database system with over 35 years of active development that has earned it a strong reputation for reliability, feature robustness, and performance.

Open Source: Apache Wayang

Apache Wayang (incubating) is an API for big data cross-platform processing. It provides an abstraction over other platforms like Apache Spark and Apache Flink as well as a default built-in stream-based “platform”. The goal is to provide a consistent developer experience when writing code regardless of whether a light-weight or highly-scalable platform may eventually be required. Execution of the application is specified in a logical plan which is again platform agnostic. Wayang will transform the logical plan into a set of physical operators to be executed by specific underlying processing platforms.

Groovy and Data Science - JVM Advent (javaadvent.com)
“Wayang is a Java library typically used in Big Data applications. Incubator-wayang has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License, and it has low support. You can download it from GitHub.
In contrast to traditional data processing systems that provide one dedicated execution engine, Apache Wayang (incubating) is a cross-platform data processing system: Users can specify any data processing application using one of Wayang's APIs and then Wayang will choose the data processing platform(s), e.g., Postgres or Apache Spark, that best fits the application.”

Apache Wayang is the first cross platform system (openweaver.com)
“Execution of the application is specified in a logical plan which is again platform-agnostic. Wayang will transform the logical plan into a set of physical operators to be executed by specific underlying processing platforms.Wayang selects which platform(s) will run our application. It has numerous capabilities whereby cost functions and load estimators can be used to influence and optimize how the application is run. For our simple example, it is enough to know that even though we specified Java or Spark as options, Wayang knows that for our small data set, the Java streams option is the way to go.

Using Groovy with Apache Wayang and Apache Spark
Don't just dream of better data pipelines, make it happen.