Apache DolphinScheduler, the distributed Big Data visual workflow scheduler system, is the latest project to gain Top-Level Project status at the Apache Software Foundation (ASF). This means that the project’s community and products have been well-governed under the Apache Software Foundation’s meritocratic process and principles.
The scheduler was first created in December 2017 at Analysys to help solve complex task dependencies and triggers in the data pipeline. The project entered the Apache Incubator in August 2019. The project now has over 4,000 users in China, with many Internet companies and banks making up the user base.
“Apache DolphinScheduler is designed for cloud-native,” said Lidong Dai, vice president of Apache DolphinScheduler. “We are proud to have built a reliable and cloud friendly data workflow system while using next generation architecture and smart UI design.”
The project supports multi-cloud/data center workflow management as well as Kubernetes, Docker deployment and custom task types. It has high reliability with a decentralized multi-master and multi-worker architecture.
All process definition operations are visualized and the project supports scenarios such as streaming, pause, recover operations, multi-tenants, and additional task types such as spark, hive, mr, shell, python, flink, sub_process, and more.
“Congratulations to open source project DolphinScheduler for graduating from the Apache incubator and becoming ASF’s top project,” said Chen Liang, vice president of Apache CarbonData. “DolphinScheduler has been developing the community in accordance with the Apache Way and has attracted many open-source developers to join. With the joint efforts of community members, the project has become more and more mature.”
