Workflow Management Tools Overview
Overview based on:
- Ecosystem - Documentation, Active Development, Open License, Ease of Use
- Features - Topics and Queues, Reliable Messaging, REST Management API, Streams processing
| Workflow Management Tool | Ok | Ecosystem | Features |
|---|---|---|---|
| luigi | ~ | ✔️️ | No Good UI |
| UnifiedViews | ✔️️ | ~ | ~ |
| LinkedPipes | ✔️️ | ~ | Similar to UnifiedViews in features |
| Airflow | ~ | ✔️️ | Somewhat more acceptable UI |
| Azkaban 3 | ✔️️ | ||
| ProActive Workflows & Scheduling | ✔️️ | ||
| Apache Nifi | ✔️️ | ✔️️ | |
| Wings | ✔️️ | ~ | Semantic Workflows |
| Karma | ~ | ~ | |
| LDIF – Linked Data Integration Framework | ~ | ~ |
References
- Integrating Apache NiFi and Apache Kafka http://bryanbende.com/development/2016/09/15/apache-nifi-and-apache-kafka
- Luigi vs Airflow vs Pinball http://bytepawn.com/luigi-airflow-pinball.html
- curated list of notable ETL (extract, transform, load) frameworks, libraries and software. https://github.com/pawl/awesome-etl#etl-tools-gui
- Computational Data Analysis Workflow Systems https://github.com/common-workflow-language/common-workflow-language/wiki/Existing-Workflow-systems
- Difference between Apache NiFi and StreamSets http://stackoverflow.com/questions/36899612/difference-between-apache-nifi-and-streamsets
- Apache NiFi: Thinking Differently About DataFlow https://blogs.apache.org/nifi/entry/basic_dataflow_design
- NiFi vs Falcon vs Oozie https://www.linkedin.com/pulse/nifi-vs-falcon-oozie-birender-saini
- http://funnifi.blogspot.fi/
- Plugins and Examples https://github.com/jfrazee/awesome-nifi
- https://blogs.apache.org/nifi/entry/basic_dataflow_design
- https://community.hortonworks.com/articles/72727/extracting-nifi-provenance-data-using-sitetositepr.html