Apache Spark

Awesome Spark ¶

A curated list of awesome Apache Spark packages and resources.

Apache Spark is an open-source cluster-computing framework. Originally developed at the University of California, Berkeley's AMPLab, the Spark codebase was later donated to the Apache Software Foundation, which has maintained it since. Spark provides an interface for programming entire clusters with implicit data parallelism and fault-tolerance (Wikipedia 2017).

Users of Apache Spark may choose between different the Python, R, Scala and Java programming languages to interface with the Apache Spark APIs.

Packages¶

Language Bindings¶

Notebooks and IDEs¶

almond - A scala kernel for Jupyter.
Apache Zeppelin - Web-based notebook that enables interactive data analytics with plugable backends, integrated plotting, and extensive Spark support out-of-the-box.
Polynote - Polynote: an IDE-inspired polyglot notebook. It supports mixing multiple languages in one notebook, and sharing data between them seamlessly. It encourages reproducible notebooks with its immutable data model. Originating from Netflix.

General Purpose Libraries¶

SQL Data Sources¶

SparkSQL has serveral built-in Data Sources for files. These include csv, json, parquet, orc, and avro. It also supports JDBC databases as well as Apache Hive. Additional data sources can be added by including the packages listed below, or writing your own.

Storage¶

lakeFS - Integration with the lakeFS atomic versioned storage layer.

Bioinformatics¶

GIS¶

Graph Processing¶

Machine Learning Extension¶

Apache SystemML - Declarative machine learning framework on top of Spark.
Mahout Spark Bindings [status unknown] - linear algebra DSL and optimizer with R-like syntax.
KeystoneML - Type safe machine learning pipelines with RDDs.
MLflow - Machine learning orchestration platform.

Middleware¶

Monitoring¶

Utilities¶

Natural Language Processing¶

Streaming¶

Apache Bahir - Collection of the streaming connectors excluded from Spark 2.0 (Akka, MQTT, Twitter. ZeroMQ).

Interfaces¶

Apache Beam - Unified data processing engine supporting both batch and streaming applications. Apache Spark is one of the supported execution environments.

Data quality¶

Testing¶

Web Archives¶

Workflow Management¶

Resources¶

Books¶

Learning Spark, 2^nd Edition - Introduction to Spark API with Spark 3.0 covered. Good source of knowledge about basic concepts.
Advanced Analytics with Spark - Useful collection of Spark processing patterns. Accompanying GitHub repository: sryza/aas.
Mastering Apache Spark - Interesting compilation of notes by Jacek Laskowski. Focused on different aspects of Spark internals.

Papers¶

Large-Scale Intelligent Microservices - Microsoft paper that presents an Apache Spark-based micro-service orchestration framework that extends database operations to include web service primitives.
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing - Paper introducing a core distributed memory abstraction.
Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark - Structured Streaming is a new high-level streaming API, it is a declarative API based on automatically incrementalizing a static relational query.

MOOCS¶

Data Science and Engineering with Apache Spark (edX XSeries) - Series of five courses (Introduction to Apache Spark, Distributed Machine Learning with Apache Spark, Big Data Analysis with Apache Spark, Advanced Apache Spark for Data Science and Data Engineering, Advanced Distributed Machine Learning with Apache Spark) covering different aspects of software engineering and data science. Python oriented.
Big Data Analysis with Scala and Spark (Coursera) - Scala oriented introductory course. Part of Functional Programming in Scala Specialization.

Workshops¶

AMP Camp - Periodical training event organized by the UC Berkeley AMPLab. A source of useful exercise and recorded workshops covering different tools from the Berkeley Data Analytics Stack.

Projects Using Spark¶

Oryx 2 - Lambda architecture platform built on Apache Spark and Apache Kafka with specialization for real-time large scale machine learning.
Photon ML - A machine learning library supporting classical Generalized Mixed Model and Generalized Additive Mixed Effect Model.
PredictionIO - Machine Learning server for developers and data scientists to build and deploy predictive applications in a fraction of the time.
Crossdata - Data integration platform with extended DataSource API and multi-user environment.

Docker Images¶

apache/spark - Apache Spark Official Docker images.
jupyter/docker-stacks/pyspark-notebook - PySpark with Jupyter Notebook and Mesos client.
sequenceiq/docker-spark - Yarn images from SequenceIQ.
datamechanics/spark - An easy to setup Docker image for Apache Spark from Data Mechanics.

Miscellaneous¶

Spark with Scala Gitter channel - "A place to discuss and ask questions about using Scala for Spark programming" started by @deanwampler.
Apache Spark User List and Apache Spark Developers List - Mailing lists dedicated to usage questions and development topics respectively.

References¶

Wikipedia. 2017. “Apache Spark — Wikipedia, the Free Encyclopedia.” https://en.wikipedia.org/w/index.php?title=Apache_Spark&oldid=781182753.

License¶

This work (Awesome Spark, by https://github.com/awesome-spark/awesome-spark), identified by Maciej Szymkiewicz, is free of known copyright restrictions.

Apache Spark, Spark, Apache, and the Spark logo are trademarks of The Apache Software Foundation. This compilation is not endorsed by The Apache Software Foundation.

Inspired by sindresorhus/awesome.