Spark Libraries

37 Components & Libraries

Sortby

Spark Libraries

Apache Spark - A unified analytics engine for large-scale data processing

Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for da…

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

Redash is designed to enable anyone, regardless of the level of technical sophistication, to harness the power of data big and small. SQL users leverage Redash to explore, query, visualize, and share…

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

GitHub GitHub issues -- file bug reports / track issues here Stack Overflow -- ask all code/software questions here Cross Validated (Stack Exchange) -- ask algorithm/theory questions here h2ostream G…

macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults.

Run all: To update later on, just run that command again. Alternatively, to update while avoiding the confirmation prompt: To sync dev-setup to your local home directory without Git, run the followin…

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs for Scala, Java, Rust, Ruby, and Python.

There are two types of APIs provided by the Delta Lake project. Delta Lake guarantees backward compatibility for all Delta Lake tables (i.e., newer versions of Delta Lake will always be able to read …

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

Linkis builds a layer of computation middleware between upper applications and underlying engines. By using standard interfaces such as REST/WS/JDBC provided by Linkis, the upper applications can eas…

Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray

Then, define the BigDL model using Keras-style API:

Expressive analytics in Python at any scale.

Ibis is a Python library that provides a lightweight, universal interface for data wrangling. It helps Python users explore and transform data of any size, stored anywhere. Ibis has three primary com…

LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.

LakeSoul is a cloud-native Lakehouse framework that supports scalable metadata management, ACID transactions, efficient and flexible upsert operation, schema evolution, and unified streaming & ba…

A minimal benchmark for scalability, speed and accuracy of commonly used open source implementations (R packages, Python scikit-learn, H2O, xgboost, Spark MLlib etc.) of the top machine learning algorithms for binary classification (random forests, gradient boosted trees, deep neural networks etc.).

The algorithms studied are in various commonly used open source implementations like In summary, we are focusing on which algos/implementations can be used to train relatively accurate binary classif…

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Contents For example to trigger installation of GPU version of tensorflow and opencv, use the following pip command: Petastorm supports extensible data codecs. These enable a user to use one of the s…

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

The reference book for these and other Spark related topics is: The following notebooks can be examined individually, although there is a more or less linear 'story' when followed in sequence. By usi…

Django email backends and webhooks for Amazon SES, Brevo (Sendinblue), MailerSend, Mailgun, Mailjet, Postmark, Postal, Resend, SendGrid, SparkPost and more

Anymail currently supports these ESPs: Anymail includes: Here's how to send a message. This example uses Mailgun, but you can substitute Mailjet or Postmark or SendGrid or SparkPost or any other supp…

Machine Learning Platform and Recommendation Engine built on Kubernetes

Seldon Server is a machine learning platform that helps your data science team deploy models into production. Seldon supports models built with TensorFlow, Keras, Vowpal Wabbit, XGBoost, Gensim and a…

:zap: Data quality management for SQL, Spark, and Pandas-accessible data

Soda Core is a free, open-source, command-line tool that enables you to use the Soda Checks Language to turn user-defined input into aggregated SQL queries. When it runs a scan on a dataset, Soda Cor…

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Pipeline Consists of various modules: EMR - I used a 3 node cluster with below Instance Types: Finally, pyspark uses python2 as default setup on EMR. To change to python3, setup environment variable…

MapReduce, Spark, Java, and Scala for Data Algorithms Book

The book's codebase can also be downloaded from the git repository at: Thank you!

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times

Currently we support the following databases and data warehouses For connecting and loading all your tooling data into a data warehouse, we are integrating with Airbyte connectors. For everything rel…

An open source framework for building data analytic applications.

CDAP is an integrated, open source application development platform for the Hadoop ecosystem that provides developers with data and application abstractions to simplify and accelerate application dev…

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

See Also: repos which contains hundreds more scripts and programs for Cloud, Big Data, SQL, NoSQL, Web and Linux. Hari Sekhon Cloud & Big Data Contractor, United Kingdom List all programs: Run an…

Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition

It was last updated to a fully running version in late October, 2021. You should refer to the Jupyter Notebooks in this repository rather than the book's source code, which is badly outdated and will…

Version 1 of Technical Best Practices of Azure Databricks based on real world Customer and Technical SME inputs

Authors: Dhruv Kumar, Senior Solutions Architect, Databricks Premal Shah, Azure Databricks PM, Microsoft Bhanu Prakash, Azure Databricks PM, Microsoft Written by: Priya Aswani, WW Data Engineering &a…

Zeek Analysis Tools (ZAT): Processing and analysis of Zeek network data with Pandas, scikit-learn, Kafka and Spark

The ZAT Python package supports the processing and analysis of Zeek data with Pandas, scikit-learn, Kafka, and Spark Zeek already has a flexible, powerful scripting language why should I use ZAT?

Fundamentals of Spark with Python (using PySpark), code examples

But, PySpark+Jupyter combo needs a little bit more love :-) Resilient Distributed Datasets (RDD) is a fundamental data structure of Spark. It is an immutable distributed collection of objects. Each d…

Open Source Contributor Index

OSCI, an open source project, aiming to track and measure open source activity on GitHub by commercial organizations. It allows organizations, communities, analysts and individuals involved in Open S…

:snowflake: :whale: Awesome tools and libs for AI, Deep Learning, Machine Learning, Computer Vision, Data Science, Data Analytics and Cognitive Computing that are baked in the oven to be Native on Kubernetes and Docker with Python, R, Scala, Java, C#, Go, Julia, C++ etc

The entire computing industry has adopted Kubernetes as the way forward to running distibuted computing loads from on-premise servers to large multi-cloud deployments. For Machine Learning and Data S…

A concise resource repository for machine learning

A concise resource repository for machine learning. Here, It'll remain concise but yet to comprehensive for machine learning resources and related stuff. It'll be updated continually with times. Some…

Spark 2.0 Python Machine Learning examples

This repository is part of a series on Apache Spark examples, aimed at demonstrating the implementation of Machine Learning solutions in different programming languages supported by Spark. Java is t…

Related tags