Data-engineering Libraries

Learn the foundations of machine learning through intuitive explanations, clean code and visualizations. Learn how to combine machine learning with software engineering to build production-grade appl…

With Dagster, you declare—as Python functions—the data assets that you want to build. Dagster then helps you run your functions at the right time and keep your assets up-to-date. Here is an example o…

Feast allows ML platform teams to: The list below contains the functionality that contributors are planning to develop for Feast. We welcome contribution to all items in the roadmap! Feast is a commu…

Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON…

A simple YAML API to get started quickly, a powerful Python API for total flexibility. Automatically cache your pipeline’s previous results and only re-compute tasks that have changed since your last…

  Taipy is designed for data scientists and machine learning engineers to build full-stack apps.         To install Taipy stable release run:   Below is our filter function. This is a typical Python …

🚀 Join our thriving community of likeminded developers and build the future together! dlt supports Python 3.8+. Load chess game data from chess.com API and save it in DuckDB: The dlt project is quick…

A comprehensive list of +180 YouTube Channels (in no particular order) for Data Science, Data Engineering, Machine Learning, Deep Learning, Artificial Intelligence, Computer Science, programming, sof…

Pipeline Consists of various modules: EMR - I used a 3 node cluster with below Instance Types: Finally, pyspark uses python2 as default setup on EMR. To change to python3, setup environment variable…

MLRun is an open MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the del…

Hamilton is a novel paradigm for specifying a flow of delayed execution in python. It works on python objects of any type and dataflows of any complexity. Core to the design of Hamilton is a clear ma…

The general purpose micro-framework for creating dataflows from python functions! Specifically, Hamilton defines a novel paradigm, that allows you to specify a flow of (delayed) execution, that forms…

An Airflow environment using version 2.2 or later. Running Python 3.8 or later in your Airflow environment. Currently, we support the following remote storages: Tasks don’t pass information to each o…

A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data v…

SAYN is a modern data processing and modelling framework. Users define tasks (incl. Python, automated SQL transformations and more) and their relationships, SAYN takes care of the rest. It is designe…

The goal of this repository template is to make it easy for you to get started with Prefect. Ideally, you should be able to:

Beneath is a serverless real-time data platform. Our goal is to create one end-to-end platform for data workers that combines data storage, processing, and visualization with data quality management …

Magniv is the easiest way to build and deploy data applications, pipelines, or cron-like jobs. A Python decorator based orchestration library at the core, Magniv allows data scientists and developers…

Example repository showing how to leverage Prefect, dbt and Snowflake together to build a data platform. This project includes automation to make it easy to use this repositor as a template to get st…