Data-engineering Libraries

25 Components & Libraries

Sortby

Data-engineering Libraries

Your single tool to express data, ML, and LLM pipelines with simple python functions. Runs anywhere that python runs, E.G. spark, airflow, jupyter, fastapi, etc. Incrementally adoptable. Use Hamilton to build testable, reusable, and self-documenting dataflows with lineage and metadata out of the box.

Hamilton is a novel paradigm for specifying a flow of delayed execution in python. It works on python objects of any type and dataflows of any complexity. Core to the design of Hamilton is a clear ma…

Artificial-Intelligence-Important-Documents-Collections

AI technology is significant because it allows software to do human functions—understanding, reasoning, planning, communication, and perception—increasingly effectively, efficiently, and affordably.

AI technology is significants because it allows softwares to do human functions—understanding, reasoning, planning, communication, and perception—increasingly effectively, efficiently, and affordably…

Example repository showing how to build a data platform with Prefect, dbt and Snowflake

Example repository showing how to leverage Prefect, dbt and Snowflake together to build a data platform. This project includes automation to make it easy to use this repositor as a template to get st…

Project demonstrating how to automate Prefect 2.0 deployments to AWS ECS Fargate

The goal of this repository template is to make it easy for you to get started with Prefect. Ideally, you should be able to:

Magniv Core - A Python-decorator based job orchestration platform. Avoid responsibility handoffs by abstracting infra and DevOps.

Magniv is the easiest way to build and deploy data applications, pipelines, or cron-like jobs. A Python decorator based orchestration library at the core, Magniv allows data scientists and developers…

Data Science Roadmap from A to Z

Pandas

Turns Data and AI algorithms into production-ready web applications in no time.

Taipy is designed for data scientists and machine learning engineers to build full-stack apps. To install Taipy stable release run: Below is our filter function. This is a typical Python …

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

🚀 Join our thriving community of likeminded developers and build the future together! dlt supports Python 3.8+. Load chess game data from chess.com API and save it in DuckDB: The dlt project is quick…

One framework to develop, deploy and operate data workflows with Python and SQL.

A comprehensive list of 180+ YouTube Channels for Data Science, Data Engineering, Machine Learning, Deep learning, Computer Science, programming, software engineering, etc.

A comprehensive list of +180 YouTube Channels (in no particular order) for Data Science, Data Engineering, Machine Learning, Deep Learning, Artificial Intelligence, Computer Science, programming, sof…

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

An Airflow environment using version 2.2 or later. Running Python 3.8 or later in your Airflow environment. Currently, we support the following remote storages: Tasks don’t pass information to each o…

A scalable general purpose micro-framework for defining dataflows. You can use it to build dataframes, numpy matrices, python objects, ML models, etc. Embed Hamilton anywhere python runs, e.g. spark, airflow, jupyter, fastapi, python scripts, etc.

The general purpose micro-framework for creating dataflows from python functions! Specifically, Hamilton defines a novel paradigm, that allows you to specify a flow of (delayed) execution, that forms…

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

SAYN is a modern data processing and modelling framework. Users define tasks (incl. Python, automated SQL transformations and more) and their relationships, SAYN takes care of the rest. It is designe…

A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.

A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data v…

Beneath is a serverless real-time data platform ⚡️

Beneath is a serverless real-time data platform. Our goal is to create one end-to-end platform for data workers that combines data storage, processing, and visualization with data quality management …

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Pipeline Consists of various modules: EMR - I used a 3 node cluster with below Instance Types: Finally, pyspark uses python2 as default setup on EMR. To change to python3, setup environment variable…

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

A simple YAML API to get started quickly, a powerful Python API for total flexibility. Automatically cache your pipeline’s previous results and only re-compute tasks that have changed since your last…

This repo contains commands that data engineers use in day to day work.

Machine Learning automation and tracking

MLRun is an open MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the del…

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON…

Feature Store for Machine Learning

Feast allows ML platform teams to: The list below contains the functionality that contributors are planning to develop for Feast. We welcome contribution to all items in the roadmap! Feast is a commu…

Learn how to responsibly develop, deploy and maintain production machine learning applications.

Learn the foundations of machine learning through intuitive explanations, clean code and visualizations. Learn how to combine machine learning with software engineering to build production-grade appl…

An orchestration platform for the development, production, and observation of data assets.

With Dagster, you declare—as Python functions—the data assets that you want to build. Dagster then helps you run your functions at the right time and keep your assets up-to-date. Here is an example o…

Apache Superset is a Data Visualization and Data Exploration Platform

A modern, enterprise-ready business intelligence web application. Superset is a modern data exploration and data visualization platform. Superset can replace or augment proprietary business intellige…