Data-engineer Libraries

26 Components & Libraries

Sortby

Data-engineer Libraries

:shell: Python-powered, cross-platform, Unix-gazing shell.

Install xonsh from pip: The xonsh shell is developed by a community of volunteers. There are a few ways to help out: We welcome new contributors!

Apache Superset is a Data Visualization and Data Exploration Platform

A modern, enterprise-ready business intelligence web application. Superset is a modern data exploration and data visualization platform. Superset can replace or augment proprietary business intellige…

An orchestration platform for the development, production, and observation of data assets.

With Dagster, you declare—as Python functions—the data assets that you want to build. Dagster then helps you run your functions at the right time and keep your assets up-to-date. Here is an example o…

Learn how to responsibly develop, deploy and maintain production machine learning applications.

Learn the foundations of machine learning through intuitive explanations, clean code and visualizations. Learn how to combine machine learning with software engineering to build production-grade appl…

Feature Store for Machine Learning

Feast allows ML platform teams to: The list below contains the functionality that contributors are planning to develop for Feast. We welcome contribution to all items in the roadmap! Feast is a commu…

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON…

Machine Learning automation and tracking

MLRun is an open MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the del…

This repo contains commands that data engineers use in day to day work.

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

A simple YAML API to get started quickly, a powerful Python API for total flexibility. Automatically cache your pipeline’s previous results and only re-compute tasks that have changed since your last…

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Pipeline Consists of various modules: EMR - I used a 3 node cluster with below Instance Types: Finally, pyspark uses python2 as default setup on EMR. To change to python3, setup environment variable…

Beneath is a serverless real-time data platform ⚡️

Beneath is a serverless real-time data platform. Our goal is to create one end-to-end platform for data workers that combines data storage, processing, and visualization with data quality management …

A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.

A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data v…

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

SAYN is a modern data processing and modelling framework. Users define tasks (incl. Python, automated SQL transformations and more) and their relationships, SAYN takes care of the rest. It is designe…

A scalable general purpose micro-framework for defining dataflows. You can use it to build dataframes, numpy matrices, python objects, ML models, etc. Embed Hamilton anywhere python runs, e.g. spark, airflow, jupyter, fastapi, python scripts, etc.

The general purpose micro-framework for creating dataflows from python functions! Specifically, Hamilton defines a novel paradigm, that allows you to specify a flow of (delayed) execution, that forms…

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

An Airflow environment using version 2.2 or later. Running Python 3.8 or later in your Airflow environment. Currently, we support the following remote storages: Tasks don’t pass information to each o…

A comprehensive list of 180+ YouTube Channels for Data Science, Data Engineering, Machine Learning, Deep learning, Computer Science, programming, software engineering, etc.

A comprehensive list of +180 YouTube Channels (in no particular order) for Data Science, Data Engineering, Machine Learning, Deep Learning, Artificial Intelligence, Computer Science, programming, sof…

One framework to develop, deploy and operate data workflows with Python and SQL.

A collection of online resources to help you on your Tech journey.

This repository will contain a list of FREE resources, projects and hands-on exercises for you to attempt and up skill yourself so that you can find a role in your respective field. This project will…

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

🚀 Join our thriving community of likeminded developers and build the future together! dlt supports Python 3.8+. Load chess game data from chess.com API and save it in DuckDB: The dlt project is quick…

Turns Data and AI algorithms into production-ready web applications in no time.

Taipy is designed for data scientists and machine learning engineers to build full-stack apps. To install Taipy stable release run: Below is our filter function. This is a typical Python …

Data Science Roadmap from A to Z

Pandas

Magniv Core - A Python-decorator based job orchestration platform. Avoid responsibility handoffs by abstracting infra and DevOps.

Magniv is the easiest way to build and deploy data applications, pipelines, or cron-like jobs. A Python decorator based orchestration library at the core, Magniv allows data scientists and developers…

Project demonstrating how to automate Prefect 2.0 deployments to AWS ECS Fargate

The goal of this repository template is to make it easy for you to get started with Prefect. Ideally, you should be able to:

Example repository showing how to build a data platform with Prefect, dbt and Snowflake

Example repository showing how to leverage Prefect, dbt and Snowflake together to build a data platform. This project includes automation to make it easy to use this repositor as a template to get st…

Artificial-Intelligence-Important-Documents-Collections

AI technology is significant because it allows software to do human functions—understanding, reasoning, planning, communication, and perception—increasingly effectively, efficiently, and affordably.

AI technology is significants because it allows softwares to do human functions—understanding, reasoning, planning, communication, and perception—increasingly effectively, efficiently, and affordably…

Your single tool to express data, ML, and LLM pipelines with simple python functions. Runs anywhere that python runs, E.G. spark, airflow, jupyter, fastapi, etc. Incrementally adoptable. Use Hamilton to build testable, reusable, and self-documenting dataflows with lineage and metadata out of the box.

Hamilton is a novel paradigm for specifying a flow of delayed execution in python. It works on python objects of any type and dataflows of any complexity. Core to the design of Hamilton is a clear ma…

Related tags