Airflow Libraries

It was last updated to a fully running version in late October, 2021. You should refer to the Jupyter Notebooks in this repository rather than the book's source code, which is badly outdated and will…

Sheets only go so far, with a StarThinker running, the ad operations team can operate on larger data sets, scale solutions across clients fast, and even automate campaign controls. All using a drag a…

Pipeline Consists of various modules: EMR - I used a 3 node cluster with below Instance Types: Finally, pyspark uses python2 as default setup on EMR. To change to python3, setup environment variable…

A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data v…

Cloud agnostic tech stack for starting an MLOps platform (Level 1) "We'll build a pipeline - after we deploy the model." Clone the project Change directories into the repo Run database migrations and…

This boilerplate has more tools than was discussed in the article. In particular, it has the following things that were not discussed in the article: Create a virtualenv for this project. Feel free t…

Soda Core is a free, open-source, command-line tool that enables you to use the Soda Checks Language to turn user-defined input into aggregated SQL queries. When it runs a scan on a dataset, Soda Cor…

An Airflow environment using version 2.2 or later. Running Python 3.8 or later in your Airflow environment. Currently, we support the following remote storages: Tasks don’t pass information to each o…

Magniv is the easiest way to build and deploy data applications, pipelines, or cron-like jobs. A Python decorator based orchestration library at the core, Magniv allows data scientists and developers…

If that sounds like a lot, don't worry. After you cover this course, you will understand everything we said before. Most importantly, you will know WHY we used all these tools and how they work toget…

Creating Workflows in the GUI is as simple as dragging and dropping Pieces to the canvas, and connecting them. The user can schedule the Workflow to run periodically, at a specific date/time, or trig…