Pipeline Libraries

Jina provides a smooth Pythonic experience for serving ML models transitioning from local deployment to advanced orchestration frameworks like Docker-Compose, Kubernetes, or Jina AI Cloud. Jina makes…

With Dagster, you declare—as Python functions—the data assets that you want to build. Dagster then helps you run your functions at the right time and keep your assets up-to-date. Here is an example o…

A simple YAML API to get started quickly, a powerful Python API for total flexibility. Automatically cache your pipeline’s previous results and only re-compute tasks that have changed since your last…

  Taipy is designed for data scientists and machine learning engineers to build full-stack apps.         To install Taipy stable release run:   Below is our filter function. This is a typical Python …

These three functionalities enable a variety of use cases for data scientists, machine learning engineers, and data engineers: From here, you can quickly log a dataset: And there you have it, you now…

A procedural Blender pipeline for photorealistic rendering. The simplest way to install blenderproc is via pip: Alternatively, if you need to make changes to blenderproc or you want to make use of th…

This package contains a lightweight data transformation framework with a focus on transparency and complexity reduction. It has a number of baked-in assumptions/ principles: Data integration pipeline…

Automated machine learning for production and analytics auto_ml is designed for production. Here's an example that includes serializing and loading the trained model, then getting predictions on sing…

MLBox has been developed and used by many active community members. Your help is very valuable to make it better for everyone.

Supercharge your SaaS development with our comprehensive starter kit, designed to accelerate your project and save you valuable time and resources. Our battle-tested boilerplate eliminates the need f…

Soda Core is a free, open-source, command-line tool that enables you to use the Soda Checks Language to turn user-defined input into aggregated SQL queries. When it runs a scan on a dataset, Soda Cor…

Pipeline Consists of various modules: EMR - I used a 3 node cluster with below Instance Types: Finally, pyspark uses python2 as default setup on EMR. To change to python3, setup environment variable…

Hamilton is a novel paradigm for specifying a flow of delayed execution in python. It works on python objects of any type and dataflows of any complexity. Core to the design of Hamilton is a clear ma…

The general purpose micro-framework for creating dataflows from python functions! Specifically, Hamilton defines a novel paradigm, that allows you to specify a flow of (delayed) execution, that forms…

If that sounds like a lot, don't worry. After you cover this course, you will understand everything we said before. Most importantly, you will know WHY we used all these tools and how they work toget…

marimo is a reactive notebook for Python. It allows you to rapidly experiment with data and models, code with confidence in your notebook's correctness, and productionize notebooks as pipelines or in…

Covalent is a Python library for AI/ML engineers, developers, and researchers. It provides a straightforward approach to running compute jobs, like LLMs, generative AI, and scientific research, on v…

or: Alternatively, you could clone and run setup.py file: Initialize a group of classifiers as base estimators Initialize, fit, predict, and evaluate with Stacking See a sample output of classifier_s…

Pipeless is inspired by modern serverless technologies. You provide some functions and Pipeless takes care of executing them for new video frames and everything involved. With Pipeless you create se…

pypyr is a free & open-source task-runner that lets you define and run sequential steps in a pipeline. Like a turbo-charged shell script, but less finicky. Less annoying than a makefile. pypyr ru…

This project provides a working, open source based, AWS Lambda handler skeleton Python code including DEPLOYMENT code with CDK and a pipeline. This project can serve as a template for new Serverless …

Foremast is a Spinnaker pipeline and infrastructure configuration and templating tool. Just create a couple JSON configuration files and then manually creating Spinnaker pipelines becomes a thing of…

This book teaches you how to take machine learning models from your personal laptop to large distributed clusters. You’ll explore key concepts and patterns behind successful distributed machine learn…

PDFs are hard to process and it's hard to extract information. So the results of this tool may not satisfy you. There will be more work to improve this software but altogether, it's unlikely that it …

As we all know the Machine Learning space has a lot of tools and libraries for creating pipelines to train, test & deploy models, and dealing with these many different APIs can be cumbersome. Our…

Turn any function into a component of your backend system. Inject any dependency with full control over its setup and lifecycle Installing Pulumi unlocks: