🧙 Mage

Mage is an open-source tool for buildingand runningdata pipelines that transform your data.

Here is a sample data pipeline defined across 3 files:

# data_loaders/load_data_from_file.py@data_loaderdef load_csv_from_file():    return pd.read_csv('default_repo/titanic.csv')

# transformers/select_columns.py@transformerdef select_columns_from_df(df, *args):    return df[['Age', 'Fare', 'Survived']]

# data_exporters/export_to_file.py@data_exporterdef export_titanic_data_to_disk(df) -> None:    df.to_csv('default_repo/titanic_transformed.csv')

What the data pipeline looks like in the UI:

New? We recommend reading about blocks andlearning from a hands-on tutorial.

Table of contents

1. Quick start
2. Demo
3. Tutorials
4. Features
5. Core design principles
6. Core abstractions
7. Documentation

🏃‍♀️ Quick start

Install Mage using Docker or pip:

Using Docker

Create a new project and launch tool (change demo_project to any other name if you want):

docker run -it -p 6789:6789 -v $(pwd):/home/src \  mageai/mageai mage start demo_project

_{Follow the guide if you want to use PySpark kernel in your notebook.}

Using pip or conda

1. Install Mage

pip install mage-ai

or

conda install -c conda-forge mage-ai

_{For additional packages (e.g. spark, postgres, etc), please see Installing extra packages.}

_{If you run into errors, please see Install errors.}

2. Create new project and launch tool (change demo_project to any other name if you want):

mage start demo_project

Open tool in browser

Open http://localhost:6789 in your browser and build a pipeline.

🎮 Demo

Live demo

Try a hosted version of the tool here: http://demo.mage.ai.

WARNING

The live demo is public, please don’t save anything sensitive.

Demo video (2 min)

_{Click the image to play video}

👩‍🏫 Tutorials

• Train model on Titanic dataset
• Load data from API, transform it, and export it to PostgreSQL
• Integrate Mage into an existing Airflow project

🔮 Features

• Data pipeline management
• Notebook for building data pipelines

_{Read more here.}

🏔️ Core design principles

Every user experience and technical design decision adheres to these principles.

1. Easy developer experience
2. Engineering best practices built-in
3. Data is a first-class citizen
4. Scaling made simple

_{Read more here.}

🛸 Core abstractions

These are the fundamental concepts that Mage uses to operate.

• Project
• Pipeline
• Block
• Data product
• Trigger
• Run

_{Read more here.}

📚 Documentation

Read more here.

🙋‍♀️ Contributing

Check out the 🎁 contributing guide to get startedby setting up your development environment andexploring the code base.

🤔 Frequently Asked Questions (FAQs)

Check out our FAQ page to find answers to some of our most asked questions.

🧙 Community

We love the community of Magers (/ˈmājər/);a group of mages who help each other realize their full potential!

To live chat with the Mage team and community,please join the free Mage Slackchannel.

For real-time news and fun memes, check out the Mage Twitter.

To report bugs or add your awesome code for others to enjoy,visit GitHub.

🪪 License

See the LICENSE file for licensing information.