One framework to develop, deploy and operate data workflows with Python and SQL.

vmware Last update: Apr 24, 2024

One framework to🧑‍💻 Develop ▶️ Deploy and 📊 Operate
data workflows with Python and SQL

🎯 Write shorter, more readable code.

🔄 Ready-to-use data ETL/ELT patterns.

🧩 Lego-like extensibility.

🚀 Single click deployment.

🛠 Operate and monitor. ️

Introduction to the VDK SDK

Framework to simplify data ingestion and data processing.
Write any code using Python or SQL.
A toolset enabling you to run data jobs.

Get started with VDK SDK:

➡ Install Quickstart VDK. Only requirement is Python 3.7+.

pip install quickstart-vdk
vdk --help

➡ Develop your First Data Job if you are impatient to start quickly.

VDK.SDK.2.mp4

Data Ingestion

Extract data from various sources (HTTP APIs, Databases, CSV, etc.).
Ensure data fidelity with minimal transformations.
Load data to your preferred destination (database, cloud storage).

Ingestion examples:

➡ Ingesting data from REST API into Database
➡ Ingesting data from DB into Database
➡ Ingesting local CSV file into Database
➡ Incremental ingestion using Job Properties

VDK.Ingestion.2.mp4

Data Transformation

SQL and Python parameterized transformations.
Extensible templates for data modeling.
Creates a dataset or table as a product.

Get started with transforming data:

➡ Data Modeling: Treating Data as a Product
➡ Processing data using SQL and local database
➡ Processing data using Kimball warehousing templates

Transform.VDK.2.mp4

Data Job Deployment (build, deploy, release)

VDK Control Service provides REST API for users to create, deploy, manage, and execute data jobs in a Kubernetes runtime environment.

Scheduling, packaging, dependencies management, deployment.
Execution management and monitoring.
Source code versioning and tracking. Fast rollback.
Manage state and credentials using Properties and Secrets.

Get started with deploying jobs in control service:

➡ Install Local Control Service with vdk server --install
➡ Scheduling a Data Job for automatic execution
➡ Using VDK DAGs to orchestrate Data Jobs

VDK.CS.2.mp4

Operations and Monitoring

Use Operations UI to monitor, troubleshoot data workloads in production.
Notifications for errors during Data Job deployment or execution.
Route errors to correct people by classifying them into User or Platform errors.

Get started with operating and monitoring data jobs:

➡ Versatile Data Kit UI - Installation and Getting Started
➡ VDK Operations User Interface - Versatile Data Kit

VDK.UI.2.mp4

Lego like extensibility

Modular: use only what you need. Extensible: build what you miss.
Easy to install any plugins as python packages using pip.
Used in enhancing data processing, ingestion, job execution, command-line lifecycle

Get started with using some VDK plugins:

➡ Browse available plugins
➡ Interesting plugins to check out:
Track Lineage of your jobs using vdk-lineage
Import/Ingest or Export CSV files using vdk-csv
➡ Write your own plugin

VDK.plugins.2.mp4

Support and Contributing

For Support, you can join our Slack channel, create an issue or pull request on GitHub to submit suggestions or changes.
If you are interested in contributing as a developer, visit the contributing page.

Contacts

Message us on Slack:
☝️ Join the CNCF Slack workspace.
✌️ Join the #versatile-data-kit channel.
Join the next Community Meeting
Follow us on Twitter.
Subscribe to the Versatile Data Kit YouTube Channel.
Join our development mailing list, used by developers and maintainers of VDK.

Code of Conduct

Everyone involved in working on the project's source code, or engaging in any issue trackers, Slack channels, and mailing lists is expected to be familiar with and follow the Code of Conduct.

One framework to develop, deploy and operate data workflows with Python and SQL.

Introduction to the VDK SDK

Data Ingestion

Data Transformation

Data Job Deployment (build, deploy, release)

Operations and Monitoring

Lego like extensibility

Support and Contributing

Contacts

Code of Conduct

🤖 Template for telegram bot using postgres, pgbouncer, redis, docker, amplitude, prometheus, grafana, CI

Data science and Machine Learning GUI programs/ desktop apps with PySimpleGUI package

ML-capsule is a Project for beginners and experienced data science Enthusiasts who don't have a mentor or guidance and wish to learn Machine learning. Using our repo they can learn ML, DL, and many related technologies with different real-world projects and become Interview ready.

The easiest way to use Machine Learning. Mix and match underlying ML libraries and data set sources. Generate new datasets or modify existing ones with ease.

Text analytics for LLM apps. PostHog for prompts. Extract evaluations, intents and events from text messages. phospho leverages LLM (OpenAI, MistralAI, Ollama, etc.)

a flask profiler which watches endpoint calls and tries to make some analysis.

Analytics services for Django projects

A flexible, easy to use, automation framework allowing users to integrate their capabilities and devices to cut through the repetitive, tedious tasks slowing them down. #nsacyber

Apache Superset is a Data Visualization and Data Exploration Platform

Beginner-friendly collection of Python notebooks for various use cases of machine learning, deep learning, and analytics. For each notebook there is a separate tutorial on the relataly.com blog.