Language-model Libraries

21 Components & Libraries

Sortby

Language-model Libraries

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow 🤗 Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio. Thes…

:mag: Haystack is an open source NLP framework to interact with your data using Transformer models and LLMs (GPT-4, ChatGPT and alike). Haystack offers production-ready tools to quickly build complex question answering, semantic search, text generation applications, and more.

This command installs everything needed for basic Pipelines that use an in-memory DocumentStore. If you want to install only the dependencies needed for model inference on your local hardware (not re…

💡 All-in-one open-source embeddings database for semantic search, LLM orchestration and language model workflows

txtai is an all-in-one embeddings database for semantic search, LLM orchestration and language model workflows. Embeddings databases are a union of vector indexes (sparse and dense), graph networks a…

Open-source offline translation library written in Python

Argos Translate also manages automatically pivoting through intermediate languages to translate between languages that don't have a direct translation between them installed. For example, if you have…

Machine Learning Engineering Online Book

An open collection of methodologies to help with successful training of large language models and multi-modal models. This is a technical material suitable for LLM/VLM training engineers and operator…

🪢 Open source LLM engineering platform. Observability, metrics, evals, prompt management, testing -- 🍊YC W23 🤖 integrate via Typescript, Python, OpenAI, Langchain, LlamaIndex, Litellm

Managed deployment by the Langfuse team, generous free-tier (hobby plan), no credit card required. You may want to use another integration to get started quickly or implement a use case that we do no…

Library to scrape and clean web pages to create massive datasets.

A straightforward library that allows you to crawl, clean up, and deduplicate webpages to create massive monolingual datasets. Using this library, you should be able to create datasets larger than th…

Swirl is open-source software that uses AI to simultaneously search multiple content and data sources, finds the best results using a reader LLM, then prompts Generative AI, enabling you to get answers from your own data.

Swirl is open source software that simultaneously searches multiple content sources and returns AI ranked results. Prompt your choice of Generative AI using the top N results to get answers incorpora…

ChatArena (or Chat Arena) is a Multi-Agent Language Game Environments for LLMs. The goal is to develop communication and collaboration capabilities of AIs.

Multi-Agent Language Game Environments for LLMs ChatArena is a library that provides multi-agent language game environments and facilitates research about autonomous LLM agents and their social inter…

ChatArena (or Chat Arena) is a Multi-Agent Language Game Environments for LLMs. The goal is to develop communication and collaboration capabilities of AIs.

A chat app that transcribes audio in real-time, streams back a response from a language model, and synthesizes this response as natural-sounding speech.

A complete chat app that transcribes audio in real-time, streams back a response from a language model, and synthesizes this response as natural-sounding speech. This repo is meant to serve as a star…

Ship RAG based LLM web apps in seconds.

for built-in data readers (github, pdf, docx, ipynb, epub, mbox, websites..), install with: huggingface - ollama example: microsoft azure - openai example: google - vertexai example: aws bedrock - cl…

⛓️ Serving LangChain LLM apps and agents automagically with FastApi. LLMops

LangCorn is an API server that enables you to serve LangChain models and pipelines with ease, leveraging the power of FastAPI for a robust and efficient experience. To get started with LangCorn, simp…

Get-Things-Done-with-Prompt-Engineering-and-LangChain

LangChain & Prompt Engineering tutorials on Large Language Models (LLMs) such as ChatGPT with custom data. Jupyter notebooks on loading and indexing data, creating prompt templates, CSV agents, and using retrieval QA chains to query the custom data. Projects for using a private LLM (Llama 2) for chat with PDF files, tweets sentiment analysis.

Build real-world AI apps with ChatGPT/GPT-4 and LangChain in Python

PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks such as the extraction of n-grams and frequency lists, and to build simple language model. There are also more complex data types and algorithms. Moreover, there are parsers for file formats common in NLP (e.g. FoLiA/Giza/Moses/ARPA/Timbl/CQL). There are also clients to interface with various NLP specific servers. PyNLPl most notably features a very extensive library for working with FoLiA XML (Format for Linguistic Annotation).

PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks s…

NucliaDB, The vector database optimized for documents and video search

NucliaDB is a robust database that allows storing and searching on unstructured data. It is an out of the box hybrid search database, utilizing vector, full text and graph indexes. NucliaDB is writte…

API for the GPT-J language model 🦜. Including a FastAPI backend and a streamlit frontend

An API to interact with the GPT-J language model and variants! You can use and test the model in two different ways: These are the endpoints of the public API and require no authentication. Click on …

🏭 PDF text extraction pipeline: self-hosted, local-first, Docker-based

PDFs are hard to process and it's hard to extract information. So the results of this tool may not satisfy you. There will be more work to improve this software but altogether, it's unlikely that it …

Train a language model to answer Slack messages as you.

DoppelBot is a Slack app that scrapes a target user's messages in Slack and fine-tunes a large language model (OpenLLaMa) to learn how to respond like them. This repo contains everything you need to …