Raw Libraries

Scrapy is a BSD-licensed fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, fro…

It's compatible with python 3. Say we want to fetch all related post titles in a stackoverflow page: Here's the output: Say we want to scrape live stock prices from yahoo finance: Now we can get the …

Python GraphQL library based on dataclasses The quick start method provides a server and CLI to get going quickly. Install with: To run the debug server run the following command: This will open Grap…

Toapi give you the ability to make every web site provides APIs. you will get the result like: Write code and test code and pull request.

Toapi give you the ability to make every web site provides APIs. you will get the result like: Write code and test code and pull request.

Advanced web crawling and text discovery: Parallel processing of online and offline input: Robust and configurable extraction of key elements: Multiple output formats: Optional add-ons: Actively main…

Simply run: or download the project via git clone and run the following: Some methods do require authentication: Two Factor Authentication is also supported through cli interface, simply use 'True' f…

I myself have not used Grab for many years. I am not sure it is being used by anybody at present time. Nonetheless I decided to refactor the project, just for fun. I have annotated whole code base wi…

news-please supports three main use cases, which are explained in more detail in the following. It's super easy, we promise! news-please runs on Python 3.5+. You can access the core functionality of …

Ruia is still under developing, feel free to open issues and pull requests:

If you have Pip installed on your system, you can use it to install the latest Dirhunt stable version: This software must not be used on third-party servers without permission. Dirhunt has been creat…

Poetry version must be greater or equal than 1.1.8. Otherwise it won't be able to install SQLAlchemy. Templator in action You can install it directly from pypi with pip. If you want to install it fro…

Google, Naver multiprocess image crawler (High Quality & Speed & Customizable) Install Chrome pip install -r requirements.txt Write search keywords in keywords.txt Files will be downloaded to…

diskover is an open source file system indexer that uses Elasticsearch to index and manage data across heterogeneous storage systems. Using diskover, you are able to more effectively search and organ…

You might be doing basic stuff, like copying and pasting text on spread sheets, you might be running large scale automated platforms with sophisticated algorithms, or somewhere in between. In any cas…

Getting filings of various companies at once is really a pain, but SEC-Edgar does that for you. You can download all of a company’s periodic reports, filings and forms from the EDGAR database with a …

StringZilla is the Godzilla of string libraries, splitting, sorting, and shuffling large textual datasets faster than you can say "Tokyo Tower" 😅 StringZilla is built on a very simple heuristic: If t…

Sorry for this. Faster updates The following dependencies are required and must be installed separately. upgrade: (If a YouTube video has any closed captions, they will be downloaded together with th…

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a …

This package is a mini framework of web crawlers. With modularization design, it is easy to use and extend. It supports media data like images and videos very well, and can also be applied to texts a…

You can send me an email by clicking on the badge. I look forward to hearing from you! This is a comprehensive Reddit scraping tool that integrates multiple features: This is a shout-out section for …

Emoji explained: Translation is also performed by ChatGPT, with a single extra step in the prompt. Currently supported languages:

Just the code of my OSINT bot searching for sensitive data leaks on paste sites. Search terms: Search terms can be customized. You can learn more about it in the configuration section. For pastebin.c…

The primary goal of Scrapple is to abstract the process of designing web content extractors. The focus is laid on what to extract, rather than how to do it. The user-specified configuration file cont…

Copy this code snippet: Example: Both webpack dev server and Streamlit should run at the same time.