Crawler Libraries

Scrapy is a BSD-licensed fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, fro…

It's compatible with python 3. Say we want to fetch all related post titles in a stackoverflow page: Here's the output: Say we want to scrape live stock prices from yahoo finance: Now we can get the …

Toapi give you the ability to make every web site provides APIs. you will get the result like: Write code and test code and pull request.

Toapi give you the ability to make every web site provides APIs. you will get the result like: Write code and test code and pull request.

Trafilatura consistently outperforms other open-source libraries in text extraction benchmarks, showcasing its efficiency and accuracy in extracting web content. The extractor tries to strike a balan…

Simply run: or download the project via git clone and run the following: Some methods do require authentication: Two Factor Authentication is also supported through cli interface, simply use 'True' f…

I myself have not used Grab for many years. I am not sure it is being used by anybody at present time. Nonetheless I decided to refactor the project, just for fun. I have annotated whole code base wi…

news-please supports three main use cases, which are explained in more detail in the following. It's super easy, we promise! news-please runs on Python 3.5+. You can access the core functionality of …

Ruia is still under developing, feel free to open issues and pull requests:

If you have Pip installed on your system, you can use it to install the latest Dirhunt stable version: This software must not be used on third-party servers without permission. Dirhunt has been creat…

Google, Naver multiprocess image crawler (High Quality & Speed & Customizable) Install Chrome pip install -r requirements.txt Write search keywords in keywords.txt Files will be downloaded to…

diskover is an open source file system indexer that uses Elasticsearch to index and manage data across heterogeneous storage systems. Using diskover, you are able to more effectively search and organ…

You might be doing basic stuff, like copying and pasting text on spread sheets, you might be running large scale automated platforms with sophisticated algorithms, or somewhere in between. In any cas…

Getting filings of various companies at once is really a pain, but SEC-Edgar does that for you. You can download all of a company’s periodic reports, filings and forms from the EDGAR database with a …

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a …

This package is a mini framework of web crawlers. With modularization design, it is easy to use and extend. It supports media data like images and videos very well, and can also be applied to texts a…

Emoji explained: Translation is also performed by ChatGPT, with a single extra step in the prompt. Currently supported languages:

Just the code of my OSINT bot searching for sensitive data leaks on paste sites. Search terms: Search terms can be customized. You can learn more about it in the configuration section. For pastebin.c…

The primary goal of Scrapple is to abstract the process of designing web content extractors. The focus is laid on what to extract, rather than how to do it. The user-specified configuration file cont…

PyWebCopy is a free tool for copying full or partial websites locally onto your hard-disk for offline viewing. PyWebCopy will scan the specified website and download its content onto your hard-disk. …

Init the project Please feel free to add any contribution. If you've been working on a script that could be integrated in this app. Please feel free to do it. Don't hesitate to open issues to ask que…

🚨 Dude is currently in Pre-Alpha. Please expect breaking changes. To install, simply run the following from terminal. The simplest web scraper will look like this: Here is the summary of features sup…

🔹 Main menu 🔹 Download posts 🔹 Like posts 🔹 Bulk unfollow 🔹 Bulk commenting 🔹 Bulk comment deletion

Comic Crawler 是用來扒圖的一支 Python Script。擁有簡易的下載管理員、圖書館功能、 與方便的擴充能力。 安裝時記得要選「Add python.exe to path」，才能使用 pip 指令。 Windows 10 (1709) 以上的版本，可以直接在 cmd 底下輸入以下指令安裝︰ 在 cmd 底下輸入以下指令︰ 更新時︰ 最後在 cmd 底下輸入以下指令執行 Com…

Python script to download all images/webms of a 4chan thread You can parse a file instead of a thread url. In this file you can put as many links as you want, you just have to make sure that there's …