Crawler Libraries

Scrapy is a BSD-licensed fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, fro…

A free utility to extract email address by crawling a given url upto a given depth or number of urls to crawl provided by the user. Web email extractor by url crawl using command line interface. Emai…

I myself have not used Grab for many years. I am not sure it is being used by anybody at present time. Nonetheless I decided to refactor the project, just for fun. I have annotated whole code base wi…

Python script to download all images/webms of a 4chan thread You can parse a file instead of a thread url. In this file you can put as many links as you want, you just have to make sure that there's …

This is a Webcrawler based on Scrapy and Selenium frameworks This spider crawls thru the directferries.com website in order to generate a json file with all tickets available for one of these directi…

Getting filings of various companies at once is really a pain, but SEC-Edgar does that for you. You can download all of a company’s periodic reports, filings and forms from the EDGAR database with a …

Comic Crawler 是用來扒圖的一支 Python Script。擁有簡易的下載管理員、圖書館功能、 與方便的擴充能力。 安裝時記得要選「Add python.exe to path」，才能使用 pip 指令。 Windows 10 (1709) 以上的版本，可以直接在 cmd 底下輸入以下指令安裝︰ 在 cmd 底下輸入以下指令︰ 更新時︰ 最後在 cmd 底下輸入以下指令執行 Com…

Emoji explained: Translation is also performed by ChatGPT, with a single extra step in the prompt. Currently supported languages:

The primary goal of Scrapple is to abstract the process of designing web content extractors. The focus is laid on what to extract, rather than how to do it. The user-specified configuration file cont…

Features: Pomp makes no attempt to accomodate: Pomp is written and maintained by Evgeniy Tatarkin and is licensed under the BSD license.

This package is a mini framework of web crawlers. With modularization design, it is easy to use and extend. It supports media data like images and videos very well, and can also be applied to texts a…

PubMed can send you regular update on new articles matching your specified search criteria. PyperGrabber will automatically download thoe papers, saving you much time tracking on downloading those ma…

news-please supports three main use cases, which are explained in more detail in the following. It's super easy, we promise! news-please runs on Python 3.5+. You can access the core functionality of …

N.Y.A.W.C is a Python library that enables you to test your payload against all requests of a certain domain. It crawls all requests (e.g. GET, POST or PUT) in the specified scope and keeps track of …

diskover is an open source file system indexer that uses Elasticsearch to index and manage data across heterogeneous storage systems. Using diskover, you are able to more effectively search and organ…

Scrapy tutorial to build a Craigslist crawler that scrapes engineering jobs in New York. Check this blog post for the full Scrapy tutorial: In Terminal or CMD, navigate to the main Scrapy project fol…

You might be doing basic stuff, like copying and pasting text on spread sheets, you might be running large scale automated platforms with sophisticated algorithms, or somewhere in between. In any cas…

A basic scrapper made in python with BeautifulSoup and Tor support to - These instructions will get you a copy of the project up and running on your local machine for development and testing purposes…

A simple python package for saving images from Bing and Google without using API keys. This package utilizes web browsers to help scrape images found on web searches. This should only be used for edu…

Toapi give you the ability to make every web site provides APIs. you will get the result like: Write code and test code and pull request.

Toapi give you the ability to make every web site provides APIs. you will get the result like: Write code and test code and pull request.

If you have Pip installed on your system, you can use it to install the latest Dirhunt stable version: This software must not be used on third-party servers without permission. Dirhunt has been creat…

Just the code of my OSINT bot searching for sensitive data leaks on paste sites. Search terms: Search terms can be customized. You can learn more about it in the configuration section. For pastebin.c…

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a …

Ruia is still under developing, feel free to open issues and pull requests: