Scrap Libraries

Scrapy is a BSD-licensed fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, fro…

Read more about DDS in the ReadTheDocs documentation: There is a mailing list on Google Groups, feel free to ask questions or make suggestions: Infos about new releases and updates on Twitter:

This can be useful if you wish to scrape or crawl a website protected with Cloudflare. Cloudflare's anti-bot page currently just checks if the client supports JavaScript, though they may add addition…

instagram-scraper is a command-line application written in Python that scrapes and downloads an instagram user's photos and videos. Use responsibly. To install instagram-scraper: To update instagram-…

I myself have not used Grab for many years. I am not sure it is being used by anybody at present time. Nonetheless I decided to refactor the project, just for fun. I have annotated whole code base wi…

This is a Webcrawler based on Scrapy and Selenium frameworks This spider crawls thru the directferries.com website in order to generate a json file with all tickets available for one of these directi…

Easy to type, click, select, toggle, drag & drop, and more. 💡 SeleniumBase gives you clean error output when a test fails. With raw Selenium, error messages can get very messy. ✅ SeleniumBase use…

A high performance, easy to use, multithreaded command line tool which downloads images from the given webpage. You can also download using pip: You can also use it in your Python scripts. (Deprecate…

The primary goal of Scrapple is to abstract the process of designing web content extractors. The focus is laid on what to extract, rather than how to do it. The user-specified configuration file cont…

Python Script for 'searching' and 'downloading' hundreds of Google images to the local hard disk! This program lets you download tons of images from Google. Please do not download or use any image th…

Features: Pomp makes no attempt to accomodate: Pomp is written and maintained by Evgeniy Tatarkin and is licensed under the BSD license.

The list of tools, programming libraries and web services used in web scraping and data processing. These two links point to same captcha service, it is just a different language versions

Simple script for downloading Youtube comments without using the Youtube API. The output is in line delimited JSON. Or directly from the GitHub repository: For example: or using the Youtube ID: You c…

Web scraping and automation using python

This package is a mini framework of web crawlers. With modularization design, it is easy to use and extend. It supports media data like images and videos very well, and can also be applied to texts a…

PubMed can send you regular update on new articles matching your specified search criteria. PyperGrabber will automatically download thoe papers, saving you much time tracking on downloading those ma…

Version 1.1 is available to install via Pip: Run the search on the rightmove website and copy the URL of the first results page. Create an instance of the class with the URL as the init argument. Fin…

Supported archives include: mailingListScraper is composed of several spiders.Each spider targets a specific email archive, which can host one or several mailing lists. You can launch a spider runnin…

news-please supports three main use cases, which are explained in more detail in the following. It's super easy, we promise! news-please runs on Python 3.5+. You can access the core functionality of …

N.Y.A.W.C is a Python library that enables you to test your payload against all requests of a certain domain. It crawls all requests (e.g. GET, POST or PUT) in the specified scope and keeps track of …

A python command-line (CLI) bot for automating research and promotion on popular social media platforms (reddit, twitter, facebook, [TODO: instagram]). With a single command, scrape social media site…

The scraper is not actually functional because Zipru is not a real site. The code, however, is otherwise complete and can easily be adapted to work on other sites.

I have not included every code sample you've found in the book, but I have included a majority of the finished scripts. Although these are included, I encourage you to write out each code sample on y…

If you found this code useful in your programming journey, please consider sponsoring the creation of my content by clicking on the "Sponsor" button in this repo. The funding goes to the creation of …