The Best 86 Web scraping Libraries

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining…

Pattern is a web mining module for Python. It has tools for: It is well documented, thoroughly tested with 350+ unit tests and comes bundled with 50+ examples. The source code is licensed under BSD. …

It's compatible with python 3. Say we want to fetch all related post titles in a stackoverflow page: Here's the output: Say we want to scrape live stock prices from yahoo finance: Now we can get the …

The list of tools, programming libraries and web services used in web scraping and data processing. These two links point to same captcha service, it is just a different language versions

The web changes, libraries update, and make mistakes and typos more frequently than I'd like to admit! If you think you've spotted an error, please feel free to make a pull request against this repos…

Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy Coupons & enroll you to PAID UDEMY COURSES, ABSOLUTE…

I myself have not used Grab for many years. I am not sure it is being used by anybody at present time. Nonetheless I decided to refactor the project, just for fun. I have annotated whole code base wi…

A straightforward library that allows you to crawl, clean up, and deduplicate webpages to create massive monolingual datasets. Using this library, you should be able to create datasets larger than th…

For video tutorials see this Youtube playlist:

Ruia is still under developing, feel free to open issues and pull requests:

Originally named "Web Scraping Open Project", this repository wants so create a common knowledge among web scraping experts, interesting enough for both rookies and experts in the field. Anyone can s…

Our goal is to scrape as many sites as we can so we've always looked for these key elements to make a successful large-scale web scraping project. At the moment they are focused on web scraping of E-…

Some examples of the tasks: To copy the scripts quickly via the command-line; by default, a ./search-script-scrape directory will be created: To run a script: To reiterate: each of these scripts are …

A Python library to utilize AWS API Gateway's large IP pool as a proxy to generate pseudo-infinite IPs for web scraping and brute forcing. This library will allow the user to bypass IP-based rate-lim…

DaProfiler is an OSINT tool capable of tracing the digital identity of a target via social networks, emails, public information such as directories, business listings, etc. DaProfiler is also able to…

Scripts/Programs in python that scrape the web, create a bot or use an API to solve a problem. We now have a section for miscellaneous scripts/programs as well.

dryscrape is a lightweight web scraping library for Python. It uses a headless Webkit instance to evaluate Javascript on the visited pages. This enables painless scraping of plain web pages as well a…

The primary goal of Scrapple is to abstract the process of designing web content extractors. The focus is laid on what to extract, rather than how to do it. The user-specified configuration file cont…

Dorks Eye is a script I made in python 3. With this tool, you can easily find Google Dorks. Dork Eye collects potentially vulnerable web pages and applications on the Internet or other awesome info t…

The scraper is not actually functional because Zipru is not a real site. The code, however, is otherwise complete and can easily be adapted to work on other sites.

List of major web + mobile browser user agent strings. +1 Bonus script to scrape :)

🚨 Dude is currently in Pre-Alpha. Please expect breaking changes. To install, simply run the following from terminal. The simplest web scraper will look like this: Here is the summary of features sup…

Cyborg is an asyncio Python 3 web scraping framework that helps you write programs to extract information from websites by reading and inspecting their HTML. Scraping websites for data can be fairly …

Essa série de vídeos contém o seguinte conteúdo:

Just want to see the code? Here it is:

This project was created just for educational proposes. The code shows how to do web scraping dynamic content pages generated from Javascript using Python and Selenium. These instructions will get yo…

Version 1.1 is available to install via Pip: Run the search on the rightmove website and copy the URL of the first results page. Create an instance of the class with the URL as the init argument. Fin…

The web is full of data. Transistor is a web scraping framework for collecting, storing, and using targeted data from structured web pages. Successfully scraping is now a complex affair. Most website…