Scraping Libraries

Scrapy is a BSD-licensed fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, fro…

It's compatible with python 3. Say we want to fetch all related post titles in a stackoverflow page: Here's the output: Say we want to scrape live stock prices from yahoo finance: Now we can get the …

The list of tools, programming libraries and web services used in web scraping and data processing. These two links point to same captcha service, it is just a different language versions

Easy to type, click, select, toggle, drag & drop, and more. 💡 SeleniumBase gives you clean error output when a test fails. With raw Selenium, error messages can get very messy. ✅ SeleniumBase use…

This can be useful if you wish to scrape or crawl a website protected with Cloudflare. Cloudflare's anti-bot page currently just checks if the client supports JavaScript, though they may add addition…

Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy Coupons & enroll you to PAID UDEMY COURSES, ABSOLUTE…

Advanced web crawling and text discovery: Parallel processing of online and offline input: Robust and configurable extraction of key elements: Multiple output formats: Optional add-ons: Actively main…

Generate visualizations of GitHub user and repository statistics with GitHub Actions. Visualizations can include data for both private repositories, and for repositories you have contributed to, but …

I myself have not used Grab for many years. I am not sure it is being used by anybody at present time. Nonetheless I decided to refactor the project, just for fun. I have annotated whole code base wi…

Originally named "Web Scraping Open Project", this repository wants so create a common knowledge among web scraping experts, interesting enough for both rookies and experts in the field. Anyone can s…

To install beta releases: To install unstable version from GitHub: Notes: More concurrency: ScrapeNinja handles headless browsers, proxies, timeouts, retries, and helps with data extraction, so you c…

Our goal is to scrape as many sites as we can so we've always looked for these key elements to make a successful large-scale web scraping project. At the moment they are focused on web scraping of E-…

Fansly Downloader is the go-to app for all your bulk media downloading needs. Download photos, videos, audio or any other media from Fansly, this powerful tool has got you covered! Say goodbye to the…

Read more about DDS in the ReadTheDocs documentation: There is a mailing list on Google Groups, feel free to ask questions or make suggestions: Infos about new releases and updates on Twitter:

If you found this code useful in your programming journey, please consider sponsoring the creation of my content by clicking on the "Sponsor" button in this repo. The funding goes to the creation of …

Recently, Twitter has banned almost every Twitter scraper. This repository presents an alternative tool to scrape Twitter based on 3 functions: Note : You must have Chrome installed on your system. T…

Sorry for this. Faster updates The following dependencies are required and must be installed separately. upgrade: (If a YouTube video has any closed captions, they will be downloaded together with th…

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a …

Currently we support the following databases and data warehouses For connecting and loading all your tooling data into a data warehouse, we are integrating with Airbyte connectors. For everything rel…

A high performance, easy to use, multithreaded command line tool which downloads images from the given webpage. You can also download using pip: You can also use it in your Python scripts. (Deprecate…

Scripts/Programs in python that scrape the web, create a bot or use an API to solve a problem. We now have a section for miscellaneous scripts/programs as well.

If you want to use the python source directly, please install following requirements into your environment: Any kind of positive contribution is welcome! Please help it grow by contributing to the pr…

The primary goal of Scrapple is to abstract the process of designing web content extractors. The focus is laid on what to extract, rather than how to do it. The user-specified configuration file cont…

Python tool to automatically archive social media posts, videos, and images from a Google Sheets, the console, and more. Uses different archivers depending on the platform, and can save content to lo…

Tests are (so far) only run on static html files. One of which is a linkedin profile, the other is just used to test some utility functions. Because of Linkedin's anti-scraping measures, you must mak…