Scraper Libraries

Read more about DDS in the ReadTheDocs documentation: There is a mailing list on Google Groups, feel free to ask questions or make suggestions: Infos about new releases and updates on Twitter:

instagram-scraper is a command-line application written in Python that scrapes and downloads an instagram user's photos and videos. Use responsibly. To install instagram-scraper: To update instagram-…

This is a Webcrawler based on Scrapy and Selenium frameworks This spider crawls thru the directferries.com website in order to generate a json file with all tickets available for one of these directi…

A high performance, easy to use, multithreaded command line tool which downloads images from the given webpage. You can also download using pip: You can also use it in your Python scripts. (Deprecate…

The primary goal of Scrapple is to abstract the process of designing web content extractors. The focus is laid on what to extract, rather than how to do it. The user-specified configuration file cont…

Python Script for 'searching' and 'downloading' hundreds of Google images to the local hard disk! This program lets you download tons of images from Google. Please do not download or use any image th…

Simple script for downloading Youtube comments without using the Youtube API. The output is in line delimited JSON. Or directly from the GitHub repository: For example: or using the Youtube ID: You c…

PubMed can send you regular update on new articles matching your specified search criteria. PyperGrabber will automatically download thoe papers, saving you much time tracking on downloading those ma…

Version 1.1 is available to install via Pip: Run the search on the rightmove website and copy the URL of the first results page. Create an instance of the class with the URL as the init argument. Fin…

Supported archives include: mailingListScraper is composed of several spiders.Each spider targets a specific email archive, which can host one or several mailing lists. You can launch a spider runnin…

news-please supports three main use cases, which are explained in more detail in the following. It's super easy, we promise! news-please runs on Python 3.5+. You can access the core functionality of …

N.Y.A.W.C is a Python library that enables you to test your payload against all requests of a certain domain. It crawls all requests (e.g. GET, POST or PUT) in the specified scope and keeps track of …

A python command-line (CLI) bot for automating research and promotion on popular social media platforms (reddit, twitter, facebook, [TODO: instagram]). With a single command, scrape social media site…

The scraper is not actually functional because Zipru is not a real site. The code, however, is otherwise complete and can easily be adapted to work on other sites.

Scrapy tutorial to build a Craigslist crawler that scrapes engineering jobs in New York. Check this blog post for the full Scrapy tutorial: In Terminal or CMD, navigate to the main Scrapy project fol…

The repository is set up in a modular fashion so you can easily substitute in your own email address and report contents. I have personally found this to be a fun use for custom web scrapers that gra…

A basic scrapper made in python with BeautifulSoup and Tor support to - These instructions will get you a copy of the project up and running on your local machine for development and testing purposes…

Extract emails and linkedins profiles from a given website

A simple python package for saving images from Bing and Google without using API keys. This package utilizes web browsers to help scrape images found on web searches. This should only be used for edu…

Tests are (so far) only run on static html files. One of which is a linkedin profile, the other is just used to test some utility functions. Because of Linkedin's anti-scraping measures, you must mak…

Learn how to leverage Python's amazing tools to scrape data from other websites. The end goal of this course is to scrape blogs to analyze trending keywords and phrases. We'll be using Python 3.6, Re…

Install requirements from Terminal: Go grab a cup of coffee while waiting... oh wait, it's already done!

You can use the API in your command line, or develop your own application using the API as imported package. Both options are explained below. The query API allows you to search and filter all 18 mil…

LinkedinScraper is an another information gathering tool written in python. You can scrape employees of companies on Linkedin.com and then create these employee names, titles and emails. Install requ…

You can send me an email by clicking on the badge. I look forward to hearing from you! This is a comprehensive Reddit scraping tool that integrates multiple features: This is a shout-out section for …