Python
  • Blog
  • Contact
    • Overview
    Issues 0

    This topic explains how to implement web scraping and python web development. Web scraping topics such as scrapy, beautiful soup, and others will be covered. A case study based on a Malaysian website.

    drshahizan drshahizan Last update: Feb 02, 2024

    Stars Badge Forks Badge Pull Requests Badge Issues Badge GitHub contributors Visitors

    Don't forget to hit the ⭐ if you like this repo.

    About Us

    The information on this Github is part of the materials for the subject High Performance Data Processing (SECP3133). This folder contains general big data information as well as big data case studies using Malaysian datasets. This case study was created by a Bachelor of Computer Science (Data Engineering), Universiti Teknologi Malaysia student.

    📚 Course: High Performance Data Processing

    Contents:

    Web Scraping

    Tutorial

    Selenium

    Beautiful Soup

    Scrapy

    Requests

    Lxml

    🌟 Case Study: Web Scraping

    Team Library Website GitHub
    Group 10 Beautiful soup StudyMalaysia.com Open in GitHub
    High Five Beautiful soup EduSpiral Consultant Services Open in GitHub
    QwQ Beautiful soup States and federal territories of Malaysia Open in GitHub
    SDS Scrapy Book Depository Open in GitHub
    BigMac Scrapy CompAsia.com Open in GitHub
    SIX Scrapy bukukita.com Open in GitHub
    AdMiPeQa Selenium Lazada Open in GitHub
    SamVerse Selenium Malaysia General Election (GE-15) Open in GitHub
    Group 9 Selenium Lazada Shopee Open in GitHub
    No Name Requests Puma: sneakers Open in GitHub
    Quad Lxml Jobstreet.com Open in GitHub

    Contribution 🛠️

    Please create an Issue for any improvements, suggestions or errors in the content.

    You can also contact me using Linkedin for any other queries or feedback.

    Visitors

    Tags:

    Owner

    drshahizan

    drshahizan
    GitHub Repository https://github.com/drshahizan
    Dimorf is a ransomware using 256-bit AES with a self-destructing, randomly generated key for Linux OS´s

    Dimorf is a ransomware using 256-bit AES with a self-destructing, randomly generated key for Linux OS´s Previously the code extracted the home directory of a common user by the HOME environment varia…

    Ort0x36
    50 Feb 24, 2024
    Improper Authorization Vulnerability in Confluence Data Center and Server + bonus 🔥

    Improper Authorization Vulnerability in Confluence Data Center and Server. Atlassian has alerted administrators about a critical vulnerability in Confluence. Exploiting this issue can lead to data lo…

    ForceFledgling
    50 Feb 23, 2024
    Solid Python toolkit for those in the security industry. Some by me, most by smarter people.

    If you think a project should be included here I really want to know, but keep in mind that this isn't supposed to be a dictionary. I want to keep the best, covering as many bases as possible while m…

    MythicStack
    50 Apr 02, 2024
    [180+ scripts] There are a few genuine gems in there. And a lot of spaghetti code. Most of these scripts were for solving CTF's. If you googles something for a CTF and landed here look at the scripts they're all fairly malleable. Sorry for the shitty naming conventions (not really). If you are a recruiter stop. I wont be able to rewrite half this shit in a formal interview ¯\_(ツ)_/¯

    [115+ scripts] There are a few genuine gems in there.

    OlivierLaflamme
    51 Apr 11, 2024
    An automation script written in Python to convert a Raspberry Pi into a WiFi hacking device
    teja156
    51 Mar 31, 2024
    Identify connection of sessions for social engineering attacks.

    Identify connection of sessions for social engineering attacks. ========== It is the first framework to keep track of victims through social support live attacks. This tool was created by the inspira…

    jofpin
    52 Feb 29, 2024
    import os def menu(): print(""" .########..#######...#######..##........######...######. ....##....##.....##.##.....##.##.......##....##.##....## ....##....##.....##.##.....##.##.......##.......##...... ....##....##.....##.##.....##.##........######...######. ....##....##.....##.##.....##.##.............##.......## ....##....##.....##.##.....##.##.......##....##.##....## ....##.....#######...#######..########..######...######. ======================================== Created By AnonHacker Channel: youtube.com/kalinuxx Facebook: facebook.com/kalinuxtutorials Ver: 2.0 ---- ONLY FOR TERMUX! ---- ========================================== 00. Turn your Android into Hacking Machine. ------------------------------------------ 1. Install Nmap 2. Install Hydra 3. Install SQLMap 4. Install Metasploit 5. Install ngrok 6. Install Kali Nethunter 7. Install angryFuzzer 8. Install Red_Hawk 9. Install Weeman 10. Install IPGeoLocation 11. Install Cupp 12. Instagram Bruteforcer (instahack) 13. Twitter Bruteforcer (TwitterSniper) 14. Install Ubuntu 15. Install Fedora 16. Install viSQL 17. Install Hash-Buster 18. Install D-TECT 19. Install routersploit ------------------------------------------ 99. Exit ========================================== """) loop = True while loop: menu() what = input("#: ") if what == "00": print("================================") print("This will install: nmap, hydra, sqlmap, metasploit, ngrok, angryFuzzer, red_hawk, weeman, IPGeoLocation, cupp, instahack, TwitterSniper, Hash-Buster, D-TECT, routersploit and viSQL with one click.") print("----------------") hm = input("[?] Do you want to continue? (y/n): ") print("================================") if hm == "y": print("========================================================") print("[+] Please put down you android and go to the toilet...") print("Because this will take a long time.") print("========================================================") os.system("pkg update") os.system("pkg install -y git") os.system("pkg install -y python") os.system("pkg install -y python2") os.system("pkg install -y wget") os.sysetm("pkg install -y nmap") os.system("plg install -y hydra ") os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install python2") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/sqlmapproject/sqlmap.git") os.system("cd /data/data/com.termux/files/home") os.system("pkg install wget") os.system("cd /data/data/com.termux/files/home && wget https://Auxilus.github.io/metasploit.sh") os.system("cd /data/data/com.termux/files/home && bash metasploit.sh") os.system("cd /data/data/com.termux/files/home && cd metasploit-framework") os.system("cd /data/data/com.termux/files/home && gem install bundle --pre") os.system("cd /data/data/com.termux/files/home && bundle config build.nokogiri --use-system-libraries") os.system("cd /data/data/com.termux/files/home && bundle install") os.system("cd /data/data/com.termux/files/home") os.system("pkg update -y") os.system("pkg install -y git") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/themastersunil/ngrok.git") os.system("cd /data/data/com.termux/files/home") os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python2") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/ihebski/angryFuzzer.git") os.system("cd /data/data/com.termux/files/home && cd angryFuzzer") os.system("cd /data/data/com.termux/files/home && pip2 install -r requirements.txt") os.system("cd /data/data/com.termux/files/home && pip2 install requests") os.system("cd /data/data/com.termux/files/home") os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y php") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/Tuhinshubhra/RED_HAWK") os.system("cd /data/data/com.termux/files/home") os.system("pkg update -y") os.system("pkg install -y python2") os.system("pkg install -y git") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/evait-security/weeman.git") os.system("cd /data/data/com.termux/files/home && cd weeman") os.system("cd /data/data/com.termux/files/home && chmod +x weeman.py") os.system("cd /data/data/com.termux/files/home") os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/maldevel/IPGeoLocation.git") os.system("cd /data/data/com.termux/files/home && cd IPGeoLocation") os.system("cd /data/data/com.termux/files/home && pip install -r requirements.txt") os.system("cd /data/data/com.termux/files/home") os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/Mebus/cupp.git") os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python") os.system("pkg install -y nano") os.system("pip install requests") os.system("pip install beautifulsoup4") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/avramit/instahack.git") os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python") os.system("pip install mechanicalsoup") os.system("pkg install -y nano") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/abdallahelsokary/TwitterSniper.git") os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python2") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/blackvkng/viSQL.git") os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python2") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/UltimateHackers/Hash-Buster.git") os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python2") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/shawarkhanethicalhacker/D-TECT.git") os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python2") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/reverse-shell/routersploit.git") os.system("cd /data/data/com.termux/files/home && cd routersploit") os.system("pip2 install -r requirements.txt") os.system("pip2 install -r requirements-dev.txt") os.system("pip2 install -r requests") os.system("clear") print("========================================") print("[+] everything successfully installed :)") print("[+] Happy Hacking <3") print("========================================") else: rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break if what == "1": os.system("cd /data/data/com.termux/files/home") os.system("pkg update -y") os.system("pkg install -y nmap") os.system("cd /data/data/com.termux/files/home") print("====================================") print("[+] nmap installed successfully :)") print("[+] Type 'nmap' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "2": os.system("cd /data/data/com.termux/files/home") os.system("pkg update -y") os.system("pkg install -y hydra") os.system("cd /data/data/com.termux/files/home") print("====================================") print("[+] hydra installed successfully :)") print("[+] Type 'hydra' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "3": os.system("cd /data/data/com.termux/files/home") os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install python2") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/sqlmapproject/sqlmap.git") os.system("cd /data/data/com.termux/files/home") print("====================================") print("[+] SQLMap installed successfully :)") print("[+] Go to sqlmap folder and type 'python2 sqlmap.py' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "4": os.system("pkg install -y wget") os.system("cd /data/data/com.termux/files/home && wget https://Auxilus.github.io/metasploit.sh") os.system("cd /data/data/com.termux/files/home && bash metasploit.sh") os.system("cd /data/data/com.termux/files/home && cd metasploit-framework") os.system("cd /data/data/com.termux/files/home && gem install bundle --pre") os.system("cd /data/data/com.termux/files/home && bundle config build.nokogiri --use-system-libraries") os.system("cd /data/data/com.termux/files/home && bundle install") os.system("cd /data/data/com.termux/files/home") print("====================================") print("[+] Metasploit installed successfully :)") print("[+] Type 'msfconsole' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "5": os.system("pkg update -y") os.system("pkg install -y git") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/themastersunil/ngrok.git") os.system("cd /data/data/com.termux/files/home") print("====================================") print("[+] ngrok installed successfully :)") print("[+] Go to ngrok folder and type './ngrok http 80' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "6": os.system("pkg update -y") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/Hax4us/Nethunter-In-Termux.git") os.system("cd /data/data/com.termux/files/home && cd Nethunter-In-Termux && chmod +x kalinethunter") os.system("cd /data/data/com.termux/files/home") print("====================================") print("[+] Nethunter installed successfully :)") print("[+] Go to Nethunter-In-Termux folder and type './kalinethunter' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "7": os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python2") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/ihebski/angryFuzzer.git") os.system("cd /data/data/com.termux/files/home && cd angryFuzzer") os.system("cd /data/data/com.termux/files/home && pip2 install -r requirements.txt") os.system("cd /data/data/com.termux/files/home && pip2 install requests") os.system("cd /data/data/com.termux/files/home") print("====================================") print("[+] angryFuzzer installed successfully :)") print("[+] Go to angryFuzzer folder and type 'python2 angryFuzzer.py' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "8": os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y php") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/Tuhinshubhra/RED_HAWK") os.system("cd /data/data/com.termux/files/home") print("====================================") print("[+] RED_HAWK installed successfully :)") print("[+] Go to RED_HAWK folder and type 'php rhawk.php' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "9": os.system("pkg update -y") os.system("pkg install -y python2") os.system("pkg install -y git") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/evait-security/weeman.git") os.system("cd /data/data/com.termux/files/home && cd weeman") os.system("cd /data/data/com.termux/files/home && chmod +x weeman.py") os.system("cd /data/data/com.termux/files/home") print("====================================") print("[+] weeman installed successfully :)") print("[+] Go to weeman folder and type 'python2 weeman.py' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "10": os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/maldevel/IPGeoLocation.git") os.system("cd /data/data/com.termux/files/home && cd IPGeoLocation") os.system("cd /data/data/com.termux/files/home && pip install -r requirements.txt") os.system("cd /data/data/com.termux/files/home") print("====================================") print("[+] IPGeoLocation installed successfully :)") print("[+] Go to IPGeoLocation folder and type 'python ipgeolocation.py' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "11": os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/Mebus/cupp.git") print("====================================") print("[+] Cupp installed successfully :)") print("[+] Go to cupp folder and type 'python cupp3.py' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "12": os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python") os.system("pkg install -y nano") os.system("pip install requests") os.system("pip install beautifulsoup4") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/avramit/instahack.git") print("====================================") print("[+] Instahack installed successfully :)") print("[+] Go to instahack folder and type 'python hackinsta.py' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "13": os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python") os.system("pip install mechanicalsoup") os.system("pkg install -y nano") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/abdallahelsokary/TwitterSniper.git") print("====================================") print("[+] TwitterSniper installed successfully :)") print("[+] Go to TwitterSniper folder and type 'python TwitterSniper.py' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "14": os.system("pkg update -y") os.system("pkg install -y git") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/Neo-Oli/termux-ubuntu.git") os.system("cd /data/data/com.termux/files/home && cd termux-ubuntu && bash ubuntu.sh") print("====================================") print("[+] Ubuntu installed successfully :)") print("[+] Go to termux-ubuntu folder and type './start.sh' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "15": os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y wget") os.system("apt update && apt install wget -y && /data/data/com.termux/files/usr/bin/wget https://raw.githubusercontent.com/nmilosev/termux-fedora/master/termux-fedora.sh") print("====================================") print("[+] Fedora installed successfully :)") print("[+] Type 'sh termux-fedora.sh f26_arm64' or 'sh termux-fedora.sh f26_arm' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "16": os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python2") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/blackvkng/viSQL.git") print("====================================") print("[+] viSQL installed successfully :)") print("[+] Go to viSQL folder and type 'python2 viSQL.py --help' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "17": os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python2") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/UltimateHackers/Hash-Buster.git") print("====================================") print("[+] Hash-Buster installed successfully :)") print("[+] Go to Hash-Buster folder and type 'python2 hash.py' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "18": os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python2") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/shawarkhanethicalhacker/D-TECT.git") print("====================================") print("[+] Hash-Buster installed successfully :)") print("[+] Go to Hash-Buster folder and type 'python2 hash.py' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break os.system("pkg update -y") os.system("pkg install -y git") os.system("pkg install -y python2") os.system("cd /data/data/com.termux/files/home && git clone https://github.com/reverse-shell/routersploit.git") os.system("cd /data/data/com.termux/files/home && cd routersploit") os.system("pip2 install -r requirements.txt") os.system("pip2 install -r requirements-dev.txt") os.system("pip2 install -r requests") print("====================================") print("[+] routersploit installed successfully :)") print("[+] Go to routersploit folder and type 'python2 rsf.py' to start.") print("====================================") rmenu = input("[?] Back to Menu? (y/n): ") if rmenu == "y": menu() else: break elif what == "99": print("Bye.") break
    DUFFIHACKEUR
    53 Apr 05, 2024
    For beginners which zero knowledge on ethical hacking!
    CSI-SFIT
    53 Apr 07, 2024
    Brain Wallet passphares and private key crack woth python hack

    for Created Address From Private Key used This Code : first install package's: for running use this common : for use online generetad and crack brain wallet run code:

    Pymmdrza
    55 Apr 13, 2024
    python projects for ethical hacking and the students to learn python for cyper security (<_>)
    fikrado-orgnasation
    55 Mar 21, 2024

    Subscribe to our newsletter