[Analyst Arsenal (A²)™]

A toolkit for Security Researchers

Description

With aa_adhoc, run through a list of URLs and check sites for malicious files based on predefined file extensions.

With aa_certstream, find out when a phishing kit has been staged on a domain. With this information, you can be amongst the first to:

Know
Block
Report
Analyze

With aa_urlscan, easily search urlscan.io and check sites for malicious files based on predefined file extensions.

With aa_whoisds, download a list of newly registered domains from WHOIS Domain Search, score the domains, and search for signs of malicious activity.

Prerequisites

Ubuntu 18.04+ (should work on other Linux distros)
Python 2.7.14
DEB Packages:
- gcc
- Git (optional)
- Torsocks (optional: used with flag --tor)

Setup

Open a terminal and run the following command:

git clone https://github.com/ecstatic-nobel/Analyst-Arsenal.git  cd Analyst-Arsenal  bash py_pkg_update.sh

Usage

aa_adhoc
The following command will:

Make requests to the domains retrieved from a file
Download files from the site when an open directory is found hosting a file with the desired file extension

1 positional arguments needed:

Input File : Path to the file containing URLs

Optional arguments:

--directory : Download data to CAP_DIR (default: ./Captures)
--level : Recursion depth (default=1, infinite=0)
--max-redirect : Maximum redirects (default=0)
--quiet : Don't show wget output
--threads : Numbers of threads to spawn
--timeout : Set the connection timeout to TIMEOUT
--tor : Download files via the Tor network
--very-verbose : Show error messages

python aa_adhoc.py <INPUT_FILE> [--directory] [--level] [--max-redirect] [--quiet] [--threads] [--timeout] [--tor] [--very-verbose]

aa_certstream
The following command will:

Stream CT logs via Certstream
Score and add suspicious domains to a queue while other domains continue to be scored
Simultaneously make requests to the domains in the queue to search for predefined file extensions
Recursively download the site when an open directory is found hosting a file with a particular extension

Optional arguments:

--ctl-server : Certstream server URL to connect to
--dns-twist : Check the twisted keywords found in dns_twisted.yaml
--directory : Download data to CAP_DIR (default: ./Captures)
--level : Recursion depth (default=1, infinite=0)
--log-nc : File to store domains that have not been checked
--quiet : Don't show wget output
--score : Minimum score to trigger a session (Default: 75)
--threads : Numbers of threads to spawn
--timeout : Set the connection timeout to TIMEOUT
--tor : Download files via the Tor network
--verbose : Show domains being scored
--very-verbose : Show error messages

python aa_certstream.py [--ctl-server] [--dns-twist] [--directory] [--level] [--log-nc] [--quiet] [--score] [--threads] [--timeout] [--tor] [--verbose] [--very-verbose]

aa_urlscan
The following command will:

Make requests to the domains retrieved from urlscan.io
Recursively download the site when an open directory hosting a file with the desired file extension

3 positional arguments needed:

Query Type : automatic, manual, certstream, openphish, phishtank, twitter, urlhaus
Delta : Number of days back to search (GMT)
Query String : String to search (and does not include spaces)

Optional arguments:

--directory : Download data to CAP_DIR (default: ./Captures)
--level : Recursion depth (default=1, infinite=0)
--max-redirect : Maximum redirects (default=0)
--quiet : Don't show wget output
--threads : Numbers of threads to spawn
--timeout : Set the connection timeout to TIMEOUT
--tor : Download files via the Tor network
--very-verbose : Show error messages

python aa_urlscan.py <QUERY_TYPE> <DELTA> <QUERY_STRING> [--directory] [--level] [--max-redirect] [--quiet] [--threads] [--timeout] [--tor] [--very-verbose]

Note: If the path is a file, it will be automatically downloaded.

aa_whoisds

Download a list of newly registered domains from WHOIS Domain Search (whoisds.com)
Score and add suspicious domains to a queue while other domains continue to be scored
Simultaneously make requests to the domains in the queue to search for predefined file extensions
Recursively download the site when an open directory is found hosting a file with a particular extension

1 positional argument needed:

Delta : Number of days back to search (GMT)

Optional arguments:

--dns-twist : Check the twisted keywords found in dns_twisted.yaml
--directory : Download data to CAP_DIR (default: ./Captures)
--level : Recursion depth (default=1, infinite=0)
--log-nc : File to store domains that have not been checked
--quiet : Don't show wget output
--score : Minimum score to trigger a session (Default: 75)
--threads : Numbers of threads to spawn
--timeout : Set the connection timeout to TIMEOUT
--tor : Download files via the Tor network
--verbose : Show domains being scored
--very-verbose : Show error messages

python aa_whoisds.py <DELTA> [--dns-twist] [--directory] [--level] [--log-nc] [--quiet] [--score] [--threads] [--timeout] [--tor] [--verbose] [--very-verbose]

Things to know

Be responsible!!!
Output messages:
- Complete: download complete or the site canceled it prematurely
- Critical: a domain was found with a score above 120
- Directory: the output directory is unavailable
- Download: checks passed and a download was started
- Empty: the output directory was empty and removed
- Failed: a connection to the site couldn't be made
- Session: checking the site for data included in external.yaml
- Suspicious: a domain was found with a score above 90
- Triggered: a domain was found with the minimum score specified
Check the queue_file.txt file to get a better understanding of how large the queue is. If it's too large, either increase the threads, raise the score, or decrease the level.
If the keywords in config.yaml have been modified and --dns-twist is going to be used, regenerate dns_twisted.yaml by running the following command:
```
bash dnstwist.sh PATH_TO_DNSTWIST_SCRIPT
```
Using the --dns-twist flag will default to a minimum of 20 threads
Downloads via Tor happen over 127.0.0.1:9050
These scripts will not check Torsocks settings