Malwoverview is a simple suite to perform an initial and quick triage of malware samples, URLs and hashes.
Additionally, Malwoverview is able to show some cyber threat intelligence information. Written in Python.
This suite aims to :
- Determine similar executable malware samples (PE/PE+) according to the import table (imphash) and group them by different colors (pay attention to the second column from output). Thus, colors matter!
- Show hash information on Virus Total, Hybrid Analysis, Malshare, Polyswarm, URLhaus, Alien Vault, Malpedia and ThreatCrowd engines.
- Determining whether the malware samples contain overlay and, if you want, extract it.
- Check suspect files on Virus Total, Hybrid Analysis and Polyswarm.
- Check URLs on Virus Total, Malshare, Polyswarm, URLhaus engines and Alien Vault.
- Download malware samples from Hybrid Analysis, Malshare, URLHaus, Polyswarm and Malpedia engines.
- Submit malware samples to VirusTotal, Hybrid Analysis and Polyswarm.
- List last suspected URLs from URLHaus.
- List last payloads from URLHaus.
- Search for specific payloads on the Malshare.
- Search for similar payloads (PE32/PE32+) on Polyswarm engine.
- Classify all files in a directory searching information on Virus Total and Hybrid Analysis.
- Make reports about a suspect domain using different engines such as VirusTotal, Malpedia and ThreatCrowd.
- Check APK packages directly from Android devices against Hybrid Analysis and Virus Total.
- Submit APK packages directly from Android devices to Hybrid Analysis and Virus Total.
- Show URLs related to an user provided tag from URLHaus.
- Show payloads related to a tag (signature) from URLHaus.
- Show information about an IP address from Virus Total, Alien Vault, Malpedia and ThreatCrowd.
- Show IP address, domain and URL information from Polyswarm.
- Perform meta-search on Polyswarm Network using several criteria: imphash, IPv4, domain, URL and malware family.
- Gather threat hunting information from AlienVault using different criteria.
- Gather threat hunting information from Malpedia using different criteria.
- Gather threat hunting information from ThreatCrowd using different criteria.
- Provide Yara rules and associated information from Valhalla.
INSTALLATION
This software has been tested on Ubuntu, Kali Linux 2020, REMnux, Windows 8.1 and 10. Malwoverview can be installed by executing the following command:
pip3.8 install git+https://github.com/alexandreborges/malwoverview (preferred method) or... python -m pip install -U malwoverview or... git clone https://github.com/alexandreborges/malwoverview
To use Malwoverview you should insert VirusTotal, Hybrid Analysis, URLHaus, Malshare, Polyswarm, Alien Vault and Malpedia APIs into the .malwapi.conf configuration file (the default one at the home directory — if the file doesn’t exist, so you should create it) or you could create a custom configuration file and indicate it by using the -c option.
A special note about the Alien Vault: it is necessary to subscribe to pulses on Alien Vault website before using -n 1 option.
The .malwapi.conf configuration file (from the the home directory) has the following format:
[VIRUSTOTAL] VTAPI = [HYBRID-ANALYSIS] HAAPI = [MALSHARE] MALSHAREAPI = [HAUSSUBMIT] HAUSSUBMITAPI = [POLYSWARM] POLYAPI = [ALIENVAULT] ALIENAPI = [MALPEDIA] MALPEDIAAPI = [VALHALLA] VALHALLAAPI =
The APIs can be requested on the respective service websites:
- Virus Total (community and paid API): https://www.virustotal.com/gui/join-us
- Hybrid Analysis: https://www.hybrid-analysis.com/signup
- Malshare: https://malshare.com/doc.php
- URLHaus: https://urlhaus.abuse.ch/api/#account
- Polyswarm: https://docs.polyswarm.io/consumers
- Alien Vault: https://otx.alienvault.com/api
- Malpedia: It doesn’t offer open registration, but you can request an user account directly through Twitter (DM) or feedback e-email. The Malpedia Twitter handle is @malpedia.
- Valhalla: You can use a demo API key or contract the Valhalla service for a private key. Valhalla demo-API key is: 1111111111111111111111111111111111111111111111111111111111111111.
- ThreadCrowd: It isn’t necessary an API.
In Windows systems, create the .malwapi.conf file in the C:\Users\[username] directory (Windows home user directory). In Linux systems, create the .malwapi.conf file in /home/[username] directory (Linux home user directory).
Additionally, you don’t need to specify “-w 1” option while using malwoverview.py on Windows anymore.
To check the installation, execute:
malwoverview.py --help
Further information is available on:
(PYPI.org repository) https://pypi.org/project/malwoverview/ (Github) https://github.com/alexandreborges/malwoverview
If you want to perform the manual steps (usually, it is not necessary), so few steps will be necessary:
Kali Linux (manual steps)
1. Python version 3.8 or later (Only Python 3.x !!! It does NOT work using Python 2.7)
$ apt-get install python3.8 (for example)
2. Python-magic.
To install python-magic package you can execute the following command:
$ pip3.8 install python-magic
Or compiling it from the github repository:
$ git clone https://github.com/ahupp/python-magic $ cd python-magic/ $ python3.8 setup.py build $ python3.8 setup.py install
As there are serious problems about existing two versions of python-magic package, my recommendation is to install it from github (second procedure above) and copy the magic.py file to the SAME directory of malwoverview tool.
3. Install several Python packages:
$ pip3.8 install -r requirements.txt OR $ pip3.8 install -U pefile $ pip3.8 install -U colorama $ pip3.8 install -U simplejson $ pip3.8 install -U python-magic $ pip3.8 install -U requests $ pip3.8 install -U validators $ pip3.8 install -U geocoder $ pip3.8 install -U polyswarm-api $ pip3.8 install -U pathlib $ pip3.8 install -U configparser $ pip3.8 install -U valhallaAPI
4. To check an Android mobile you need to install the “adb” program by executing the following command:
# apt get install adb
PS: before trying Android’s options, check:
* If the adb program is listed in the PATH environment variable. * If the system has authorized access to the device by using "adb devices -l"
5. Finally, you can install Malwoverview using one of the following methods:
pip3.8 install git+https://github.com/alexandreborges/malwoverview (preferred method) or... python -m pip install -U malwoverview or... git clone https://github.com/alexandreborges/malwoverview
Windows (manual steps)
1. Install the Python version 3.8.x or later from https://www.python.org/downloads/windows/
2. Python-magic.
To install python-magic package you can execute the following command:
C:\> python.exe -m pip install python-magic
Or compiling it from the github repository:
C:\> git clone https://github.com/ahupp/python-magic C:\> cd python-magic/ C:\> python.exe setup.py build C:\> python.exe setup.py install
3. Install several Python packages:
C:\> python.exe -m pip install -r requirements.txt OR: C:\> python.exe -m pip install -U pefile C:\> python.exe -m pip install -U colorama C:\> python.exe -m pip install -U simplejson C:\> python.exe -m pip install -U python-magic C:\> python.exe -m pip install -U requests C:\> python.exe -m pip install -U validators C:\> python.exe -m pip install -U geocoder C:\> python.exe -m pip install -U polyswarm-api C:\> python.exe -m pip install -U pathlib C:\> python.exe -m pip install -U configparser C:\> python.exe -m pip install -U python-magic-bin C:\> python.exe -m pip install -U valhallaAPI
4. To check an Android mobile you need to install the “adb” program by:
* Downloading and installing the Android Studio from: https://developer.android.com/ studio#downloads (Recommended) * Downloading it from https://dl.google.com/android/repository/platform-tools-latest-windows.zip
PS: before trying Android’s options, check:
* If the adb program is listed in the PATH environment variable. * If the system has authorized access to the device by using "adb devices -l"
5. Finally, you can install Malwoverview using one of the following methods:
pip3.8 install git+https://github.com/alexandreborges/malwoverview (preferred method) or... python -m pip install -U malwoverview or... git clone https://github.com/alexandreborges/malwoverview
USAGE
To use the malwoverview, execute the command as shown below:
root@ubuntu19:~/malwoverview# python3.8 malwoverview.py | more usage: python malwoverview.py -c <API configuration file> -d <directory> -f <fullpath> -b <0|1> -v <0|1|2|3> -a <0|1|2|3|4|5> -x <0|1> -w <0|1> -u <url> -H <hash file> -V <filename> -D <0|1> -e<0|1|2|3|4> -A <filename> -g <job_id> -r <domain> -t <0|1> -l <1-14> -L <hash> -U <url> -S <url> -z <tags> -K <0|1|2> -j <hash> -J <hash> -P <filename> -R <PE file, IP address, domain or URL> -G <0|1|2|3|4> -y <0|1|2|3> -Y <file name> -Y <file name> -T <file name> -W <tag> -k <signature> -I <ip address> -n <1|2|3|4|5> -N <argument> -M <1-8> -m <argument> -Q <1-5> -q <argument> -E <1|2|3|4|5> -C <argument>
Malwoverview is a malware triage tool written by Alexandre Borges.
Optional arguments:
-h, –help Show this help message and exit
-c CONFIG FILE, –config CONFIG FILE
Use a custom config file to specify API’s
-d DIRECTORY, –directory DIRECTORY
Specifies the directory containing malware samples.
-f FILENAME, –filename FILENAME
Specifies a full path to a malware sample. It returns general information about the file (any filetype)
-b BACKGROUND, –background BACKGROUND
Adapts the output colors to a white terminal. The default is black terminal
-x OVERLAY, –overlay OVERLAY
Extracts the overlay (it is used with -f option).
-v VIRUSTOTAL, –virustotal VIRUSTOTAL
If using “-v 1”, so it queries the Virus Total database for positives and totals. If “v 2” (which can be used only together with -f option), so it shows antivirus reports from the main players. If “v 3”, so the binary’s IAT and EAT are also shown. Remember: you need to edit the .malwapi.conf and insert your VT API.
-a HYBRID_ANALYSIS, –hybrid HYBRID_ANALYSIS
Queries the Hybrid Analysis database for getting a general report. Possible values are: 1: Windows 7 32-bit; 2: Windows 7 32-bit (HWP Support); 3: Windows 64-bit; 4: Android; 5: Linux 64-bit. Remember: you need to edit the .malwapi.conf and insert your HA API and secret.
-u URL_VT, –vturl URL_VT
SUBMITS a URL to the Virus Total scanning.
-I IP_VT, –ipaddrvt IP_VT
This option checks an IP address on Virus Total.
-r URL_DOMAIN, –urldomain URL_DOMAIN
This option gets a domain’s report from Virus Total.
-H FILE_HASH, –hash FILE_HASH
This option specifies the hash to be checked on Virus Total and Hybrid Analysis.
For the Hybrid Analysis report you must use it with the -e option.
-V FILENAME_VT, –vtsubmit FILENAME_VT
Submits a file(up to 32MB) for Virus Total scanning and gets the report. Attention: use forward slash to specify the target file even on Windows systems. Furthermore, the minimum waiting time is set up in 90 seconds because the Virus Total waiting
queue. If an error occurs, so wait few minutes and try to access the report by using -f option.
-A SUBMIT_HA, –submitha SUBMIT_HA
Submits a file(up to 32MB) to be scanned by the Hybrid Analysis engine. Use the -e option to specify the best environment to run the suspicious file.
-g HA_STATUS, –hastatus HA_STATUS
Checks the report’s status of submitted samples to Hybrid Analysis engine by providing the job ID. Possible returned status values are: IN_QUEUE, SUCCESS, ERROR, IN_PROGRESS and PARTIAL_SUCCESS.
-D DOWNLOAD, –download DOWNLOAD
Downloads the sample from Hybrid Analysis, Malshare and Polyswarm. Options
-H or -L (Hybrid Analysis and Malshare, respectively) must be specified as well
-O option for Polyswarm engine.
-e HA_ENVIRONMENT, –haenv HA_ENVIRONMENT
This option specifies the used environment to be used to test the samlple on Hybrid Analysis: <0> Windows 7 32-bits; <1> Windows 7 32-bits (with HWP Support);
<2> Windows 7 64-bits; <3> Android; <4> Linux 64-bits environment. This option is used together either -H option or the -A option.
-t MULTITHREAD, –thread MULTITHREAD
(optional) This option has several different meanings according to chosen the value. Possible values: <1>: This value is used to force multithreads on Linux whether: the -d option is specified AND you have a PAID Virus Total API or you are NOT checking the VT while using the -d option. PS1: using this option causes
Imphashes not to be grouped anymore; PS2: it also works on Windows, but there is not gain in performance; <2>: This value should be used with -d option in two scenarios: 1) either including the “-v 1” option (Virus Total — you’ll see a complete VT response whether you have the private API) for a multithread searching and reduced output; 2) or including the -a option (Hybrid Analysis) for a multithread searching to get a complete and amazing output. If you are using the -a option, so you should pickup the right number represening the testing environment to adjust the output to your sample types. PS1: certainly, if you have a directory holding many malware samples, so you will want to test this option with -a option; PS2: it also works on Windows, but there is not gain in performance; <3>: You should use this value with -v option if you have a public Virus Total API. It forces a one
minute wait every 4 malware samples, but allows obtaining a complete evaluation of the malware repository.
-l MALSHARE_HASHES, –malsharelist MALSHARE_HASHES
This option shows hashes of a specific type from the last 24 hours from Malshare
repository. Possible values are: 1: PE32 (default) ; 2: Dalvik ; 3: ELF ; 4: HTML ;5: ASCII ; 6: PHP ; 7: Java ; 8: RAR ; 9: Zip ; 10: UTF-8 ; 11: MS-DOS ; 12: data ; 13: PDF ; 14: Composite(OLE). You need to insert your Malshare API into the .malwapi.conf file.
-L MALSHARE_HASH_SEARCH, –malsharehash MALSHARE_HASH_SEARCH
Searches for the provided hash on the Malshare repository. You need to insert your Malshare API into the .malwapi.conf file. PS: sometimes the Malshare website is unavailable, so should check the website availability if you get some error message.
-K HAUS_PAYLOAD_URL, –haus_payloadbatch HAUS_PAYLOAD_URL
THis option has few possible values: <1> Retrieves a list of downloadable links of recent PAYLOADS (last 3 days, limited to 1000 entries) from URLHaus website; <2>: Retrieves a list of recent URLs (last 3 days, limited to 1000 entries) from URLHaus
website. Take care: each link take you to download a passworless zip file containing a malware, so your AV can generate alerts!
-U URL_HAUS_QUERY, –haus_query URL_HAUS_QUERY
Queries a URL on the URLHaus website.
-j HAUS_HASH, –haus_hash HAUS_HASH
Queries information about a provided payload’s hash (md5 or sha256) on the URLHaus website.
-S URL_HAUS_SUB, –haus_submission URL_HAUS_SUB
Submits a URL used to distribute malware (executable, script, document) to the URLHaus website. Pay attention: Any other submission will be ignored/deleted from URLhaus. You have to register your URLHaus API into the .malwapi.conf file.
-z [HAUSTAG [HAUSTAG …]], –haustag [HAUSTAG [HAUSTAG …]] Associates tags (separated by spaces) to the specified URL. Please, only upper case, lower case, ‘-‘ and ‘.’ are allowed. This parameter is optional, which could be used with the -S option.
-W [HAUSTAGSEARCH [HAUSTAGSEARCH …]], –haustagsearch [HAUSTAGSEARCH [HAUSTAGSEARCH …]] This option is for searching malicious URLs by tag on URLhaus. Tags are case-senstive and only upper case, lower case, ‘-‘ and ‘.’ are allowed.
-k [HAUSSIGSEARCH [HAUSSIGSEARCH …]], –haussigsearch [HAUSSIGSEARCH [HAUSSIGSEARCH …]] This option is for searching malicious payload by tag on URLhaus. Tags are case-sensitive and only upper case, lower case, ‘-‘ and ‘.’ are allowed.
-J HAUS_DOWNLOAD, –haus_download HAUS_DOWNLOAD
Downloads a malware sample (if it is available) from the URLHaus repository. It is necessary to provide the SHA256 hash.
-P POLYSWARMFILE, –polyswarm_scan POLYSWARMFILE
(Only for Linux) Submits a sample to Polyswarm engine and performs a file scan.
-O POLYSWARMHASH, –polyswarm_hash POLYSWARMHASH
(Only for Linux) Performs a hash scanning using the Polyswarm engine. Optionally, you can specify -D option to download the sample. Take care: Polyswarm enforces a restriction to number of downloaded samples in 20/month.
-R POLYSWARMMETA, –polyswarm_meta POLYSWARMMETA
(Only for Linux) Provides the argument value for searches on Polyswarm engine through imphash (the PE file must be provided), ipv4, domain, URL and family. This argument must be used with -G option, so check it, please. Pay attention: you should check your metadata search limit on your Polyswarm account because once you have got the limit, so you will got an error.
-G METATYPE, –metatype METATYPE
(Only for Linux) This parameter specifies search type for arguments provided by -R option (above) while searching on Polyswarm engine. Thus, the following values are valid — 0: PE Executable (look for samples with the same ImpHash); 1: IP Address ;
2: Domain ; 3. URL; 4. Family
-y ANDROID_HA, –androidha ANDROID_HA
This option has multiple options: <1>: Check all third-party APK packages from the USB-connected Android device against Hybrid Analysis using multithreads. The Android device does not need to be rooted and the system does need to have the adb tool in the PATH environment variable; <2>: Check all third-party APK packages from the USB-connected. Android device against VirusTotal using Public API (slower because of 60 seconds delay for each 4 hashes). The Android device does not need to be rooted and the system does need to have adb tool in the PATH environment variable; <3>: Check all third-party APK packages from the USB-connected Android device against VirusTotal using multithreads (only for Private Virus API). The Android device does not need to be rooted and the system needs to have adb tool in the PATH environment variable.
-Y ANDROID_SEND_HA, –androidsendha ANDROID_SEND_HA
Sends an third-party APK package from your USB-connected Android device to Hybrid Analysis. The Android device does not need to be rooted and the system needs to have adb tool in the PATH environment variable.
-T ANDROID_SEND_VT, –androidsendvt ANDROID_SEND_VT
Sends an third-party APK package from your USB-connected Android device to Virus Total. The Android device does not need be rooted and the system needis to have the adb tool in the PATH environment variable.
-n ALIENVAULT, –alienvault ALIENVAULT
Checks multiple information from AlienVault. The possible values are: 1: Get the subscribed pulses ; 2: Get information about an IP address; 3: Get information about a domain; 4: Get information about a hash; 5: Get information about a URL
-N ALIENVAULT_ARGS, –alienvaultargs ALIENVAULT_ARGS
Provides argument to AlienVault -n option.The allowed values are: 1, 2, 3, 4, 5.
-M MALPEDIA, –malpedia MALPEDIA
This option is related to MALPEDIA and presents different meanings depending on the chosen value. Thus, 1: List meta information for all families ; 2: List all actors ID ; 3: List all available payloads organized by family from Malpedia; 4: Get meta
information from an specific actor, so it is necessary to use the -m option. Additionally, try to confirm the correct actor ID by executing malwoverview with option -M 3; 5: List all families IDs; 6: Get meta information from an specific family, so it is necessary to use the -m option. Additionally, try to confirm the correct family ID by executing malwoverview with option -M 5; 7: Get a malware sample from malpedia (zip format — password: infected). It is necessary to specify the requested hash by using -m option; 8: Get a zip file containing Yara rules for a specific family (get the possible families using -M 5), which must be specified by using -m option.
-m MALPEDIAARG, –malpediarg MALPEDIAARG
This option provides an argument to the -M option, which is related to MALPEDIA.
-Q THREATCROWD, –threatcrowd THREATCROWD
Checks multiple information from ThreatCrowd. The possible values are:
1: Get information about the provided e-mail ; 2: Get information about an IP address; 3: Get information about a domain; 4: Get
information about a provided MD5 hash; 5: Get information about a specific malware
family.
-q THREATCROWDARG, –threatcrowdarg THREATCROWDARG
This option provides an argument to the -Q option, which is related to THREATCROWD.
-E VALHALLA, –valhalla VALHALLA
This option is used for getting Yara rules from the Valhalla service given an argument (-C option below). Valid values are 1: searches for Yara rules matching the provided keyword; 2: search for Yara rules matching a minimal score (40-49: anomaly and threat hunting rules / 60-74: rules for suspicious objects / 75-100: hard malicious matches); 3: Look for Yara rules to the following products, which must be specified using the -C option: FireEyeAX, FireEyeNX, FireEyeEX, CarbonBlack, Tanium, Tenable, SymantecMAA, GRR, osquery, McAfeeATD3 and McAfeeATD4; 4: Given the hash (SHA 256) through -C option, show associated Yara rules; 5: Shows information about a specific Yara rule provided through the -C option.
-C VALHALLAARG, –valhallaarg VALHALLAARG
This option is used for providing argument to the Vahalla service (-E option).
Remember that public VT API only allows 4 searches per second (as shown at the image above). Therefore, if you are willing to wait some minutes, so you can use the -p option, which forces a one minute wait every 4 malware samples, but allows obtaining a complete evaluation of the repository.
* ATTENTION 1: if the directory contains many malware samples while using -d option, so malwoverview.py could take some time. Nonetheless, you can use the new -t option (multithreading) to speed-up things. 🙂
** ATTENTION 2: All engines enforces quota of submission and/or verification per day and/or month.
*** ATTENTION 3: Some options searching on Hybrid Analysis strongly depend of the “-e” option, which specifies the environment. Therefore, to check an Android sample (for example) it is necessary to use the right environment (-e 3 for Android).
Also Read: What is API Security?
