ivre — Python Network Recon Framework
iSpeech.org
IVRE (Instrument de veille sur les réseaux extérieurs) or DRUNK (Dynamic Recon of UNKnown networks) is a network recon framework, including two modules for passive recon (one p0f-based and one Bro-based) and one module for active recon (mostly Nmap-based, with a bit of ZMap).
External programs / dependencies
IVRE relies on:
- Python 2, version 2.6 minimum
- Nmap & ZMap
- Bro & p0f
- MongoDB, version 2.6 minimum
- a web server (successfully tested with Apache and Nginx, should work with anything capable of serving static files and run a Python-based CGI), although a test web server is now distributed with IVRE (
httpd-ivre) - a web browser (successfully tested with recent versions of Firefox and Chromium)
- Maxmind GeoIP free databases
- optionally Tesseract, if you plan to add screenshots to your Nmap scan results
- optionally Docker & Vagrant (version 1.6 minimum)
[adsense size='1']
IVRE comes with:
Installation
Dependencies
If you plan to run scans from a machine, install Nmap and optionally ZMap.
If you plan to analyze PCAP file on a machine, install Bro and p0f.
To install IVRE, you’ll need Python 2, version 2.6 minimum (prefer 2.7), with the following modules:
- Crypto
- pymongo version 2.7.2 minimum.
[adsense size='1']
The installation of IVRE itself can be done by:
- using the
setup.py(classical./setup.py build; sudo ./setup.py install) script. - using pip: with on a Debian-based system for example, install the packages
python-pipandpython-dev(needed to build dependencies) and runpip install ivre(this will download and install for you IVRE and its Python dependencies from PyPI, the Python Package Index). - building an RPM package (you can use the provided
buildrpmscript, or use thesetup.pyscript with your own options) and then installing it. - using Docker (in this case you do not need to follow the following instructions, as the Docker containers are already configured).
[adsense size='1']
Configuration
Default configuration values are hard-coded in ivre/config.py. You should not change this file, unless you are modifying IVRE and you want to change the default configuration. You do not need to do this if you want to install IVRE with a non-default configuration, you just need to distribute a proper configuration file.
You can override default values in three files:
- two system-wide:
/etc/ivre.conf/usr/local/etc/ivre.conf(read after, so higher priority)
- one user-specific:
~/.ivre.conf(the last to be read, so highest priority)
The file should contain lines of type key = value. Empty lines and comments (starting with the # character) are ignored. The following values can be changed:
DB: the URL to use; default ismongodb:///, meaning use default database (ivre) on the default host (localhost). Here is a more complete example:mongodb://user:password@host/db?colname_aaa=bbb&colname_ccc=dddDB_NMAP,DB_PASSIVEandDB_DATA: specific URLs to use; default is to use the URL fromDBsetting.GEOIP_PATH: default is[INSTALL PREFIX]/share/ivre/geoip/.
For the full and up-to-date list of settings that can be changed, see the ivre/config.py file.
It might be a good idea to have a read-only account everywhere except for some specific users or hosts that need write access to the database (the users that insert scan results with nmap2db, the users or the hosts that run p0f2db and/or passiverecon2db). It is best to avoid using a configuration with write access to the database when you only need a read access. This can be achieved with users or hosts dedicated to insertion tasks.
DB creation
Once IVRE has been properly configured, it’s time to initialize its databases.
For that, the command-line tools (namely ipdata, ipinfo, scancli and runscans-agentdb, respectively for information about IP addresses, passive information, active information and running scans through agents) have a --init option.
So you can run, with a user or from a host where the configuration has a write access to the database (add < /dev/null to skip the confirmation):
$ scancli --init
This will remove any scan result in your database. Process ? [y/N] y
$ ipinfo --init
This will remove any passive information in your database. Process ? [y/N] y
$ ipdata --init
This will remove any country/AS information in your database. Process ? [y/N] y
# runscans-agentdb --init
This will remove any agent and/or scan in your database and files. Process ? [y/N] y
Getting IP data
# ipdata --download
$ ipdata --import-all --dont-feed-ipdata-cols
[adsense size='1']
Web Server
Once IVRE has been installed, to also install the web interface, you have to copy or symlink IVRE files to your web server directories, or configure your web server to use IVRE files directly.
The files the web server should serve statically are located in [PREFIX]/share/ivre/web/static, the folder the web server should serve as CGI is located in [PREFIX]/share/ivre/web/cgi-bin, and the (optional) folders to use as Dokuwiki content are located in [PREFIX]/share/ivre/dokuwiki/doc and [PREFIX]/share/ivre/dokuwiki/media. Make sure your Dokuwiki has been configured with server-side URL rewriting; this means using proper rewrite in your Web server configuration (with mod_rewrite when using Apache; you can use the provided Dockerfiles as examples on how to configure Apache or Nginx) and adding $conf['userewrite'] = 1 in your Dokuwiki config file.
You may want to change some values, by creating the file [PREFIX]/share/ivre/web/static/config.js based on the -sample file and by creating or modifying /etc/ivre.conf.
On a typical Debian/Ubuntu installation with Apache and Dokuwiki installed with the distribution packages, these files should be copied or (sym)linked at these locations:
[PREFIX]/share/ivre/web/static/*->/var/wwwor/var/www/html[PREFIX]/share/ivre/web/cgi-bin/scanjson.py->/usr/lib/cgi-bin/[PREFIX]/share/ivre/dokuwiki/doc->/var/lib/dokuwiki/data/pages/[PREFIX]/share/ivre/dokuwiki/media/logo.png->/var/lib/dokuwiki/data/media/[PREFIX]/share/ivre/dokuwiki/media/doc->/var/lib/dokuwiki/data/media/
The values WEB_SKIP and WEB_LIMIT from IVRE’s configuration must match the values skip and limit in the dflt object in config.js.
Agent
If you do not plan to run active scans with remote agents (where IVRE will not be installed), you can skip this section.
The agent does not require IVRE to be installed. It is a script that needs to be adapted to each situation.
The agent is only needed when you cannot install IVRE on the machine used to scan or when you want to use many machines to run one scan.
It requires a POSIX environment, and the commands screen, rsync and nmap (of course). See the AGENT file for more information about that.
Passive recon
The following steps will show some examples of passive network recon with IVRE. If you only want active (for example, Nmap-based) recon, you can skip this part.
Using Bro
You need to run bro (2.3 minimum) with the option -b and the location of the passiverecon.bro file. If you want to run it on the eth0 interface, for example, run:
# mkdir logs
# bro -b /usr/local/share/ivre/passiverecon/passiverecon.bro -i eth0
If you want to run it on the capture file (capture needs to a PCAP file), run:
$ mkdir logs
$ bro -b /usr/local/share/ivre/passiverecon/passiverecon.bro -r capture
This will produce log files in the logs directory. You need to run a passivereconworker to process these files. You can try:
$ passivereconworker --directory=logs
This program will not stop by itself. You can (p)kill it, it will stop gently (as soon as it has finished to process the current file).
[adsense size='1']
Using p0f
To start filling your database with information from the eth0 interface, you just need to run (passiverecon is just a sensor name here):
# p0f2db -s passiverecon iface:eth0
And from the same capture file:
$ p0f2db -s passiverecon capture
Using the results
You have two options for now:
- the
ipinfocommand line tool - the
db.passiveobject of theivre.dbPython module
For example, to show everything stored about an IP address or a network:
$ ipinfo 1.2.3.4
$ ipinfo 1.2.3.0/24
See the output of ipinfo --help.
To use the Python module, run for example:
$ python
>>> from ivre.db import db
>>> db.passive.get(db.passive.flt_empty)[0]
For more, run help(db.passive) from the Python shell.
Active recon
Scanning
The easiest way is to install IVRE on the “scanning” machine and run:
# runscans --routable --limit 1000 --output=XMLFork
This will run a standard scan against 1000 random hosts on the Internet by running 30 nmap processes in parallel. See the output of runscans --help if you want to do something else.
When it’s over, to import the results in the database, run:
$ nmap2db -c ROUTABLE-CAMPAIGN-001 -s MySource -r scans/ROUTABLE/up
Here, ROUTABLE-CAMPAIGN-001 is a category (just an arbitrary name that you will use later to filter scan results) and MySource is a friendly name for your scanning machine (same here, an arbitrary name usable to filter scan results; by default, when you insert a scan result, if you already have a scan result for the same host address with the same source, the previous result is moved to an “archive” collection (fewer indexes) and the new result is inserted in the database).
There is an alternative to installing IVRE on the scanning machine that allows to use several agents from one master. See the AGENT file, the program runscans-agent for the master and the agent/ directory in the source tree.
Using the results
You have three options:
- the
scanclicommand line tool - the
db.nmapobject of theivre.dbPython module - the web interface
CLI: scancli
To get all the hosts with the port 22 open:
$ scancli --port 22
[adsense size='1']
Python module
To use the Python module, run for example:
$ python
>>> from ivre.db import db
>>> db.nmap.get(db.nmap.flt_empty)[0]
For more, run help(db.nmap) from the Python shell.
The Web interface
The top navigation bar
It contains several elements; from left to right:
- A shortcut to the start page, that cleans every keyword.
- A button to display this help page.
- Some menus with shortcuts to add filtering, sort or display commands.
- Some links to “share” (export) the current page.
The left side bar
The first part allows to navigate within the results. Be careful with the last button that goes to the last result page, as it can be very slow when a lot of results are available.
The progress bar shows where the currently displayed results are within the whole results set.
The second part allows to add, modify or remove filter, sort or display commands.
The third part allows to explore the results by generating graphs displayed in the rightmost part of the screen.
- The first field displays a graph with the 15 most common values of a variable in the filtered results. This can be slow when the number of results to scan is important. Here is a list of (sometimes) interesting values to try here:
category,sourcecountry,city,asdomains,domains:[level]hop,hop:[number]port,port:[open/closed/filtered],port:[service]portlist:[open/closed/filtered]service,service:[port],probedservice,probedservice:[port],product,product:[port],version,version:[port]cpe,cpe.[type/vendor/product/version],cpe:[cpe spec],cpe.[type/vendor/product/version]:[cpe spec](examples:cpe.product:a:microsoftwill show top product names in CPEs from vendormicrosoft,cpe.vendor:o:/^m/will show top vendor names in CPEs that start with anm)devicetype,devicetype:[port]script,portscript(likescript),hostscriptscript:[scriptname],portscript:[scriptname](likescript:),hostscript:[scriptname]smb.os,smb.lanmanager,smb.domain,smb.dnsdomain,smb.forest,smb.workgroupcert.issuer,cert.subjectmodbus.deviceid,enip.vendor,enip.product,enip.serial,enip.devtype,enip.prodcode,enip.rev,enip.ip
- The Address space button displays a graphical representation of the filtered addresses. The abscissa axis represents the two high bytes (or the three when the results belong to the same /16 network), and the ordinate axis represents the two low bytes (or the low byte).
- The Map button displays the locations of the results on a world map.
- The Timeline and Timeline 24h buttons display time-lines where the abscissa axis represents the time and the ordinate axis represents the IP addresses.
Scan results
Ten results (maximum) are displayed per page by default.
Each result has its own frame. In the default display mode, it displays a summary for the host. Long-clicking a result frame toggles between the summary display and the full display for the result.
The pencil icon in the upper-right corner opens the notepad page for the current host (see below) in the rightmost part of the screen.
Each blue element in the results can be clicked to add a filter.
Available commands
Command specification
The commands might require a parameter, provided after the colon sign :. Some commands can be used negatively, by prefixing them with ! or -.
The commands can be entered in the input boxes in the second part of the left side bar or added by clicking on a shortcut in the top bar menus.
In the following list, a [!] before the command shows it can be used negatively, and a : after the command indicates it requires a parameter.
When a parameter is required the full value must be specified, or when appropriate, a regular expression can be used, with the /[expression]/[flags] syntax (e.g.: hostscript:smb-enum-shares:/WRITE/).
If your command includes spaces, you need to protect it by using single or double quotes.
Commands list
Filters
[!]archivesshow results from the archives database (and!archiveshas no effect since it is the default behavior).[!]host:[IP address]filter a specific IP address. Using the IP address directly (withouthost:) is equivalent.[!]net:[IP address/netmask]filter a specific network (CIDR notation). Using the CIDR notation directly (withoutnet:) is equivalent.[!]range:[IP address]-[IP address]filter a specific IP address range[!]hostname:[FQDN]look for results with a matching hostname.[!]domain:[FQDN]look for results with a hostname within a matching domain name.[!]category:filter a category.[!]country:[two letters code]filter a country.[!]city:filter a city (use withcountry:).[!]asnum:filter by AS number (lists allowed).[!]asname:filter by AS name (regular expressions allowed).[!]source:filter a source (specify the source name).[!]timerange:[timestamp]-[timestamp]filter results within a specific time range.[!]timeago:filter recent enough results; the value can be specified in seconds or with the appropriate suffix in minutes (m), hours (h), days (d) or years (y).service:[expression],service:[expression]:[port number]look for an expression in the name of a service.probedservice:[expression],probedservice:[expression]:[port number]look for an expression in the name of a service discovered with a service probe.product:[service]:[product],product:[service]:[product]:[port number]look for a product.version:[service]:[product]:[version],product:[service]:[product]:[version]:[port number]look for a specific version of a product.script:[scriptid],script:[scriptid]:[output]look for a specific (port) script (usingportscript:is equivalent).hostscript:[scriptid],hostscript:[scriptid]:[output]look for a specific host script.anonftpfilter results with anonymous FTP allowed.anonldaplook for LDAP servers with anonymous bind working.authbypassvnclook for VNC servers with authentication that can be bypassed.authhttplook for HTTP servers with authentication and a default (e.g.,admin/admin) login/password working. The Nmap script seems to get a lot a false positives.banner:look for a specific banner of a service.cookie:look for HTTP servers setting a specific cookie.file:look for a pattern in the shared files (FTP, SMB, …).geovisionlook for GeoVision web-cams.httptitle:look for a specific HTML title value of the homepage of a web site.nfslook for NFS servers.nis,yplook for NIS servers.mssqlemptypwdlook for MS-SQL servers with an empty password for thesaaccount.mysqlemptypwdlook for MySQL servers with an empty password for therootaccount.owalook for OWA (Outlook Web App) servers.phpmyadminlook for phpMyAdmin servers.smb.dnsdomain:[FQDN]search results with SMB service in a specific DNS domain.smb.domain:[NetBIOS]search results with SMB service in a specific NetBIOS domain.smb.fqdn:[NetBIOS]search results with SMB service in a specific host name (FQDN).smb.forest:[FQDN]search results with SMB service in a specific forest (DNS name).smb.lanmanager:[LAN Manager]search results with SMB service with a specific LAN Manager.smb.os:[OS]search results with SMB service with a specific OS.smb.server:[NetBIOS]search results with SMB service in a specific host name (NetBIOS).smb.workgroup:[NetBIOS]search results with SMB service in a specific workgroup (NetBIOS).sshkey:look for a particular SSH key.torcertlook for Tor certificates.webfileslook for “typical” web files in the shared folders.webminlook for Webmin servers.x11openlook for open X11 servers.x11srvlook for X11 servers.xp445look for Windows XP machines with TCP/445 port open.os:look for a specific value in the OS discovery results.devtype:,devicetype:look for a type of devices.netdev,networkdevicelook for network devices (firewalls, routers, …).phonedevlook for telephony devices.cpe(:[type](:[vendor](:[product](:[version]))))look for a given cpe. Each field can be a /regex/.[!]hop:look for a particular IP address in the traceroute results.[!]hopname:look for a matching hostname in the traceroute results.[!]hopdomain:look for a hostname within a matching domain name in the traceroute results.[!]tcp/[port number],[!]udp/[port number], look for an open TCP or UDP port (using[!][port number]directly is equivalent to[!]tcp/[port number]).[!]openportlook for hosts with at least one open port.notessearch results with an associated note.
[adsense size='1']
Sort
skip:[count]skipcountfirst results.limit:[count]only displaycountresults.[!]sortby:[field name]sort according to a field value. Be careful with this setting as consequences on the performances can be terrible.
Display
display:hostset the default display mode.display:cpeonly display CPEs.display:script:[scriptid]only display a particular script output.display:screenshotonly display screenshots.
Gloss