Published on July 15th, 2015 📆 | 7292 Views ⚑0
ivre — Python Network Recon Framework
IVRE (Instrument de veille sur les réseaux extérieurs) or DRUNK (Dynamic Recon of UNKnown networks) is a network recon framework, including two modules for passive recon (one p0f-based and one Bro-based) and one module for active recon (mostly Nmap-based, with a bit of ZMap).
External programs / dependencies
IVRE relies on:
- Python 2, version 2.6 minimum
- Nmap & ZMap
- Bro & p0f
- MongoDB, version 2.6 minimum
- a web server (successfully tested with Apache and Nginx, should work with anything capable of serving static files and run a Python-based CGI), although a test web server is now distributed with IVRE (
- a web browser (successfully tested with recent versions of Firefox and Chromium)
- Maxmind GeoIP free databases
- optionally Tesseract, if you plan to add screenshots to your Nmap scan results
- optionally Docker & Vagrant (version 1.6 minimum)
IVRE comes with:
If you plan to run scans from a machine, install Nmap and optionally ZMap.
If you plan to analyze PCAP file on a machine, install Bro and p0f.
To install IVRE, you’ll need Python 2, version 2.6 minimum (prefer 2.7), with the following modules:
- pymongo version 2.7.2 minimum.
The installation of IVRE itself can be done by:
- using the
./setup.py build; sudo ./setup.py install) script.
- using pip: with on a Debian-based system for example, install the packages
python-dev(needed to build dependencies) and run
pip install ivre(this will download and install for you IVRE and its Python dependencies from PyPI, the Python Package Index).
- building an RPM package (you can use the provided
buildrpmscript, or use the
setup.pyscript with your own options) and then installing it.
- using Docker (in this case you do not need to follow the following instructions, as the Docker containers are already configured).
Default configuration values are hard-coded in
ivre/config.py. You should not change this file, unless you are modifying IVRE and you want to change the default configuration. You do not need to do this if you want to install IVRE with a non-default configuration, you just need to distribute a proper configuration file.
You can override default values in three files:
- two system-wide:
/usr/local/etc/ivre.conf(read after, so higher priority)
- one user-specific:
~/.ivre.conf(the last to be read, so highest priority)
The file should contain lines of type
key = value. Empty lines and comments (starting with the
# character) are ignored. The following values can be changed:
DB: the URL to use; default is
mongodb:///, meaning use default database (
ivre) on the default host (
localhost). Here is a more complete example:
DB_DATA: specific URLs to use; default is to use the URL from
GEOIP_PATH: default is
For the full and up-to-date list of settings that can be changed, see the
It might be a good idea to have a read-only account everywhere except for some specific users or hosts that need write access to the database (the users that insert scan results with
nmap2db, the users or the hosts that run
passiverecon2db). It is best to avoid using a configuration with write access to the database when you only need a read access. This can be achieved with users or hosts dedicated to insertion tasks.
Once IVRE has been properly configured, it’s time to initialize its databases.
For that, the command-line tools (namely
runscans-agentdb, respectively for information about IP addresses, passive information, active information and running scans through agents) have a
So you can run, with a user or from a host where the configuration has a write access to the database (add
< /dev/null to skip the confirmation):
$ scancli --init This will remove any scan result in your database. Process ? [y/N] y $ ipinfo --init This will remove any passive information in your database. Process ? [y/N] y $ ipdata --init This will remove any country/AS information in your database. Process ? [y/N] y # runscans-agentdb --init This will remove any agent and/or scan in your database and files. Process ? [y/N] y
# ipdata --download $ ipdata --import-all --dont-feed-ipdata-cols
Once IVRE has been installed, to also install the web interface, you have to copy or symlink IVRE files to your web server directories, or configure your web server to use IVRE files directly.
The files the web server should serve statically are located in
[PREFIX]/share/ivre/web/static, the folder the web server should serve as CGI is located in
[PREFIX]/share/ivre/web/cgi-bin, and the (optional) folders to use as Dokuwiki content are located in
[PREFIX]/share/ivre/dokuwiki/media. Make sure your Dokuwiki has been configured with server-side URL rewriting; this means using proper rewrite in your Web server configuration (with
mod_rewrite when using Apache; you can use the provided
Dockerfiles as examples on how to configure Apache or Nginx) and adding
$conf['userewrite'] = 1 in your Dokuwiki config file.
You may want to change some values, by creating the file
[PREFIX]/share/ivre/web/static/config.js based on the
-sample file and by creating or modifying
On a typical Debian/Ubuntu installation with Apache and Dokuwiki installed with the distribution packages, these files should be copied or (sym)linked at these locations:
WEB_LIMIT from IVRE’s configuration must match the values
limit in the
dflt object in
If you do not plan to run active scans with remote agents (where IVRE will not be installed), you can skip this section.
The agent does not require IVRE to be installed. It is a script that needs to be adapted to each situation.
The agent is only needed when you cannot install IVRE on the machine used to scan or when you want to use many machines to run one scan.
It requires a POSIX environment, and the commands
nmap (of course). See the AGENT file for more information about that.
The following steps will show some examples of passive network recon with IVRE. If you only want active (for example, Nmap-based) recon, you can skip this part.
You need to run bro (2.3 minimum) with the option
-b and the location of the
passiverecon.bro file. If you want to run it on the
eth0 interface, for example, run:
# mkdir logs # bro -b /usr/local/share/ivre/passiverecon/passiverecon.bro -i eth0
If you want to run it on the
capture file (
capture needs to a PCAP file), run:
$ mkdir logs $ bro -b /usr/local/share/ivre/passiverecon/passiverecon.bro -r capture
This will produce log files in the
logs directory. You need to run a
passivereconworker to process these files. You can try:
$ passivereconworker --directory=logs
This program will not stop by itself. You can (
kill it, it will stop gently (as soon as it has finished to process the current file).
To start filling your database with information from the
eth0 interface, you just need to run (
passiverecon is just a sensor name here):
# p0f2db -s passiverecon iface:eth0
And from the same
$ p0f2db -s passiverecon capture
Using the results
You have two options for now:
ipinfocommand line tool
db.passiveobject of the
For example, to show everything stored about an IP address or a network:
$ ipinfo 184.108.40.206 $ ipinfo 220.127.116.11/24
See the output of
To use the Python module, run for example:
$ python >>> from ivre.db import db >>> db.passive.get(db.passive.flt_empty)
For more, run
help(db.passive) from the Python shell.
The easiest way is to install IVRE on the “scanning” machine and run:
# runscans --routable --limit 1000 --output=XMLFork
This will run a standard scan against 1000 random hosts on the Internet by running 30 nmap processes in parallel. See the output of
runscans --help if you want to do something else.
When it’s over, to import the results in the database, run:
$ nmap2db -c ROUTABLE-CAMPAIGN-001 -s MySource -r scans/ROUTABLE/up
ROUTABLE-CAMPAIGN-001 is a category (just an arbitrary name that you will use later to filter scan results) and
MySource is a friendly name for your scanning machine (same here, an arbitrary name usable to filter scan results; by default, when you insert a scan result, if you already have a scan result for the same host address with the same source, the previous result is moved to an “archive” collection (fewer indexes) and the new result is inserted in the database).
There is an alternative to installing IVRE on the scanning machine that allows to use several agents from one master. See the AGENT file, the program
runscans-agent for the master and the
agent/ directory in the source tree.
You have three options:
scanclicommand line tool
db.nmapobject of the
- the web interface
To get all the hosts with the port 22 open:
$ scancli --port 22
To use the Python module, run for example:
$ python >>> from ivre.db import db >>> db.nmap.get(db.nmap.flt_empty)
For more, run
help(db.nmap) from the Python shell.
The Web interface
The top navigation bar
It contains several elements; from left to right:
- A shortcut to the start page, that cleans every keyword.
- A button to display this help page.
- Some menus with shortcuts to add filtering, sort or display commands.
- Some links to “share” (export) the current page.
The left side bar
The first part allows to navigate within the results. Be careful with the last button that goes to the last result page, as it can be very slow when a lot of results are available.
The progress bar shows where the currently displayed results are within the whole results set.
The second part allows to add, modify or remove filter, sort or display commands.
The third part allows to explore the results by generating graphs displayed in the rightmost part of the screen.
- The first field displays a graph with the 15 most common values of a variable in the filtered results. This can be slow when the number of results to scan is important. Here is a list of (sometimes) interesting values to try here:
cpe.product:a:microsoftwill show top product names in CPEs from vendor
cpe.vendor:o:/^m/will show top vendor names in CPEs that start with an
- The Address space button displays a graphical representation of the filtered addresses. The abscissa axis represents the two high bytes (or the three when the results belong to the same /16 network), and the ordinate axis represents the two low bytes (or the low byte).
- The Map button displays the locations of the results on a world map.
- The Timeline and Timeline 24h buttons display time-lines where the abscissa axis represents the time and the ordinate axis represents the IP addresses.
Ten results (maximum) are displayed per page by default.
Each result has its own frame. In the default display mode, it displays a summary for the host. Long-clicking a result frame toggles between the summary display and the full display for the result.
The pencil icon in the upper-right corner opens the notepad page for the current host (see below) in the rightmost part of the screen.
Each blue element in the results can be clicked to add a filter.
The commands might require a parameter, provided after the colon sign
:. Some commands can be used negatively, by prefixing them with
The commands can be entered in the input boxes in the second part of the left side bar or added by clicking on a shortcut in the top bar menus.
In the following list, a
[!] before the command shows it can be used negatively, and a
: after the command indicates it requires a parameter.
When a parameter is required the full value must be specified, or when appropriate, a regular expression can be used, with the
/[expression]/[flags] syntax (e.g.:
If your command includes spaces, you need to protect it by using single or double quotes.
[!]archivesshow results from the archives database (and
!archiveshas no effect since it is the default behavior).
[!]host:[IP address]filter a specific IP address. Using the IP address directly (without
host:) is equivalent.
[!]net:[IP address/netmask]filter a specific network (CIDR notation). Using the CIDR notation directly (without
net:) is equivalent.
[!]range:[IP address]-[IP address]filter a specific IP address range
[!]hostname:[FQDN]look for results with a matching hostname.
[!]domain:[FQDN]look for results with a hostname within a matching domain name.
[!]category:filter a category.
[!]country:[two letters code]filter a country.
[!]city:filter a city (use with
[!]asnum:filter by AS number (lists allowed).
[!]asname:filter by AS name (regular expressions allowed).
[!]source:filter a source (specify the source name).
[!]timerange:[timestamp]-[timestamp]filter results within a specific time range.
[!]timeago:filter recent enough results; the value can be specified in seconds or with the appropriate suffix in minutes (
m), hours (
h), days (
d) or years (
service:[expression]:[port number]look for an expression in the name of a service.
probedservice:[expression]:[port number]look for an expression in the name of a service discovered with a service probe.
product:[service]:[product]:[port number]look for a product.
product:[service]:[product]:[version]:[port number]look for a specific version of a product.
script:[scriptid]:[output]look for a specific (port) script (using
hostscript:[scriptid]:[output]look for a specific host script.
anonftpfilter results with anonymous FTP allowed.
anonldaplook for LDAP servers with anonymous bind working.
authbypassvnclook for VNC servers with authentication that can be bypassed.
authhttplook for HTTP servers with authentication and a default (e.g.,
admin) login/password working. The Nmap script seems to get a lot a false positives.
banner:look for a specific banner of a service.
cookie:look for HTTP servers setting a specific cookie.
file:look for a pattern in the shared files (FTP, SMB, …).
geovisionlook for GeoVision web-cams.
httptitle:look for a specific HTML title value of the homepage of a web site.
nfslook for NFS servers.
yplook for NIS servers.
mssqlemptypwdlook for MS-SQL servers with an empty password for the
mysqlemptypwdlook for MySQL servers with an empty password for the
owalook for OWA (Outlook Web App) servers.
phpmyadminlook for phpMyAdmin servers.
smb.dnsdomain:[FQDN]search results with SMB service in a specific DNS domain.
smb.domain:[NetBIOS]search results with SMB service in a specific NetBIOS domain.
smb.fqdn:[NetBIOS]search results with SMB service in a specific host name (FQDN).
smb.forest:[FQDN]search results with SMB service in a specific forest (DNS name).
smb.lanmanager:[LAN Manager]search results with SMB service with a specific LAN Manager.
smb.os:[OS]search results with SMB service with a specific OS.
smb.server:[NetBIOS]search results with SMB service in a specific host name (NetBIOS).
smb.workgroup:[NetBIOS]search results with SMB service in a specific workgroup (NetBIOS).
sshkey:look for a particular SSH key.
torcertlook for Tor certificates.
webfileslook for “typical” web files in the shared folders.
webminlook for Webmin servers.
x11openlook for open X11 servers.
x11srvlook for X11 servers.
xp445look for Windows XP machines with TCP/445 port open.
os:look for a specific value in the OS discovery results.
devicetype:look for a type of devices.
networkdevicelook for network devices (firewalls, routers, …).
phonedevlook for telephony devices.
cpe(:[type](:[vendor](:[product](:[version]))))look for a given cpe. Each field can be a /regex/.
[!]hop:look for a particular IP address in the traceroute results.
[!]hopname:look for a matching hostname in the traceroute results.
[!]hopdomain:look for a hostname within a matching domain name in the traceroute results.
[!]udp/[port number], look for an open TCP or UDP port (using
[!][port number]directly is equivalent to
[!]openportlook for hosts with at least one open port.
notessearch results with an associated note.
[!]sortby:[field name]sort according to a field value. Be careful with this setting as consequences on the performances can be terrible.
display:hostset the default display mode.
display:cpeonly display CPEs.
display:script:[scriptid]only display a particular script output.
display:screenshotonly display screenshots.