Twitter OSINT framework: Birdwatcher
https://www.ispeech.org/text.to.speech
Birdwatcher is a data analysis and OSINT framework for Twitter. Birdwatcher supports creating multiple workspaces where arbitrary Twitter users can be added and their Tweets harvested through the Twitter API for offline storage and analysis.
Birdwatcher comes with several modules which can be invoked to further enrich collected data or work with it, e.g. Retrieving user’s score, generating social graphs between users and weighted word clouds based on their Tweets.
Installation
1. Ruby
Birdwatcher is written in Ruby and requires at least version 1.9.3 or above. To check which version of Ruby you have installed, simply run ruby --version in a terminal.
Should you have an older version installed, it is very easy to upgrade and manage different versions with the Ruby Version Manager (RVM). Please see the RVM website for installation instructions.
2. RubyGems
Birdwatcher is packaged as a Ruby gem to make it easy to install and update. To install Ruby gems you’ll need the RubyGems tool installed. To check if you have it already, type gem in a Terminal. If you got it already, it is recommended to do a quick gem update --system to make sure you have the latest and greatest version. In case you don’t have it installed, download it from here and follow the simple installation instructions.
3. PostgreSQL
Birdwatcher uses a PostgreSQL database to store all its data. If you are setting up Birdwatcher in theKali linux distribution you already have it installed, you just need to make sure it’s running by executingservice postgresql start and perhaps install a dependency with apt-get install libpq-dev in a terminal. Here’s an excellent guide on how to install PostgreSQL on a Debian based Linux system. If you are setting up Birdwatcher on a Mac, the easiest way to install PostgreSQL is with Homebrew. Here’s aguide on how to install PostgreSQL with Homebrew.
3.1 PostgreSQL user and database
You need to set up a user and a database in PostgreSQL for Birdwatcher. Execute the following commands in a terminal:
sudo su postgres # Not necessary on Mac OS X
createuser -s birdwatcher --pwprompt
createdb -O birdwatcher birdwatcher
You now have a new PostgreSQL user with the name birdwatcher and with the password you typed into the prompt. You also created a database with the name birdwatcher which is owned by thebirdwatcher user.
4. Graphviz
Some Birdwatcher modules use Graphviz to generate visual graphs and other things. On a Mac you can install Graphviz with homebrew by typing brew update && brew install graphviz in a terminal. On a Debian based Linux distro, Graphviz can be installed by typing sudo apt-get update && sudo apt-get install graphviz in a terminal.
5. ImageMagick
Some Birdwatcher modules use ImageMagick to generate images. On a Mac you can install Imagemagick with homebrew by typing brew update && brew install imagemagick in a terminal. On a Debian based Linux distro, ImageMagick can be installed by typing sudo apt-get update && sudo apt-get install libmagickwand-dev imagemagick in a terminal.
6. Birdwatcher
Finally with all the dependencies in place, Birdwatcher can now be installed with a simple command in a terminal:
$ gem install birdwatcher
This will download and set up Birdwatcher and install all its code depencies.
Usage
Birdwatcher is built as a console and if you have any experience with other frameworks such asMetasploit or Recon-ng, you should feel right at home as Birdwatcher has many of the same concepts and commands.
Birdwatcher, like Metasploit and Recon-ng, works with the concept of Workspaces. Workspaces enable you to segment and manage users and data stored in the database. You can use workspaces to create logical separation between different users. For example, you may want to create a workspace for a company, a department or for a specific topic.
The core of the Birdwatcher framework is of course its commands. You already got an introduction to the workspace command in the previous section, but there are many more. One of the most important commands is help:
birdwatcher[top5]> help
[+] Available commands:
back Unloads current module
exit Exit Birdwatcher
help [COMMAND] Show help and detailed command usage
irb Start an interactive Ruby shell
module ACTION Show modules
query QUERY Execute SQL query
query_csv QUERY Execute SQL query and return result as CSV
resource FILE Execute commands from a resource file
run Run current module
schema [TABLE_NAME] Show schema for database table
set OPTION VALUE Set module option
shell COMMAND Execute shell command
show DETAILS Show module details and options
status [ACTION] Manage statuses
unset OPTION Unset module option
use MODULE_PATH Load specified module
user [ACTION] Manage users
workspace [ACTION] Manage workspaces
birdwatcher[top5]>[adsense size='1']
Modules
Being able to fetch Twitter users and statuses from the API is cool and all, but if that was all Birdwatcher would just be an offline Twitter client. Modules are where the fun begins. Modules either enrich collected data with more data or do some sort of work on the collected data.
Here are some of the things the modules can do:
- Generate weighted word clouds based on statuses
- Listing the most shared URLs in a certain time frame
- Generate visual social graphs between users
- Crawl shared URLs to fetch status codes, content types and page titles
- Generate KML files with geo-enabled statuses for viewing in Google Earth
- Generate Punchard-style plots of when users are most engaged with Twitter
To see all available modules, use the module list command:
birdwatcher[top5]> module list
[+] Available Modules:
Name: KML Document
Description: Creates a KML document of statuses with Geo locations
Path: statuses/kml
================================================================================
Name: Status Sentiment Analysis
Description: Enrich statuses with sentiment score
Path: statuses/sentiment
================================================================================
Name: Social Graph
Description: Graphs the social relations between users
Path: users/social_graph
================================================================================
birdwatcher[top5]>The name, description and path is listed for each available module. The module path is how the modules are divided up into folders on the file system. The folders can be seen as namespaces and gives a clue about what sort of objects they work on. The path is used to select modules with the usecommand. The use command simply loads a module by providing the modules path as an argument. This also changes the command prompt to display the currently loaded module in square brackets next to the currently active workspace.
MODULES:
https://github.com/michenriksen/birdwatcher/wiki/Creating-a-Birdwatcher-Module
https://github.com/michenriksen/birdwatcher
Gloss