Oct 272013

As I have been dealing more and more with web page related security I have found the need for a tool that can crawl a web page and search for potential malicious data.
Many of you might already know urlquery.net and the services they provide. It is a great site which loads your web page and tries to find malicious content which might not be visible to the end user.
It does this by loading the web page in firefox, and passing the traffic through Snort, and a couple of other analysis engines. It will also create a graph over the HTTP requests as well as a and overview over the HTTP headers.

Having a real Firefox, with real plugins loading your site proves to be very useful as more and more malware is getting quite sneaky, and will only serve malware to a very limited amount of visitors to avoid detection.

All these features is gold for anyone working with web page security (eg. a hosting provider).

The only problem for me is that urlquery.net isnt open source, which means that every test I do will become public, which I might not be ready for.

I looked into Thug, but lacking a web interface/report viewing interface, I did not find it perfect enough for my use.
I had therefore to create something on my own.
This has resulted in FjoSpidie, a now Open Source Spider/Honey Client.

This spider runs Firefox through Selenium and records the traffic with tcpdump which is passed through Snort to search for known, malicious content.


  • Runs Firefox through Selenium with a custom User Agent for maximal malware hit rate.
  • Processes the traffic from the Firefox session with Snort to find potential malicious content.
  • Stores information about each request and response along with their respective headers.
  • Creates a graph over all requests made.
  • Downloads and saves downloads offered by the site.

A scan of a malicious site will look like this in the web interface:


The Web interface is based on Bootstrap 3, Perl and Catalyst MVC and will let you submit new analysis jobs, and will give you a nice overview over the reports that has been generated.

FjoSpidie can be found here: github.com/espenfjo/fjospidie
The web interface here: github.com/espenfjo/fjospidie-interface