User login

Search Projects

Project Members

BSOD

The bsod visualisation tool uses the libtrace framework to transform network traffic into a graphical format that can be viewed in real time. Capturing from a live network interface, or from a saved trace file, bsod visualises the flow of network data between hosts, providing (at a glance) information about network usage.

The BSOD webpage is http://research.wand.net.nz/software/visualisation.php

25

Jul

2016

Short week after taking leave on Monday and Tuesday.

Spent most of my remaining week looking at some new captures I took using the upgraded Probe. The main aim was to see whether there were any new protocols that libprotoident should be able to identify. Managed to find a handful of new protocols: Facebook Zero, Forticlient SSL VPN and Discord, as well as made some improvements to the rules for existing protocols (including the AMP throughput test!).

Most of my time was actually spent unsuccessfully hunting down what appears to be a new Chinese P2P protocol, which is a shame because it was contributing a very large amount of unknown traffic in my sample dataset.

Using BSOD on the live traffic feed also allowed me to spot a student that was doing vast quantities of torrenting on the campus network (which Brad reported to ITS) and our WITS FTP server being hammered with tons of download attempts from China. Fair to say, we've gotten some good milage of the upgraded Probe already.

Fixed a couple of outstanding bugs in amp-web. Should be ready to push some new packages out to skeptic and lamp early next week now.

10

Aug

2015

Made a video demonstrating BSOD with the current University capture point. The final cut can be seen at https://www.youtube.com/watch?v=kJlDY0XvbA4

Alistair King got in touch and requested that libwandio be separated from libtrace so that he can release projects that use libwandio without having libtrace as a dependency as well. With his help, this was pretty straightforward so now libwandio has a separate download page on the WAND website.

Continued my investigation into optimal Plateau detector parameters. Used my web-app to classify ~230 new events in a morning (less than 5 of which qualified as significant) and merged those results back into my original ground truth. Re-ran the analysis comparing the results for each parameter configuration against the updated ground truth. I've now got an "optimal" set of parameters, although the optimal parameters still only achieve 55% precision and 60% recall.

Poked around at some more unknown flows while waiting for the Plateau analysis to run. Managed to identify some new BitTorrent and eMule clients and also added two new protocols: BDMP and Trion games.

20

Jul

2015

More work on the dashboard this week:
* added the ability to remove "common" events from the recent event list and made the graphs collapsible.
* added a table that shows the most frequently occuring events in the past day, e.g. "increased latency from A to B (ipv4)".
* polished up some of the styling on the dashboard and moved the dashboard-specific CSS (of which there is now quite a lot) into its own separate file.

Started thinking about how to include loss-related events in the event groups, as these are ignored at the moment.

The new capture point came online on Wednesday, so the rest of my week was spent playing with the packet captures. This involved:
* learning to operate EndaceVision.
* installing wdcap on the vDAG VM.
* adding the ability to anonymise only the local network in wdcap.
* performing a short test capture.
* getting BSOD working again, which required the application of a little "in-flow" packet sampling to run smoothly.
* running libprotoident against the test capture to see what new rules I can add.

10

Jun

2015

The source code for both BSOD and Meenakshee Mungro's reliable libprotoident collector have been added to the WAND github page. Developers can freely clone these projects and make their own modifications or additions to the source code, while keeping up with any changes that we make between releases.

This is the first time we have released the libprotoident collector under the GPLv3 license. This project is a replacement for the lpi_live tool included with libprotoident, which should now be considered deprecated.

We're also more than happy to consider pull requests for code that adds useful features to either project.

Links:
WAND on GitHub

10

Feb

2014

Started going through all the NNTSC exporting code and replacing any instances of blocking sends with non-blocking alternatives. This should ultimately make both NNTSC and netevmon more stable when processing large amounts of historical data. It is also proving a good opportunity to tidy up some of this code, which had gotten a little ropey with all the hacking done on it leading up to NZNOG.

Spent a decent chunk of my week catching up on various support requests. Had two separate people email about issues with BSOD on Friday.

Wrote a draft version of this year's libtrace assignment for 513. I've changed it quite a bit from last years, based on what the students managed to achieve last year. The assignment itself should require a bit more work this time around, but should be easily doable in just C rather than requiring the additional learning curve of the STL. It should also be much harder to just rip off the examples :)

Read through the full report on a study into traffic classifier accuracy that evaluated libprotoident along with a bunch of other classifiers ( http://vbn.aau.dk/files/179043085/TBU_Extended_dpi_report.pdf ). Pleased to see that libprotoident did extremely well in the cases where it would be expected to do well, i.e. non-web applications.

28

Jun

2013

Had a week of catching up on a few jobs I had put off in lieu of getting NNTSC, netevmon and amp2 ready for the Lightwire release.

Re-worked BSOD server to use a separate thread for communicating with clients, so that the packets can be sent to clients immediately rather than waiting for a break in the input stream. Unfortunately, this hasn't stopped the bursty appearance of packets on the client like I had hoped, so this requires further investigation. I suspect the flow management inside BSOD server isn't as optimal as it could be and may end up replacing this with libflowmanager.

With that in mind, I've modified libflowmanager to support multiple flow expiry 'plugins', as opposed to having a single defined expiry policy that all libflowmanager programs had to use. This will allow us to replicate BSOD's old expiry policy (flows expire after 20 seconds of inactivity) if we want to, although I would probably see how it goes with the classic libflowmanager policy first.

Received some bug reports for libtrace from Matt Brown as a result of Mayhem being run against the entirety of Debian. Perry had more or less patched them right away so I worked on releasing a new version of libtrace incorporating those fixes. The new release went out on Friday and also includes the rawerf fix from several weeks back. Had a few issues with both Fedora and FreeBSD that slowed down the testing process, so the release process took a bit longer than anticipated.

27

May

2013

Finished adding simple time series graphs for our switch interface byte count data. Got Brendon's event rendering working with these new graphs too, so we can now see and explore the events detected using the Plunge and ArimaShewhart detectors. They seem to be working reasonably well so far.

The next task I started on was fixing the URLs for the amp-web graphs -- the current setup is graph/// which is not sustainable going forward. Firstly, the metric needs to come first so that we can handle time series that are defined by more than just a source and target, e.g. a direction or an application protocol. Next, instead of explicitly listing the source, target or whatever else describes the time series data, we want to use the unique stream id from within NNTSC. This also avoids the problem of our URLs being really long or containing spaces. Unfortunately, much of the original code was written with only source and target in mind so there's a lot to change to be able to support LPI data, for example.

Developed a new version of libwandevent. There are two main changes in the new version. Firstly, the allocation and management of event structures is all handled internally by libwandevent -- no more filling in event structures and passing them off to libwandevent. The main reason for this is to try and minimise the chance of bugs where the programmer inadvertantly overwrites an existing event, much like the BSOD bug I had last week. However, it does break the existing API so there may be a slightly messy transition period. Secondly, I've added support for epoll so that will now be used instead of select, if available. Switched BSOD server over to use the new libwandevent and it seems to work pretty well.

20

May

2013

Spent much of my week working on getting BSOD ready to be wheeled out at Open Day once again. During this process, I managed to find and fix a couple of bugs in the server that were now causing nasty crashes. I also tracked down a bug in the client where the UI elements aren't redrawn properly if the window is resized. Normally this hasn't been a big problem, but newer versions of Gnome like to try and silently resize full-screen apps and this meant that our UI was disappearing off the bottom of the screen. As an interim fix, I've disabled resizing in BSOD client but we really should be trying to handle resize events properly.

Received a bug report for libtrace about the compression detection occasionally giving a false positive for uncompressed ERF traces. This is because the ERF header has no identifying 'magic' at the start, so every now and again the first few bytes (where the timestamp is stored) end up matching the bytes we use to identify a gzip header. I've strengthened the gzip check to use an extra byte so the chance of this happening now is 1 in 16 million. I've also added a special URI format called rawerf: so users can force libtrace to treat traces as uncompressed ERF.

Started working on trying to get amp-web to plot graphs of interface byte counts. I've managed to draw a line on the graph, but much of the graph styling is still using the smokeping style. I'm now looking at rewriting the javascript for the graph styling to be a bit more generic and configurable, rather than having one (mostly copied) javascript file for each of our metrics.

Friday was mostly consumed with looking after our displays at Open Day. BSOD continued to impress quite a few people and we were reasonably busy most of the day, so it seemed a worthwhile exercise.

05

Mar

2012

Released a new version of BSOD client on Tuesday.

Did some planning with Brendon, thinking about how we're going to bring all the components of the MSI project together into something usable.

Played around with a live libprotoident application, getting it to write results into a postgresql database and an RRD. Postgresql required a fair bit of revision of SQL and database theory. The RRD was much easier to get up and running.

Continued improvements to libprotoident - trying to get that accuracy rate up even further!

28

Feb

2012

A new version of the BSOD client (2.0.2) was released today. This release fixes the bug where particles would continue traveling past the planes instead of stopping. We've also restored movement through the 3D space using WASD which used to be present in the older clients. Now you can easily zoom in on the interesting endpoints on each plane and click on them easily to identify them!

We've built updated binaries for Mac OS X and Windows too. The Windows binary now comes with a proper installer. Both the Mac and Windows binaries are 32-bit, due to the limitations of some libraries we depend upon, but have been tested successfully on 64-bit machines.

A new version of the server was also recently released that fixes a build error on some systems and fixes a bug where input looping was not working correctly.

The new versions of BSOD server and client can be downloaded from here. Any problems or questions should be addressed to contact [at] wand [dot] net [dot] nz

27

Feb

2012

Re-ran my CAA analysis using the updated libprotoident and updated the results in my paper accordingly.

Made a few tweaks to libtcpcsm, based on suggestions from a user. Looking towards rolling out a new release soon.

Set up a build environment for BSOD client on BIGMAC. This took a bit longer than expected due to the move to Xcode 4. Managed to find and fix a bug in libwandevent that was preventing looping input from working properly. Also got the client building and running on tkn as well after a painful Windows 7 + Visual Studio install.

Finished the week by adding WASD movement back into BSOD client and an option to the server that forces it to wait for a client to connect before reading from the input
source.

20

Feb

2012

Spent most of my week working on the draft version of the paper on the effect of the CAA on DSL users. Finished the draft on Friday, having included plenty of (hopefully) interesting results. Anyone interested in reading over the paper should get in touch with me and I give you a copy.

Patched libtrace to support --with-foo configure options for all the optional dependencies. Apparently this is a bit of an issue with some Linux distros, e.g. Gentoo.

Released a new version of BSOD server on Friday to fix a crash issue that was occurring with recent libprotoident releases.

Spent some time looking at traffic that was being classed as SSL by libprotoident. Turns out that, with a bit of port and payload size analysis, I can sub-classify the SSL as Google talk, Apple push notifications, Facebook chat, PSN store, POP3S and NNTPS.

18

May

2011

I have created Trac sites for both the libprotoident and BSOD projects, so it is now possible to file tickets to report bugs or request features for either of these projects through the Trac system, rather than having to contact me directly.

The Trac sites also feature wikis which I intend to use to provide more extensive documentation for these projects, e.g. explanations of the protocols supported by libprotoident. At the moment, this is a work in progress but hopefully will get fleshed out over time.

The BSOD trac: http://wand.net.nz/trac/bsod/
The libprotoident trac: http://wand.net.nz/trac/libprotoident