User login

Blogs

04

Feb

2014

Spent most of the week tidying up things in preparation for the AMP
website to be demoed at NZNOG next week. Fixed up some graph colours,
labels and descriptions that were inconsistent across views so that they
all match. Tried to squeeze a bit more performance out of our current
database setup to make our queries faster, and wrote a quick script to
keep refreshing the matrix data into memcache.

Added some error notifications for when ajax requests failed so that the
user has some feedback rather than waiting forever with no indication
that something might have gone wrong.

Fixed up a bug with the area selection in the summary graph that would
shrink the selection if it was at the edge of the graph.

04

Feb

2014

Last week was rather short (holiday and unwell for a couple of days). Spent a bit of time looking into other fusion methods, but then decided to take a break and look into writing the eventing script's output to a database (for easier inspection). Talked to Shane and he created a separate database that I could play with, just to be safe. After looking at their current schema, I spent some time thinking about an ideal way of storing the probabilities of the different methods in the database. Finally, finalised the schema, created the tables and started working on inserting the event data into the DB.

04

Feb

2014

The process of starting the Internet simulator on a full size run was completed.

Corrections throughout the whole paper have been made. It may now be time for Tony to have a look at it, or maybe some other possible critics.

I wrote a short review on the progress and possible forward strategy of my research. Some planning is required at this stage to work out how to develop a rounded story to tell.

04

Feb

2014

Spent a lot of time chasing down deadlock behaviour in netevmon when it first starts up. The problem ultimately turned out to be that anomalyfeed was requesting a large amount of stream data from NNTSC, which was causing both ends to get stuck trying to complete a blocking send to the other. Reduced the likelihood of this occuring in the future by forcing anomalyfeed to wait for all streams for a collection to arrive before asking for any more streams, but the proper solution is going to be moving to non-blocking transmits.

Also replaced some pipes within the NNTSC exporting code with Queues, as full pipes were also causing problems. These problems were much worse, as one of the full pipes would stop NNTSC from processing new data and inserting it into the database.

Fixed a segfault in anomaly_ts due to reading off the end of a buffer. The problem was that we were using strchr to look for a newline character but never checking if the character we found was within sensible bounds.

Fixed a whole bunch of problems with colour consistency on our graphs. We now use the colour assignments provided when we receive the legend information rather than trying to replicate the colour assignment inside javascript. This fixed a lot of bugs we had where the line colours would not match the legend or the colours would cycle each time new data was fetched.

Spent last week at NZNOG where we managed to give a reasonably successful presentation of everything we've done up until now. Managed to generate a bit of interest from operators, so we must be doing something right.

04

Feb

2014

Added a parallel version of ring: which uses the PACKET_FANOUT socket option like int:.

Added a general type to be set along with the hasher function (bidirectional, unidirectional, balance and custom) which allow the format to deal with this request directly, otherwise the user supplied function or a default libtrace function will be used to meet the requirement.

Worked on some other general code refactoring and fixes.

03

Feb

2014

I thought it would be a good idea to start off the week with more refactoring and reorganising the Cuz Javascript folder structure, since it would cause minimal disruption while everyone else is away. After this was done I started going through points from the todo list on the whiteboard:

  • Reimplemented History.js based tab switching - this time using AJAX to send a request for the appropriate view ID before switching to it (had previously not been doing this, which caused problems). Functionally this works well now, but could do with a loading spinner to indicate that the page is doing something while it is waiting to receive the view ID it needs to switch to.

  • Made page titles consistent. Removed redundant strings in titles such as "CUZ - Cuz - subtitle". Titles are now always prepended with "CUZ - " at the lowest level, so higher level titles should omit it.

  • Added a new library for detecting timezone + DST and matching with a string location representation (e.g. "Pacific/Auckland"). This should be a bit nicer than the hacky timezone function that depended largely on each browser's timezone representation.

  • Work on tooltips in modal dialogs. Started using popovers instead of tooltips for displaying descriptions of field headings (but still use tooltips for radio button descriptions etc).

I made some more UI changes, one of the more noticeable being that I have replaced the Source Sans Pro font with the Roboto family (Roboto, Roboto Condensed and Roboto Slab), which I think suit the project a bit more.

I modified the Flotr2 Hit plugin to use Bootstrap tooltips instead of its standard tooltips, and made these appear relative to the mouse cursor or part of the graph being hovered over. This should help to unify our aesthetic and also solves a problem we had with text in Flotr2 tooltips overflowing off the side of the page.

I did some more work on the traceroute map, updating it to work with recent changes to the way graphs load (no loading bars and made visible as soon as data is available). Once I'd figured out the new structure I was actually able to remove a lot of the code I already had in the traceroute map thanks to this, as it had already been doing something similar - attempting to process data as soon as it was available and subsequently displaying the graph if everything had finished loading. I started working on a fix for summary data being too highly aggregated to display anything useful, with the idea that (with some restrictions) the graph should load data for the summary graph at the same resolution as the detail graph. In order to accomplish this we must request data for the summary graph at the same interval as the current detail interval. This gets pretty crazy at super high resolution however, and the code is very rough so I need to refine this.

29

Jan

2014

Warts data was analysed to find the sets of internal diamond nodes that had changed. This was then made into a graph for the paper.

A graph of unique load balancer intersection sets was also constructed for per flow and per destination data.

Further updates were made to the paper including changes to the discussion and conclusions.

The shortened Internet simulator test run was found to have completed successfully. The full length run was initiated.

28

Jan

2014

Continued working on a parallel version int: interface using PACKET_FANOUT and how parallel formats will fit in best.

Looked into hashing again and what NIC's actually implement in hardware, the de facto standard for this appears to be Toeplitz hashing to support Microsoft receive side scaling.
Wrote a software version of Toeplitz hashing for libtrace as a software fallback. With a correctly select input key for the Toeplitz hashing algorithm bi-direction hashing can be achieved.

Put a short set of slides together for NZNOG about my libtrace work.

28

Jan

2014

This week I have continued investigating BitTorrent identification using Hidden Markov Models.

I have spent time reading the BitTorrent protocol specification, and also reading up on the MSE employed by the BitTorrent protocol for obfuscation.

I have started working on code that leverages libflowmanager and libtrace in order to organise packets into flows that can be used to train the Hidden Markov Model.

The final deliverable will likely be included as an additional "tool" for libprotoident, as it does not fit nicely into the existing protocol detectors for libprotoident.

28

Jan

2014

Continued working on splitting out IPv4 and IPv6 data on the matrix. Cells in the matrix are now diagonally split in two by default to show IPv4 on the left and IPv6 on the right. I've added the option to show only IPv4 data or only IPv6 data, though clicking on a cell will still take you to the graph for IPv4/IPv6 irrespective of the view you're in. I tweaked the severity colours and added a legend for them that changes based on the current test type (relative/absolute latency, loss, hop count).

More refactoring in the matrix, fixes for matrix popovers and more general IE8 fixes, particularly to get modal dialogs working. Added helpful tooltips to radio buttons for aggregation in the modals to explain what they do.

Changed the colour of event lines on smokeping graphs with only one series. The series would be drawn in black which would clash with the grey of event lines, so I set event lines to be the same colour as their event markers in this situation.

Finished up the week by adding some performance improvements to smokeping graphs that hugely improve software rendering of the canvas (particularly in Firefox on Linux). The canvas is always antialiased so rounding our drawing coordinates to nearest integers prevents unnecessary antialiasing and results in smokeping summary graphs being drawn instantly (which is mainly where our problem was). They aren't noticeably reduced in quality because the diagonal lines connecting points are still antialiased, so we benefit greatly from this (if anything I think the sharper edges look better). I also rounded the drawing coordinates on event markers and the rainbow traceroute graphs for drawing clarity, and I separated out some of the data processing that was in the smokeping drawing code which will mean that in future if we still require better performance, it would be possible to move more of the code into a separate thread.