User login

Blogs

03

Feb

2014

I thought it would be a good idea to start off the week with more refactoring and reorganising the Cuz Javascript folder structure, since it would cause minimal disruption while everyone else is away. After this was done I started going through points from the todo list on the whiteboard:

  • Reimplemented History.js based tab switching - this time using AJAX to send a request for the appropriate view ID before switching to it (had previously not been doing this, which caused problems). Functionally this works well now, but could do with a loading spinner to indicate that the page is doing something while it is waiting to receive the view ID it needs to switch to.

  • Made page titles consistent. Removed redundant strings in titles such as "CUZ - Cuz - subtitle". Titles are now always prepended with "CUZ - " at the lowest level, so higher level titles should omit it.

  • Added a new library for detecting timezone + DST and matching with a string location representation (e.g. "Pacific/Auckland"). This should be a bit nicer than the hacky timezone function that depended largely on each browser's timezone representation.

  • Work on tooltips in modal dialogs. Started using popovers instead of tooltips for displaying descriptions of field headings (but still use tooltips for radio button descriptions etc).

I made some more UI changes, one of the more noticeable being that I have replaced the Source Sans Pro font with the Roboto family (Roboto, Roboto Condensed and Roboto Slab), which I think suit the project a bit more.

I modified the Flotr2 Hit plugin to use Bootstrap tooltips instead of its standard tooltips, and made these appear relative to the mouse cursor or part of the graph being hovered over. This should help to unify our aesthetic and also solves a problem we had with text in Flotr2 tooltips overflowing off the side of the page.

I did some more work on the traceroute map, updating it to work with recent changes to the way graphs load (no loading bars and made visible as soon as data is available). Once I'd figured out the new structure I was actually able to remove a lot of the code I already had in the traceroute map thanks to this, as it had already been doing something similar - attempting to process data as soon as it was available and subsequently displaying the graph if everything had finished loading. I started working on a fix for summary data being too highly aggregated to display anything useful, with the idea that (with some restrictions) the graph should load data for the summary graph at the same resolution as the detail graph. In order to accomplish this we must request data for the summary graph at the same interval as the current detail interval. This gets pretty crazy at super high resolution however, and the code is very rough so I need to refine this.

29

Jan

2014

Warts data was analysed to find the sets of internal diamond nodes that had changed. This was then made into a graph for the paper.

A graph of unique load balancer intersection sets was also constructed for per flow and per destination data.

Further updates were made to the paper including changes to the discussion and conclusions.

The shortened Internet simulator test run was found to have completed successfully. The full length run was initiated.

28

Jan

2014

Continued working on a parallel version int: interface using PACKET_FANOUT and how parallel formats will fit in best.

Looked into hashing again and what NIC's actually implement in hardware, the de facto standard for this appears to be Toeplitz hashing to support Microsoft receive side scaling.
Wrote a software version of Toeplitz hashing for libtrace as a software fallback. With a correctly select input key for the Toeplitz hashing algorithm bi-direction hashing can be achieved.

Put a short set of slides together for NZNOG about my libtrace work.

28

Jan

2014

This week I have continued investigating BitTorrent identification using Hidden Markov Models.

I have spent time reading the BitTorrent protocol specification, and also reading up on the MSE employed by the BitTorrent protocol for obfuscation.

I have started working on code that leverages libflowmanager and libtrace in order to organise packets into flows that can be used to train the Hidden Markov Model.

The final deliverable will likely be included as an additional "tool" for libprotoident, as it does not fit nicely into the existing protocol detectors for libprotoident.

28

Jan

2014

Continued working on splitting out IPv4 and IPv6 data on the matrix. Cells in the matrix are now diagonally split in two by default to show IPv4 on the left and IPv6 on the right. I've added the option to show only IPv4 data or only IPv6 data, though clicking on a cell will still take you to the graph for IPv4/IPv6 irrespective of the view you're in. I tweaked the severity colours and added a legend for them that changes based on the current test type (relative/absolute latency, loss, hop count).

More refactoring in the matrix, fixes for matrix popovers and more general IE8 fixes, particularly to get modal dialogs working. Added helpful tooltips to radio buttons for aggregation in the modals to explain what they do.

Changed the colour of event lines on smokeping graphs with only one series. The series would be drawn in black which would clash with the grey of event lines, so I set event lines to be the same colour as their event markers in this situation.

Finished up the week by adding some performance improvements to smokeping graphs that hugely improve software rendering of the canvas (particularly in Firefox on Linux). The canvas is always antialiased so rounding our drawing coordinates to nearest integers prevents unnecessary antialiasing and results in smokeping summary graphs being drawn instantly (which is mainly where our problem was). They aren't noticeably reduced in quality because the diagonal lines connecting points are still antialiased, so we benefit greatly from this (if anything I think the sharper edges look better). I also rounded the drawing coordinates on event markers and the rainbow traceroute graphs for drawing clarity, and I separated out some of the data processing that was in the smokeping drawing code which will mean that in future if we still require better performance, it would be possible to move more of the code into a separate thread.

24

Jan

2014

I got the congestion packet loss to work by separating the ovs bridges with veth links and using tc to limit the throughput. As expected you see a small amount of loss across the paths.

In the process I have found that the issue with occasional packets being miscounted by ovs in the change between one of rule to another is fairly significant. It turns out to be a fraction of a percent of all packets, but that still means thousands of packets getting miscounted. On the other hand, that level of inaccuracy will likely be buried under the packets lost due to standard tcp behaviour. You wont be able to see the exact levels of packet loss that you would expect however.

My next plan is to get the pica8 set up so I can test what openflow functions exist in picos that arent in ovs. Supposedly MPLS and groups work, which is fairly important for what I am doing. I'm not sure I am entirely confident about this though.

23

Jan

2014

Figured out how to use Bayes as a method to combine the beliefs/probabilities to obtain a final significance probability out of the results of several detectors. I had to use different values than the DS ones I had previously calculated, so I spent a while calculating and double-checking the values I needed for Bayes. After that, I did some manual calculations/testing before diving into implementing it in the eventing python script.

Also read a few other other papers regarding different methods of belief fusion, namely the Averaging and Cumulative functions. After talking to Richard, we decided to implement those functions so as to compare the values obtained by each method.

I also read some material on Fuzzy Logic, so I plan on implementing that next.

Modified the eventing script to enable easy addition of different belief fusion methods, since I plan on implementing more methods as I come across them.

21

Jan

2014

The scamper run on Planetlab has finished and the results are being downloaded.

Further updates have been carried out on the paper. This has included updates to the turnover section and the introduction.

My six monthly report was finalised and submitted.

21

Jan

2014

More work on improving the consistency and quality of JS and CSS in the AMP web interface this week. I fixed lots of minor bugs to get everything working in IE8, starting with the most basic ECMAScript dialect differences such as removing extraneous trailing commas and polyfilling some non-existent functions, then moving on to the URL handling code, which previously only took into account how URLs are rewritten in HTML5 browsers (which can write over the current URL). HTML4 browsers such as IE8 and below cannot rewrite the URL so they instead append as a hash/fragment a URI relative to the base URL (where the base URL is that of the page first directly accessed, i.e. typed into the address bar). HTML4 and 5 are both now handled by checking for the existence of a hash and resolving the fragment if it exists, otherwise resolving the base URL. I broke my work on the history code down into a few stages, first tackling the matrix, graph pages, then modal dialogs. I fixed up the rest of the history state handling code afterwards to restore state while moving forwards and backwards between pages, which hadn't been hooked up before.

Another common issue worth mentioning is that some browsers, particularly IE8 and below require explicit object.hasOwnProperty() checks when iterating through object properties with code like for (var x in y) { do things }, which would become for (var x in y) { if (y.hasOwnProperty(x) { do things } }. Instances where these checks weren't happening created several not easily detectable bugs, because IE will additionally iterate through the object's prototype (and debugging IE is a nightmare).

To finish off the week I ripped apart the matrix and removed its dependence on the rather bloated DataTables JS library, added documentation and did some much needed refactoring and tidy up, then started working on splitting up cells to show IPv4 and IPv6 stats.

21

Jan

2014

Moved the mapper 'packet loop' into the library so the user doesn't need to provide this. Instead the user now supplies a per packet function which will accept packets and messages.

Added a fast path for single threaded operation.

Started working on a native implementation of a parallel trace format, Linux socket (int:) using the socket option PACKET_FANOUT.