User login

Blogs

28

Jan

2014

Continued working on a parallel version int: interface using PACKET_FANOUT and how parallel formats will fit in best.

Looked into hashing again and what NIC's actually implement in hardware, the de facto standard for this appears to be Toeplitz hashing to support Microsoft receive side scaling.
Wrote a software version of Toeplitz hashing for libtrace as a software fallback. With a correctly select input key for the Toeplitz hashing algorithm bi-direction hashing can be achieved.

Put a short set of slides together for NZNOG about my libtrace work.

28

Jan

2014

This week I have continued investigating BitTorrent identification using Hidden Markov Models.

I have spent time reading the BitTorrent protocol specification, and also reading up on the MSE employed by the BitTorrent protocol for obfuscation.

I have started working on code that leverages libflowmanager and libtrace in order to organise packets into flows that can be used to train the Hidden Markov Model.

The final deliverable will likely be included as an additional "tool" for libprotoident, as it does not fit nicely into the existing protocol detectors for libprotoident.

28

Jan

2014

Continued working on splitting out IPv4 and IPv6 data on the matrix. Cells in the matrix are now diagonally split in two by default to show IPv4 on the left and IPv6 on the right. I've added the option to show only IPv4 data or only IPv6 data, though clicking on a cell will still take you to the graph for IPv4/IPv6 irrespective of the view you're in. I tweaked the severity colours and added a legend for them that changes based on the current test type (relative/absolute latency, loss, hop count).

More refactoring in the matrix, fixes for matrix popovers and more general IE8 fixes, particularly to get modal dialogs working. Added helpful tooltips to radio buttons for aggregation in the modals to explain what they do.

Changed the colour of event lines on smokeping graphs with only one series. The series would be drawn in black which would clash with the grey of event lines, so I set event lines to be the same colour as their event markers in this situation.

Finished up the week by adding some performance improvements to smokeping graphs that hugely improve software rendering of the canvas (particularly in Firefox on Linux). The canvas is always antialiased so rounding our drawing coordinates to nearest integers prevents unnecessary antialiasing and results in smokeping summary graphs being drawn instantly (which is mainly where our problem was). They aren't noticeably reduced in quality because the diagonal lines connecting points are still antialiased, so we benefit greatly from this (if anything I think the sharper edges look better). I also rounded the drawing coordinates on event markers and the rainbow traceroute graphs for drawing clarity, and I separated out some of the data processing that was in the smokeping drawing code which will mean that in future if we still require better performance, it would be possible to move more of the code into a separate thread.

24

Jan

2014

I got the congestion packet loss to work by separating the ovs bridges with veth links and using tc to limit the throughput. As expected you see a small amount of loss across the paths.

In the process I have found that the issue with occasional packets being miscounted by ovs in the change between one of rule to another is fairly significant. It turns out to be a fraction of a percent of all packets, but that still means thousands of packets getting miscounted. On the other hand, that level of inaccuracy will likely be buried under the packets lost due to standard tcp behaviour. You wont be able to see the exact levels of packet loss that you would expect however.

My next plan is to get the pica8 set up so I can test what openflow functions exist in picos that arent in ovs. Supposedly MPLS and groups work, which is fairly important for what I am doing. I'm not sure I am entirely confident about this though.

23

Jan

2014

Figured out how to use Bayes as a method to combine the beliefs/probabilities to obtain a final significance probability out of the results of several detectors. I had to use different values than the DS ones I had previously calculated, so I spent a while calculating and double-checking the values I needed for Bayes. After that, I did some manual calculations/testing before diving into implementing it in the eventing python script.

Also read a few other other papers regarding different methods of belief fusion, namely the Averaging and Cumulative functions. After talking to Richard, we decided to implement those functions so as to compare the values obtained by each method.

I also read some material on Fuzzy Logic, so I plan on implementing that next.

Modified the eventing script to enable easy addition of different belief fusion methods, since I plan on implementing more methods as I come across them.

21

Jan

2014

The scamper run on Planetlab has finished and the results are being downloaded.

Further updates have been carried out on the paper. This has included updates to the turnover section and the introduction.

My six monthly report was finalised and submitted.

21

Jan

2014

More work on improving the consistency and quality of JS and CSS in the AMP web interface this week. I fixed lots of minor bugs to get everything working in IE8, starting with the most basic ECMAScript dialect differences such as removing extraneous trailing commas and polyfilling some non-existent functions, then moving on to the URL handling code, which previously only took into account how URLs are rewritten in HTML5 browsers (which can write over the current URL). HTML4 browsers such as IE8 and below cannot rewrite the URL so they instead append as a hash/fragment a URI relative to the base URL (where the base URL is that of the page first directly accessed, i.e. typed into the address bar). HTML4 and 5 are both now handled by checking for the existence of a hash and resolving the fragment if it exists, otherwise resolving the base URL. I broke my work on the history code down into a few stages, first tackling the matrix, graph pages, then modal dialogs. I fixed up the rest of the history state handling code afterwards to restore state while moving forwards and backwards between pages, which hadn't been hooked up before.

Another common issue worth mentioning is that some browsers, particularly IE8 and below require explicit object.hasOwnProperty() checks when iterating through object properties with code like for (var x in y) { do things }, which would become for (var x in y) { if (y.hasOwnProperty(x) { do things } }. Instances where these checks weren't happening created several not easily detectable bugs, because IE will additionally iterate through the object's prototype (and debugging IE is a nightmare).

To finish off the week I ripped apart the matrix and removed its dependence on the rather bloated DataTables JS library, added documentation and did some much needed refactoring and tidy up, then started working on splitting up cells to show IPv4 and IPv6 stats.

21

Jan

2014

Moved the mapper 'packet loop' into the library so the user doesn't need to provide this. Instead the user now supplies a per packet function which will accept packets and messages.

Added a fast path for single threaded operation.

Started working on a native implementation of a parallel trace format, Linux socket (int:) using the socket option PACKET_FANOUT.

20

Jan

2014

Continued investigating a problem where ICMP test data was
intermittently failing to be reported. It appears to be due to fairly
aggressive timeouts stopping tests before they finish - resolving the
(quite large, and increasing) list of destinations was taking longer
than expected which was not leaving enough time to perform the
measurements. I've increased the timeouts to more reasonable values and
have a few ideas to exclude resolving time from the allowable test duration.

Fixed a couple of bugs in the tracking of active streams that meant
cached data was being recached with no new data being fetched. It's now
deployed and has shrunk the size of our queries. Also looked into adding
a query timeout to prevent long running queries from hosing the machine.

Spent a lot of time looking at explain/analyze output from postgresql,
trying to shave some time off fetching data for our graphs. Made a few
incremental improvements with a new index and some reordering of
queries, but I'm still looking for the magic bullet. We have a lot of
data and it takes a long time to read it!

20

Jan

2014

Replaced the event descriptions produced by netevmon with something a bit more human-readable. This was somewhat annoying to achieve, as it required passing a lot of extra parameters into each detector, e.g. the units that the time series is measured in, the metric itself, the scale factor for the raw data (e.g. bytes per period into mbps).

Made the amp-web graphs appear more responsive by displaying components as soon as their ajax call completes, rather than waiting for all the ajax to complete before rendering anything. In practical terms, this means the detail graph appears much sooner rather than having to wait for the query for 30 days of summary data to finish. I've also split the summary data query into multiple queries so the summary graph will now appear in increments, almost acting like a progress bar.

Tried to get netevmon deployed on skeptic, without much success so far. It seems that we can run it against a particular collection but as soon as you try to include all of the collections, the whole thing grinds to a halt and eventually prevents NNTSC from processing new data. Hopefully, we can find the cause of the problem early next week.

Fixed a bunch of other minor bugs / errors across Cuz in between times, as we try to get closer to something we can show off at NZNOG.