Updated the HTTP test to use a particular source address or interface if
specified. Though libcurl has options to set one of these, it doesn't
work well in the case where you need to set both an IPv4 and IPv6 source
address, before knowing what the target name resolves to. Luckily it has
a callback to completely replace the call to socket(), after name
resolution, so I can create my own socket and bind to the appropriate
Had similar problems while updating the throughput test to use a
particular interface - need to listen on both address families and then
once the control connection happens, make sure the test connection is on
the same interface.
Updated the DNS test to query the local servers listed in
/etc/resolv.conf by default if no targets are given. This works fine for
the standalone test, but it's not quite clear the best way to schedule a
test like this when it may get merged with others that do have destinations.
Added a few new unit tests for the DNS test coding/encoding of names and
fixed a few things that unit testing, valgrind and different versions of
Spent some time looking at paris traceroute and the AMP traceroute test.
Turns out that our traceroute already keeps the important header fields
stable during a test run and so behaves like paris. Confirmed this with
the fakeroute tool used to test paris traceroute.
Finished updating NNTSC to deal with traceroute data. The new QueryBuilder code should make query construction a bit less convoluted within the NNTSC dbselect module. Everything seems to work OK in basic testing, so it's now just a matter of migrating over one of our production setups and seeing what breaks.
Continued working through the events on amp.wand.net.nz, looking at events for streams that fall in the 25-100ms and the 300+ms ranges. Results still look very promising overall. Tried to fix another common source of insignificant events (namely a single very large spike that moves our mean so much that subsequent "normal" measurements are treated as slightly abnormal due to their distance from the new mean) but without any tangible success.
Moved libtrace and libprotoident from svn to git and put the repositories up on github. This should make the projects more accessible, particularly to the increasing number of people who want to add support for various formats and protocols. It should also make life easier for me when it comes to pushing out bug fixes to people having specific problems and merging in code contributed by our users.
Wrote and handed in my honours proposal on Friday(/Saturday), in the process doing a lot more research into the 6LoWPAN/RPL/CoAP stack and compiling reading material for future reference. I wrote my proposal in LaTeX also, which is new to me -- I figured it would be useful to learn now in order to use efficiently later.
My project is fairly open ended right now (evaluating the state of the art of 802.15.4 CoAP security), as nailing down goals is still going to involve further research. I found one paper in particular titled "Security of CoAP-based End-to-End Scenarios for the Internet of Things" (a master's thesis) that I think could be very relevant and perhaps help narrow my focus, but the thesis doesn't appear to be available online anywhere. Now that I've finished my initial proposal I may look at contacting the author.
The source code for both our libtrace and libprotoident libraries is now available on GitHub. Developers can freely clone these projects and make their own modifications or additions to the source code, while keeping up with any changes that we make between releases.
We're also more than happy to consider pull requests for code that adds useful features or support for new protocols / trace formats to our libraries.
Look out for more of our open-source projects to make their way onto GitHub soon!
My week has been split into two main projects, the first was continuing to work on my graph theory problems. In the end I decided since it the requirement of minimising flows is a specifically openflow problem it could be worth spending a bit of effort on, so I believe I have sorted out all the kinks in that and have just started implementing it again.
The other part was helping Brad set up the REANNZ SDN. We have it all connected now, and it passes packets, but we met a few snafus. The main one being that when you try to use the "normal" forwarding action the prontos seems to ignore the vlan for the first packet. So with 2 prontos along the path we were losing 4 pings (and 4 arp messages in each direction) every time we started over. The only solution to this was to either not use vlans or not use the normal action.
Brad also plugged things into the wrong ports and lots of nonsense like that. He might tell you it wasnt his fault, but he never writes weekly reports...
Got the new throughput test running both as a single binary, standalone
test in the style of iperf and as part of the amplet2 server. It no
longer blocks when the remote end doesn't like our SSL credentials as we
no longer expect them to follow the proper shutdown protocol, and I now
correctly check success of SSL reads and writes.
Wrote an initial attempt at a python throughput data extractor to use in
nntsc, but it is currently missing the rest of the chain (dataparser,
database tables, etc).
Spent a bit of time trying to polish small areas of documentation and
unit tests while waiting for throughput tests to run. Made basic
manpages and started work on adding a few more tests to check code
A longer run to find black holes in load balancers using our version of fast mapping has been carried out. Instead of 1000 addresses, two lots of 5000 were used. In each case there was an initial traceroute MDA run followed by eight Paris traceroute runs and in turn a final MDA run. The resulting warts files were scanned using the upgraded analysis program which when it finds a short Paris trace, compares the initial and final MDA runs for route changes and checks the Paris end point to see if it is within a load balancer. Some special cases which are flagged are when the destination is inside a load balancer itself and when the Paris stop point is at the convergence point of a load balancer. These special cases explained the possible black holes found in the 1000 destination data, however some true appearance black holes in LBs were found in the 5000 destination data. It may be possible to continue this on planetlab however it is probably necessary to use ICMP probes, and this would reduce the number of per flow load balancers seen, requiring more probing to see the same number of load balancers.
Initial runs were carried out using valgrind callgrind. It took me a while to realise that I could not use my Internet Simulator bash script as callgrind appeared to analyse the bash shell in this case. When I ran the program directly I still couldn't start with no instrumentation and then manually activate it. When I did this there were no events collected. This seems to be a callgrind bug. Another possibility is to try the programmatic activation of callgrind instrumentation by recompiling the simulator with an activation macro included. It turns out that making memory map nodes in the build method starts almost immediately, and I don't know how long it takes before it gets to making links. Running valgrind without a delayed start can help answer this. The objective is still to delay for a day and then run instrumentation for two days.
Last week was a bit slow, spent ages double-checking the values in the spreadsheet and making sure the correct ell ranges and formulas were being used. Found quite a few mistakes, so it wasn't entirely a waste of time. Then, spent the rest of the week updating the detector probabilities script with the new values. This took forever too since there are several detectors and two methods, each with their own probability values and event grouping, etc.
Started to work (again) on my honours project which is a continuation of my summer research work. Submitted a blurb about my project and re-read through a couple of the academic papers I had originally looked at to revise where to take the project next.
Spent about half of my week continuing to validate netevmon events on amp.wand.net.nz. After noticing that the TEntropy detectors were tending to alert on pairs of single measurement spikes that were 2-3 minutes apart, I modified the detectors to require a minimum number of measurements contributing entropy (4) before triggering an alert (provided the time series was in a "constant" state). This seems to have removed many insignificant events without affecting the detection of major events, except for some cases where the TEntropy detectors might trigger a little later than they had previously.
Started implementing the new traceroute table schema within NNTSC. Because there are two table involved (one for paths and one for test results), it is a bit more complicated than the ICMP and DNS tables. Having to cast a list of IP addresses into an SQL array whenever we want to insert into the path table just makes matters worse. At this stage, I've got inserts working sensibly and am now working on making sure we can query the data. As part of this, I am trying to streamline how we construct our queries so that it's easier for us to keep track of all the query components and parameters and keep them in the correct order.