The event based simulator ran successfully using 20 traces per autonomous system, rather than one. It took quite a bit longer to run with this change. I have now set up a script to run the rest of the simulations at this setting. That has now been running for four days. There should only be one memory map build necessary as the Doubletree one it generated in the test was good and it only needs one more for Traceroute. The previous simulations took two minutes once built, so these might take something like 20 times longer or a bit more.
I have started to work out the structure of the thesis, and writing the introduction and background. The references automatically cite the authors and year in the text which I find much more cumbersome than numbered references. Adding so much extra text breaks up the flow of the document to the reader, so I hope that I can change the style.
I think that the students PhD conference went quite well, but I will have to wait and see what feedback I get.
Finished translating the mode detection over to C++ and managed to get it producing the same results as my original python prototype. Started running it against all of our AMP latency streams which was mostly successful but it looks like there are one or two very rare edge cases that can cause it to fall over entirely. Unfortunately, the problems are difficult to replicate, especially as the failures can occur at a point where I have no idea which time series I'm looking at, so debugging looks like it might be painful.
Wrote a new detector that uses the modes reported by my new code to identify mode changes or the appearance of new modes. It would possibly be more effective if the mode detection was performed more often (currently I look for new modes every 10 minutes), but I'm concerned about the performance impact of doing it more frequently.
Started investigating other potential anomaly detection methods. Had a look at Twitter's recent breakout detection R module, but it didn't perform very well with our latency data. Found another changepoint module in R which appears to work much better, so will start looking at developing our own version of this algorithm.
This week, I focused on using the magnitude of a change as another indication of severity because the results of using AMP-specific probabilities were not as favourable as we expected (this can be explained by the fact that DS does not work on events where only a single detector has fired). Spent several hours looking for relevant literature, but the papers I looked at were not especially helpful or completely unrelated. I took a break from reading papers and moved on to graphing the relationship between magnitude and severity. I used the Plateau detector's absolute and relative latency changes and the TEntropy-Stddev detector's TEntropyChange as metrics and plotted them against the severity score of the matched ground truth group. Found that TEntropyChange was useless for this purpose, but the absolute latency change showed promise: there was an easily identifiable threshold for identifying significant events but there were also some outliers that were less desirable (insignificant events with a high magnitude of change).
The non event based simulator runs of Doubletree completed and the results were converted into graphs. These were added to the PhD conference slides. These results are sensible and suggest that Doubletree may be useful in the many sources to few destinations case.
While this simulator was running it wasn't clear if further efficiency was required, so I borrowed the hash processing code from IS0 and ran it in a test program in preparation to replace the arrays used for stop set information in my current analysis. I may not need to do this now as the simulator runs finished on time.
The event based simulator IS0 runs that used 19000 ASes with one trace per AS also had their results graphed and added to the PhD conference slides. An initial investigation into whether IS0 can process 180000 traces using the same resources is also being carried out, with an initial run underway.
A new run of the black hole detector was initiated and the latest results were processed.
Continued investigating why traceroute tests were sometimes lingering
when the main amplet2 process was terminated. Eventually discovered that
I wasn't closing some file descriptors after forking, so that the test
children were able to connect to a listening local unix socket that
should have been closed. Despite listening, no running process was
actually expecting this connection, so it stalled waiting for it to be
Also tidied up more of the ASN socket querying code to better detect if
it had closed, and to actually report the error back so that it could be
dealt with in a smarter way, helping prevent the test hanging around in
a bad state.
Had a quick look at the HTTP test after seeing a few unusual results and
found that some software does a poor job of following the standards
(surprise!). Updated the header parser to be slightly smarter and deal
with some different combinations of capital letters, whitespace and
Spent some time working with Brad to get an example amplet machine
running that he can use to work through the upgrade process, bringing
them up to date with Debian.
Continued the painful process of migrating my python prototype for mode detection over to C++ for inclusion in netevmon. Managed to get the embedded R portion working correctly, which should be the trickiest part.
Spent a bit of time with our new libtrace testbed, getting the DAG 7.5G2s configured and capturing correctly. Ran into some problems getting the card to steer packets captured on each interface into separate stream buffers, as the firmware we are currently running doesn't appear to support steering.
Version control was added to the latest configuration of the event based Doubletree simulator. This simulator then had a sources windows mode added. This is when traces are executed in blocks and the up and coming block has control information sent to it just before it starts probing. This creates an economy of probe and control packets. This was achieved in the simulator, as huge savings over the basic mode of operation were achieved.
A bug in the non event based Doubletree simulator was corrected. It is now necessary to rerun these simulations. Information about where the program is up to is now printed out in the std error stream allowing one to know where it is up to. The simulations are taking a bit longer than before but seem to be running correctly.
The PhD conference talk was shortened to only include Megatree, as Doubletree is being rerun, and there is only room for one analysis in the time allocated.
Spent the first couple of days rewriting/restructuring the eventing script since it was a real abomination of a script (atleast the functions had been well documented/named so it was not too painful). Also rewrote the probabilities script so that each time series subtype (e.g. AMP ICMP/rrd Smokeping) would be a separate module and have its own sets of probabilities. This also makes it easier to add new modules later on. Using the AMP-specific probabilties, I re-ran anomalyts using the original series used for the ground truth and got a list of event groups and their significance ratings. Then, I attempted to match the output produced by the eventing script (i.e. event groups and their significance probability) and the original manually classified ground truth. In theory, most of the detectors' behaviour should have been very similar to those found from the ground truth since they are using the exact same latency values, but for some reason there were missing/additional events. This was expected behaviour for the Changepoint/HMM Detectors, but there were some differences with detectors that relied on the Plateau algorithm (Plateau, TEntropy-Stddev, and TEntropy-Meandiff detectors). Spent the remainder of the week comparing events from the two sources and flagging those that needed to be investigated.
Spent some time building new amplet2 Debian packages to make sure that
the build process was up to date with any new dependencies added with
the recent changes. Had to deal with a few packages in Debian Lenny
being well out of date and missing features (though an upgrade is on the
Installed new packages on a test amplet, and configured the schedule
using the web interface. In doing so, found a few test options that
weren't properly hooked up and were setting the wrong values, and that
sites were including themselves in their test schedules.
Accidentally left some firewall rules in place while testing and found
some broken behaviour when parts of tests failed. Watchdog timers
weren't being removed if the test exited badly, which was leading to
extraneous messages reporting tests being killed (when they had already
stopped). Broken connections to the ASN server could also trigger a
SIGPIPE when querying the local cache, which weren't being properly
Spent the latter part of the week reading student honours reports.
I found a bug in the non event based Doubletree simulator. I had noticed that the packet counts for the local stop set only case were lower than expected. After fixing the code I set several reruns going. This will affect the design of my PhD student conference slides depending on when the new results are ready. If it takes too long I can present the Megatree results instead of Doubletree.
I processed some more black hole detector results.
I designed a structural layout for my thesis. This included chapter titles and the problem addressed.