The first few weeks I focused on the blurb and proposal.
I found a trace file and wrote a script to output Python code which I piped into the Python terminal to save it to a SQLite database. I then queried the data in Django and graphed some information including protocol counts and a usage timeline.
Last week I discovered Shane's Libprotoident library for application layer protocol identification for flows. I thought it would be great if I could utilise this to display application information for the flows I will collect. I went to see Shane, who told me a previous students project used Libprotoident to request the type of information I would like. I will look into this over the next few weeks.
This week I have been busy with assignments. I hope to continue with the web interface early next week.
Spent this week working on other assignments to get them finished before end of teaching recess and didn't get to work on project.
Next step is still to research popular document learning algorithms and if they exist within Mallet see how well they adapt to log files, adjusting them if needed for a better fit to this text format.
Still fighting with cross compiling a workable linux kernel for RaspberryPi
Through different tutorials for compiling and running custom kernels and configuring u-boot managed to get a very unstable (laggy) system up n running.
Continued hunting for the bug in the NNTSC live exporter with mixed success. I've narrowed it down to definitely being the per-client queue that is the problem and it doesn't appear to be due to any obvious slowness inserting into the queue. Unfortunately, the problem seems to only occur once or twice a day so it takes a day before any changes or additional debugging take effect.
Went back to working on the mechanical Turk app for event detection. Finally finished a tutorial that shows most of the basic event types and how to classify them properly. Got Brendon and Brad to run through the tutorial and tweaked it according to their feedback. The biggest problem is the length of the tutorial -- it takes a decent chunk of our survey time to just run through the tutorial so I'm working on ways to speed it up a bit (as well as event classification in general). These include adding hot-keys for significance rating and using an imagemap to make the "start time" graph clickable.
Spent a decent chunk of my week trying to track down an obscure libtrace bug that affected a couple of 513 students, which would cause the threaded I/O to segfault whenever reading from the larger trace file. Replicating the bug proved quite difficult as I didn't have much info about the systems they were working with. After going through a few VMs, I eventually figured out that the bug was specific to 32 bit little-endian architectures: due to some lazy #includes, the size of an off_t was either 4 to 8 bytes between different parts of the libwandio source code which resulted in some very badly sized reads. The bug was found and fixed a bit too late for those affected students unfortunately.
Since the last report I've been reading through the 802.15.4 standard to get an idea of what to expect from silicon radios that say they are compliant.
Both Dean and Richard made the point that just because they use the 802.15.4 radios doesn't mean they follow the standard and as such I shouldn't expect to see a whole lot of structure from the standard. For example the standard is very vague when referring to a network not using PAN controller beacons which would allow the network designer to make whatever decisions they like. This is an issue when interoperability is desired as the IEEE802.15.4 standard really doesn't offer anything that would facilitate such interoperability - the application and the networking (in cases where it is not a star network with PAN controller beacons enables) are not defined by the standard.
I also attempted to trace through the source for both RIOT OS and ContikiOS. RIOT OS is a microkernel meaning that messages get sent between different processes by the kernel. As such, I was able to trace the transmission of a TCP packet through the RPL routing, through the 802.15.4 framing to a call where the frame was sent to a transceiver. ContikiOS was far more complicated as they use a lot of preprocessor macros for various things which makes tracing a little more difficult. I wasn't interested in spending a lot of time tracing when my main interest was just figuring out how the 15.4 layer worked (PAN beacons? PAN controller elections? free for all?). I actually didn't figure out how either worked, but I imagine reading through the RPL routing document would give more information.
Finally I tried to evaluate both ContikiOS and RIOT OS. I like the RIOT OS source as it is very clean and readable. For this reason I was really hoping that I could use RIOT OS for this project. However, support for the CC2538 transceiver (distinct from the microcontroller core) is not in the mainline branch. I was able to checkout a branch that claimed support (and is in the process of a pull request) however it was hard to tell if it was actually working. ContikiOS on the other hand was ready to go out of the box. My sniffer seemed to have some kind of latency in showing me received packets (or it was rejecting them because they aren't using standard frame checksums?) which made it difficult to figure out how packets were flowing. In any case I was able to create a gateway running SLIP to a tap interface as well as a web server node that I could access over the air using the gateway.
I plan to continue evaluating both options but it appears that ContikiOS is more polished than RIOT OS in terms of CC2538 support. I find it interesting that RIOT OS claims support for the CC2538DK but doesn't support the radio (yet). RIOT OS is an embedded RTOS but I don't think I would choose it over FreeRTOS unless I wanted the networking and radio stack.
Went and saw Brad and got /var/log log files from ns1 zipped up to start testing with so I can decide whether or not I need Syslog logs at the next stage since it is simpler to use the logs already available than to set up Syslog on a machine so these will be a great start.
Unzipped them and now trying to find the best way to combine them into a .mallet file. Tried to execute it on the whole folder but after a couple hours it was still going; When I have time I will leave it running for awhile because it may just take awhile to do 400MB of logs, for the moment I'll use subfolders while waiting. From the examples I went through they had all their files in the .txt format but when testing on a single folder Mallet seems to be able to decompress and read files in the plain file format.
I ran mallet on the single sub folder /var/log/kernel to create a simple topic model which worked well but didn't really have enough info to tell anything interesting so I will be looking for some bigger subsets to test on until I can get the entire log to combine.
Now I will start researching and testing different document clustering algorithms for finding patterns within these logs.
Spent most of the week getting the last few things ready to go ahead of
the test deployment next week, including diagrams of how various
components fit together.
Fixed a copy and paste error when comparing schedule items that meant
that there was a small chance of two tests being considered the same
even if they had different end times. Also updated some error handling
branches to properly free some resources that had been forgotten about,
and had a general tidy up based on feedback from a slightly newer gcc.
Generated all the certificates for the test connections, and while doing
so found and fixed a bug that could prevent multiple certificates from
being generated when listed on the command line.
Built new packages with the recent minor updates and deployed them on
one of my test amplets to verify. While watching the results I found
some interesting behaviour with tests to www.amazon.com, where the
object counts are fluctuating between two values. A quick investigation
suggests that it's not caused by any changes to the test, but I haven't
discovered what is actually going on.
An attempt has been made to polish up the Megatree data set. I few runs have been carried out with minor corrections. These have included for the stage counts, stopping the local data after the stages expire. For this it is lightly unclear whether stopping local data usage is justified, however the argument is that when the remaining traces are carried out simultaneously, local data is unavailable if it is applied after the trace in which it was collected. This could be due to the time it takes to collect and process the data to make it available.
I have also been running the IS0 Internet simulator to show optimisation of the initial TTL with the fuller data set. Another issue to consider is whether to optimise or show the affect of packet queuing delays. Perhaps this could be done for the TTL chosen as optimum using a moderate setting for packet queuing delay. The results seem to indicate that a bias for many sources towards the a small TTL really helps to make use of global stop set data when using sources windows.
Some further time is needed to get at least a few completed runs of per flow load balancer data with the few flow IDs and many collected side by side.
First to catch up on the previous weeks, I created my brief and proposal (see attached for more info).
This week my goal was to become familiar with using the tool Mallet which I will use to analyse log files.
There aren't many tutorials on Mallet but I Found some good ones on using the command line interface (CLI) and I should be able to accomplish most of what I want to do just with the CLI; I will use it for now to start my testing and will possibly move onto the java API if I need to perform something more complicated than what is capable with the CLI.
I then looked further into Mallet's java API and there aren't really any tutorials on how to use it but if I need to accomplish something more complicated a combination of the example files and docs should be sufficient to make a useful program.
The next step is to get some log files from Syslog or var/log and start converting them to Mallet's file format ".mallet" (which could be a single file or an entire directory) and then run the data through some machine learning methods.
Continued developing code to group events by common AS path segments. Managed to add an "update tree" function to the suffix tree implementation I was using and then changed it to use ASNs rather than characters to reduce the number of comparisons required. Also developed code to query NNTSC for an AS path based on the source, destination and address family for a latency event, so all of the pieces are now in place.
In testing, I found a problem where live NNTSC exporting would occasionally fall several minutes behind the data that was being inserted in the database. Because this would only happen occasionally (and usually overnight), debugging this problem has taken a very long time. Found a potential cause in a unhandled E_WOULDBLOCK on the client socket so I've fixed that and am waiting to see if that has resolved the problem.
Did some basic testing of libtrace 4 for Richard, mainly trying to build it on the various OS's that we currently support. This has created a whole bunch of extra work for him due the various ways in which pthreads are implemented on different systems. Wrote my first parallel libtrace program on Friday -- there was a bit of a learning curve but I got it working in the end.