Another week in presentation mode.
The demonstration itself is now working much more reliably. Dean was a big help here as I'm leveraging some features of ubiquiOS (Bluetooth Smart). I'll keep the exact details a secret for this coming Tuesday!
I've also re-done all of my slides and diagrams. I tried to make the presentation more like a story where I introduce a problem, some reasons why existing technology doesn't solve that problem and then discuss my solution. I discuss at a high level to cater to a broader audience before I start drilling down into the architecture and the problems I had along the way.
I still need to write some presentation notes, take some photos (in case the demo doesn't work for some reason), and also draw some more technical diagrams to cover specific technical questions I anticipate.
I've had the system running for the past 24 hours collecting sensor data for the presentation!
I responded to critique of the chapters on "efficient analysis of per destination load balancers" and "load balancer turnover in the Internet". In the main, there were improvements in precision and detail to be made.
Converted the HTTP test to report data using protocol buffers.
Wrote a simple unit test for the DNS test to check that the data coming
out was the same as the data going in, and did some testing with NNTSC
to make sure that the data was in the appropriate format to be inserted
into the database. Found and fixed a few errors where things weren't
being set appropriately.
Spent some time looking into the slow HTTP test data I have been
collecting. Around 60% of the objects that were slow to fetch were on
new connections (usually the first one where we fetch the initial HTML)
and the delay was normally between sending the request and receiving any
response bytes. However there are enough delays in different places and
for different reasons that there is no obvious single cause, more
investigation is required.
I moved my focus back to oflops this week. I've worked through converting the packet_out test and creating a packet_in and out test. As part of this I fixed the handling of timer events adding basic thread safety and allowing more recent timers to fire when another is waiting.
I've also added some barriers into the tests so that they will only start traffic generation once the switch reports the rules are ready. Despite this, I'm likely to still rely on a timeout since some switches seem to treat barriers like echoes. I've been pushing all these oflops changes into my github this week, including earlier ones now I'm relatively happy there will not be any more significant changes.
Josh is now working merging Chris's libnl and my fastpath changes into upstream RouteFlow. I think he is planning on releasing a new Vandervecken ISO.
Brad managed to supply me with some IPFIX data. So now the only thing that I am missing is Application/layer 4 information. I may just make some up for development and demonstration purposes.
The application currently displays the top downloaders, uploaders, and top domains. Each device can be filtered to discover the specific details for that device. Next I will add a comparison of bytes and flows, which would be a comparison of a user's exchanged data and the number of places they communicated with. I will also add some (made up...) application statistics.
Started examining connections between multiple files. In other words if an events happens in one file if we look into other files around the same time are there any relations?
Firstly to be able to check a time window all the log file timestamps needed to be standardised so I made a script to process each timestamp and convert it into a unix timestamp. This allows for easy comparisons by just having to subtract one timestamp fromt he other to find the difference in seconds between two events.
Once I had usable timestamps I created a program to compare two files against one another. It goes through event by event in the first file and then goes through the second file to find events within a sixty second time window of one anther. Once an event is found a similarity between the two events is calculated and tokens for matches and stored.
Alot of the lines need to be stripped of character like: =, [, < etc. so that any useful information can be compared. Have also found that since alot of the IP addresses are 130.217 there are alot of matches on just that so it may be useful to implement a blacklist for that network prefix because IP addresses can be are assigned more weight than normal words.
Made a video demonstrating BSOD with the current University capture point. The final cut can be seen at https://www.youtube.com/watch?v=kJlDY0XvbA4
Alistair King got in touch and requested that libwandio be separated from libtrace so that he can release projects that use libwandio without having libtrace as a dependency as well. With his help, this was pretty straightforward so now libwandio has a separate download page on the WAND website.
Continued my investigation into optimal Plateau detector parameters. Used my web-app to classify ~230 new events in a morning (less than 5 of which qualified as significant) and merged those results back into my original ground truth. Re-ran the analysis comparing the results for each parameter configuration against the updated ground truth. I've now got an "optimal" set of parameters, although the optimal parameters still only achieve 55% precision and 60% recall.
Poked around at some more unknown flows while waiting for the Plateau analysis to run. Managed to identify some new BitTorrent and eMule clients and also added two new protocols: BDMP and Trion games.
This week wasn't too productive for my honours as far as progress goes, spent most of my time dealing with other papers.
I have gotten around to fixing my test environment, and that is sort of working. I'm running into an issue where freeradius won't send packets back out after making a reply. I initially thought this was to do with it trying to be clever and sending replies base on client-hardware-address instead of the source MAC (which I'm rewriting), but it's not, instead, no packets are leaving the VM. I'll address that early next week.
Coming back to the MAC rewriting, that is working properly now after ironing a couple of mistakes I made in my code I found during testing.
Still need to design a way of getting client traffic to the router in it's own VLAN pair (for segregation), and then back again. The latter is the trickier part, I think.
This week I focused on my presentation, and then secure provisioning.
The presentation feedback so far can be grouped into two categories:
* Relying too much on domain knowledge:
- MAC, PHY, CSMA
- Heavily technical explainations of 802.11i RSNA (perticularly the key derivation)
- Too many acronyms!
* Not covering the problem and my own solution enough
- The jump from IoT to IEEE 802.15.4 was too abrupt
- What about Wi-Fi and Bluetooth?
- How much work have I actually done?
- Does it work?
- Nobody else has creating an open source implementation that uses beacons. This is important!
Andreas helped me out hugely by sitting down with me to refactor my slides. We came up with something very quickly for the next day's presentation that was much better than the first. I still have some more feedback I need to fold in before Wednesday next week.
I also spent some time trying to get a demo going. I want to demonstrate what is possible with a convenient provisioning process. I've written an Android app that will scan QR codes and pull a MAC address and PSK from it. I have also added Bluetooth Smart (BLE) to my coordinator application to accept BLE connections. The aim being to transfer the MAC and PSK to the coordinator (which can then push the PSK into the security supplicant).
tl;dr. Scan QR code on network node with Android. Bluetooth used to transfer PSK to coordinator. New device can then associate and authenticate with coordinator. Profit.
Converted the DNS, TCPPing, traceroute and throughput tests to report
data using protocol buffers, and updated the scripts used by nntsc to
extract/parse the report messages. Updated the build system to
automatically build all the appropriate files from the .proto definition
Wrote some unit tests to make sure that the data being put into the
protocol buffers was the same as the data coming out and that optional
fields were appropriately present/absent.
Started collecting some more data on slow HTTP tests, dumping full
result data to try to see if there are any patterns around what objects
are slow to fetch, and which part of the transfer process is slow.