Woops. Missed a report last week...
Last week I wrote the radio driver which is able to send and receive 802.15.4 data packets. The Contiki software was useful to see which registers I needed to initialise to get data moving. There's plenty left to do but it's enough to get running.
This week I have started writing the MAC layer. I want to start with the PAN coordinator, specifically with beaconing. Once I have beacons, I can start work on the three scan types - active, passive and energy detection. This will allow the PAN coordinator to survey the site before starting a network.
Once I have the surveys and beacons working, I can get association working.
This week saw some good progress. I can successfully push rules to the switch and have traffic correctly flow through to devices. This saw the end of what my current testing enrivonment which so far was some simple mininet hosts sitting there waiting for something to happen.
I've spun up a new VM which has a tunnel between itself and the mininet VM, which I'm attaching to the current mininet topology in order to have more control over the host running dhcp, since we'll need to run OVS in front of this to deal with the triple VLAN tags according to Brad.
Currently this tunnel isn't working, which will be the next thing to fix. Once I have this running, spinning up a dhcpd and getting dhcp traffic back to the host asking for an address should be simple and said host should get an address.
Edit: I've since got this working today, error in my script in which interface was selected for the tap. Home time now.
Thursday was my first day back after my break, so spent some time
catching up on things that had happened while I was away.
Shane and Brad had found some unusual data being reported, so I looked
into that and updated schedules to help solve some of the problems. Also
exposed some more tuning knobs so that we can change inter-packet delay
when sending probes (we were sending too fast in some cases) and merged
in some fixes that Shane had written.
Built some new Debian packages with these changes and pushed them out,
which appears to have immediately improved the quality of the data we
I have been monitoring the latest black hole detector run on PlanetLab.
More changes based on Richards criticisms, to my thesis have been made. In particular I have added structure to the background chapter. This included finding some references relevant to the search for non 5-tuple fields that cause load balancing to occur. I found one paper that said they would look into this area in the future, thus helping to justify the research area.
Continued keeping an eye on BTM until Brendon got back on Thursday. Briefed Brendon on all the problems we had noticed and what we thought was required to fix them.
Finished up my event detection webapp. Started experimenting with automating the running event detectors with a range of different parameter options. The first detector I'm looking at (Plateau) has about 15,000 different parameter combinations that I would like to try, so I'm going to have to be pretty smart about recognising events as being the same across different runs.
Started adding worker threads to anomaly_ts so that we can be more parallel. Each stream will be hashed to a consistent worker thread so that measurements will always be evaluated in order, but I still have to consider the impact of the resulting events not being in strict chronological order across all streams.
Researched into document machine learning algorithms for processing small documents. Since social media is becoming ever popular there is a lot of work going into how to learn useful things from posts on social media, for example learning things about an event being posted on Twitter. I was especially interested in what techniques were used for Twitter since a message cannot be more than 140 characters, is posted by a user and has an interesting makeup of #hashtags etc. When we look at a log entry it is sort of similar in structure i.e. each event is a short line, is made by one specific program and has port numbers and IP addresses.
Some of the studies found better results by aggregating the "tweets" into one document which is already done with log files, others try and address the short nature of log files by looking at biterm pairs and many other techniques. Mostly the algorithms are based off topic modeling which infers topic's (groups of words) that could possibly generate the document, though I did find other clustering algorithms like spherical K-Means which I will look into further.
Looked into Mallet's API further and looked into how the importer works that creates the .mallet file that gets passed as input. So I was able to change the regex to parse tokens to include IP adresses and other numbers etc. and found it coverts the token strings to integers for storage. Then after getting the IP addresses etc. included in the input I tried it with topic modeling but it failed and the output was all weird characters so I need to find out what effect the numeric and punctuation characters have on both the input generation and modeling steps.
Started making progress on the actual code behind the controller. So far I have it running, getting packets in and reading said packets. Ran into a bit of delay in figuring out how I'll create the rules since I started off by trying to create rules for DHCP communication, but since I don't know what ports on datapaths it's connected to to begin with, I need to learn them once I've started.
My initial thoughts are that I know the MAC/IP of my infrastructure that connects to the core switch, so I can create ARP packets, send them out the flood port and wait til I get a reply and handle them from there. Once I've done this, I can create the flows to allow DHCP traffic to get where it needs to go.
The next part is how does my controller know when a client is allowed to have internet access (Authorisation of AAA). I assume that one a client tries to talk to the WAN router, that information of where the WAN router lies has been given to them, so should be allowed to have access to it. The only problem here is that one could just know the configuration, set themselves up manually and go for it. I think this situation is okay for now, since this is less of a priority of getting things working in the first place.
Rebuilt the bluetooth-next kernel with clean install of raspbian, still has input lag problems and breaks static IP setup (could be the keyboard).
Found another branch working with 802.15.4 implementation (linux-wpan-next) but negligible difference to the bluetooth-next branch, swapping to linux-wpan-next for kernel building as the openlabs blog quotes it as the one to use.
Source code has SoftMAC implemented with only sending a receiving implemented (basic functionality).
Nothing of HardMAC is implemented yet and no drivers present.
Working on setting up 2 rpi with same pan_id and channel then trying to exchange packets manually.
This week and last I implemented a few basic filters, configured the time series graph scales and added a table displaying the most prominent flows and the associated device.
Turns out pcap doesn't store direction information so I couldn't implement the ingress/egress filter. pcap-ng supports direction though Libtrace doesn't support this format yet. Should be easy enough to implement when I have the information available.
Recently I have been planning my presentation which is this Wednesday.
Plan on seeing Brad either this week or next week to get NetFlow configured so I can start working with real data.
Now that the per destination run is complete I started another run of black hole detector on Planetlab.
In response to the criticisms and suggestions from Richard I have been updating the background and related work chapters of my thesis.
In addition the IS0 runs, per destination and Megatree runs have resulted in updated graphs for my thesis. It is also means that I can write more in the results and discussion sections for these chapters.