Spent much of my week keeping an eye on BTM and dealing with new connections as they come online. Had a couple of false starts with the Wellington machine, as the management interface was up but was not allowing any inbound connections. This was finally sorted on Thursday night (turning the modem on and off again did the trick), so much of Friday was figuring out which Wellington connections were working and which were not.
A few of the BTM connections have a lot of difficulty running AMP tests to a few of the scheduled targets: AMP fails to resolve DNS properly for these targets but using dig or ping gets the right results. Did some packet captures to see what was going on: it looks like the answer record appears in the wrong section of the response and I guess libunbound doesn't deal with that too well. The problem seems to affect only connections using a specific brand of modem, so I am imagining there is some bug in the DNS cache software on the modem.
Continued tracing my NNTSC live export problem. It soon became apparent that NNTSC itself was not the problem: instead, the client was not reading data from NNTSC, causing the receive window to fill up and preventing NNTSC from sending new data. A bit of profiling suggested that the HMM detector in netevmon was potentially the problem. After disabling that detector, I was able to keep things running over the long weekend without any problems.
Fixed a libwandio bug in the LZO writer. Turns out that the "compression" can sometimes result in a larger block than the original uncompressed data, especially when doing full payload capture. In that case, you are supposed to write out the original block instead but we were mistakenly writing the compressed block.
This week I focused on the fastpath problem and have created a separate ryu controller and connected both the internal RouteFlow OpenFlow switch, dp0, and my real mininet switch to it. This has been a good way to learn the OpenFlow specification through practical use, the most popular protocol use to control SDNs. I've also had to learn ryu an OpenFlow python library for creating OF controllers and skill up on my python.
During this I've encountered some minor issues
* Originally to connect the controller and network with a fastpath in my emulatation environment I tried to use a Linux tun device however found that I could not add this to ovs. However Linux tap devices work - I suspect the lack of a MAC address on the tun was the cause of this.
* I've have run into problems trying to using MPLS labels to hold port information, I'm unsure if this is an OVS issue, something I'm doing wrong or an issue with trying to put these through a Linux tap tunnel which I use to add a fastpath link between my network and controller. Currently the solution to this is to use VLANs which seem to be working as expected and should be easy to swap back to MPLS when I figure out my problem with that.
Currently the implementation is quite naive and results in a duplicated rule for each port on the data-plane to add the correct tag which directs traffic to the controller. This should be easy to reduce down by adding an extra table which tags the port.
The next step is to integrate this with the existing RouteFlow controller and add support for inter-switch links.
Started using mallet on the files that I have collect. Tested it on the entire directory at once and it created a .mallet file quickly and a topic model in 1 hour 30 mins, I then used it on the Bearwall logs which I unzipped and took a lot longer over 3 hours which makes me believe that it ignores the zipped files.
Then I looked at the topic keys it had generated and it had stripped out all the numbers and only kept words so looking at the topics didn't really prove to show anything useful.
So next step is to look into other programs and methods that are better suited towards log files because it would be more useful to see it grouping events together from multiple files of different applications. Though that's not to say the topic modeling the Mallet is suited towards won't be helpful I will also need to look into if there is an option to retain numbers etc. and if so re-test.
Code was written to analyse the new data set containing per destination data based on seven flow IDs and many. Once the data was downloaded from PlanetLab a run to count paths containing per destination load balancers for each mode was set underway.
The revised Megatree simulation completed and I found a corrupted address file causing the results to be incorrect. The simulation was based on an estimate of extra packets required by Megatree to find already analysed load balancers without carrying out a complete pre run Traceroute analysis.
Whilst waiting for these results I have been going over sections of my draft thesis to check for correctness and logical flow.
Met with Brad and Christopher from REANNZ and discussed an implementation for me to work on.
Basically we'll have an OpenFlow switch sitting in the middle which would be connected our core router and the HOP, which will also be talking to the controller and a linux box running our DHCP server.
When we get DHCP traffic from the HOP, we pop a VLAN tag onto it which would then be passed to the DHCP server. This would handle the request, have the S/C VIDs in tact and be able to handle what is AAA.
Any replies from the DHCP server will have the VLAN tag we set taken off, and then thrown at the HOP. Since the S/C VIDs are preserved, this would just work.
Once we have the lease given to the customer, we install rules that allow internet access. Depending on the type of customer, we would need more/less rules depending on how much we care about the accounting of traffic.
Also need to figure out way for the controller to know when the customer has been allowed access and is authenticated, so it can install rules to facilitate this.
As for what I need to get done first, I want to get DHCP traffic coming in and then being passed to a VM in the way I mentioned. Running tcpdump on the VM will show that traffic is reaching it, which would indicate a success.
Much of my week was taken up with matter relating to the Wynyard meeting on Wednesday. Meeting itself went reasonably well and definitely got the impression there was some interest in what we do and how we do it.
Continued marking the libtrace assignment for 513. Just a handful more to go.
Started getting familiar with the new AMP deployment, so I am better able to keep an eye on it while Brendon and Brad are away. Had a few connections come online on Friday which required a little attention, but overall I think it is still running smoothly.
Short week due to the Easter break.
Prepared an extended version of my latency event detection talk to give to Wynyard Group next week. It'll be nice to not be under so much time pressure when giving the talk this time around :)
Started marking the 513 libtrace assignment.
The live exporting bug in NNTSC remains unsolved. I've narrowed it down to the internal client queue not being read from for a decent chunk of time, but am not yet sure what the client thread is doing instead of reading from the queue.
Configured the third throughput test target and updated the test
schedule to properly include all three throughput test targets. Went
through all the results to make sure that all are reporting - found and
fixed a couple where incorrect HTTP targets had been set and redirects
were happening. Double checked that some unusual throughput results were
correct (they appear to be).
Spent some time investigating some connections that appeared to up, but
wouldn't forward my data. The modem seems to think everything is fine
and there is nothing obviously wrong at my end, so I've asked for it to
Officially started my PhD this month, 1st April. As such I've switched from focusing on libtrace to software defined networking. With short weeks due to Easter, this report is covering the first two weeks of April.
I've started reading over some previous research of WAND students and other useful resources that Brad has directed me to.
With Brad's assistance I've also setup a virtual machine running a controller, running RouteFlow --- essentially a Vandervecken deployment. This connects to a mininet VM where an Open vSwitch is acting as the OpenFlow controllable device.
I've meet with Richard Nelson and he's suggested adding 'fastpath' to RouteFlow as a first project to get a feel for SDN development and break up the task of researching literature.
Packets that need to be processed by the controller are currently sent via a OpenFlow PACKET-IN message and if necessary returned to the via a PACKET-OUT message. There is a lot of overhead involved in this process, including adding extra meta data to a packet. Fastpath aims to eliminate this by using the data plane to forward packets to the controller quickly by adding OpenFlow rules. One difficulty is attaching the meta-data, i.e. the switch and port number.
I've also meet with Bill Rogers. He wanted to see some good specific use cases for SDN, particularly those which cannot be solved by the current technologies rather than the argument SDN gives people to flexibility to do whatever they want. Also while SDN can make many things easier many problems can be solved by some combination of current technologies.
The Internet simulator IS0 was tasked to do some strategic simulations. One of the angles is varying control packet queuing time for the sources window setting of 500. There are also a couple of control runs for some different initial TTL settings.
An attempt has been made to reduce the amount of extra probing required by the Megatree simulator. Currently a full simple Traceroute run is required to determine if a Traceroute contains a divergence convergence point pair that has been seen before. This is being reduced to do some extra probing when a previously seen divergence point is encountered in the main MDA trace.
I have printed my entire thesis draft and have started proof reading it from the beginning. This is an attempt to ensure that a well rounded story is told with good logical flow and appropriate discussion.