Continued keeping an eye on the BTM monitors. Changed several connections to use the ISPs DNS server rather than relying on the modem to provide DNS, which seems to have resolved many of our DNS issues.
Spent a bit of time digging into the problem of intermittent latency results for Akamai sites. It appears that our latency tests are interfering with one another as moving one of the previously failing tests to a new offset away from the others fixed the problem for that test.
Continued working on my Event Detection webapp. Added two new modes: one where the user does the tutorial, then rates 20 pre-chosen events and one where the user rates the same events without doing the tutorial. This will hopefully give us some feedback on how useful the tutorial is and whether the time required to complete the tutorial is worth it. Also added proper user tracking, with the generation of a unique code at the end of the 'survey' that the user can enter into the Mechanical Turk to indicate they have completed the task.
Dean lent me a Segger JLink which I was able to use to start tracing down the hardfault. I was unable to do this, as the newlib library doesn't contain debug symbols.
While I could build my own, I created a work around exploting the fact that vsnprintf works fine.
The JLink is still very useful as I'm very likely to need to debug other issues later on. The XDS100v3 is pretty useless for this task. The JLink also provides a way better (faster, CLI) way to program the device so I suspect I'll even become more productive.
The next step is to try and get some reliable radio sniffing going. I need to ensure I can see 802.15.4 frames so that when I'm developing my own MAC I can make sure the frames are correctly formatted. This should be a trivial task as the CC2538DK comes with a USB dongle which there is already sniffing software for. I'm just unsure if I'll be able to do the sniffing under Linux.
tl;dr: I have UART, printf and malloc working. Now I need to start radio dev.
Unfortunately spent most of my time on assignments as of late. The good news is that they went quite well, and I haven't got any to do for the next couple weeks, so should get some good development progress done.
Have also arranged weekly meetings with Brad.
The new runs on IS0 were completed and the data were used to create graphs for my thesis. The new runs were mostly controls for the sources windows data and Traceroute and Doubletree from few sources. This latter is included simply to show results from the simulator in its native state, as a starting point for this research.
The latest runs on the Megatree simulator also completed. This was further analysis of many to few situations where each source was used at least three times. The key focus was how much extra traffic to use to find load balancers that have already been seen. Two new modes were used for this in addition to the initial standard Traceroute before MDA. These were a simple estimate of extra traffic and an analysis of previously seen diamonds with the same source node to estimate require traffic.
Further analysis of the per destination run carried out on Planetlab to count unique load balancers has been carried out. As expected the count for reduced flow IDs was lower than the full analysis, however the amount of traffic was also lower and the percentage of traces with a per destination load balancer was possible higher. This latter point possible reflects the formation of clumps in the full analysis due to per packet LBs in the diamond under investigation thus masking some diamonds out from the count. There appear to be some useful advantages to the reduced analysis if locating diamond divergence points is the question of interest.
Updated fastpath to use tables, this now requires only an extra rule per port rather than an extra rule per port for every send to controller rule. Moved to using the dpset events in ryu, rather than the direct OpenFlow events, allowing access to more information in a single event. The next step is to move this into RouteFlow. Chris Lorier mentioned this work to Josh Bailey, who is interested in it and has suggested an approach to integrating this into RouteFlow. The work so far is now on Github - https://github.com/rsanger/routeflow-fastpath.
I've started reading through and watching talks on Googles WAN SDN development, B4, and the reasons for it used to connect their data-centres around the world. Among other factors some of the main ones where to keep costs down by allowing them to run links close to 100% utilisation as opposed to the typical 30-40% WAN's are traditionally provisioned at. Supporting for better Traffic Engineering has allowed for better handling of failure scenarios and achieve overall better balance of traffic across the network compare with shortest path first routing.
After the last project meeting, we decided to ditch contiki as it does not offer much that will be useful to this project. Rather than trying to hack in a new mac controller and modify the existing mac data path, it is going to be easier to just write something from scratch. The future plan will be to add networking layers on top.
In order to facilitate a new platform, we're sticking with the CC2538DK. I've spent a bunch of time getting a linker script, start up file and some basic code running. So far I've got the LEDs, UART and LCD working fine. The next step is to try and get libc (specifically newlib) going to add support for things like printf, malloc and free. Printf is extremely useful for debugging and it makes the project much simpler to implement. Malloc will be useful for packet buffers and the like (although static memory could be used instead).
Unfortunately calling printf causes a hardfault. We don't even manage to get through newlib down to the stubs that are environment specific (the part that takes the characters from printf and passes them to the UART). Using GDB, we can examine what caused the hardfault and go about debugging from there.
I have no idea why, but I am unable to read any of the useful registers. It seems like the GDB server provided by TI (which gdb connects to) doesn't like hardfaults and somehow locks us out of some registers. This is a big problem, as I need to be able to debug the CPU in order to get anything done. I've lodged a support request with the TI forum: https://e2e.ti.com/support/wireless_connectivity/f/155/p/419436/1493827#... I don't have very high hopes as I haven't had very good support in the past.
I'm really hoping that Dean (as he has a lot of experience dealing with ARM Cortex M3 parts) is able to help me solve this. It's hard to debug without printf and without gdb!
Spent much of my week keeping an eye on BTM and dealing with new connections as they come online. Had a couple of false starts with the Wellington machine, as the management interface was up but was not allowing any inbound connections. This was finally sorted on Thursday night (turning the modem on and off again did the trick), so much of Friday was figuring out which Wellington connections were working and which were not.
A few of the BTM connections have a lot of difficulty running AMP tests to a few of the scheduled targets: AMP fails to resolve DNS properly for these targets but using dig or ping gets the right results. Did some packet captures to see what was going on: it looks like the answer record appears in the wrong section of the response and I guess libunbound doesn't deal with that too well. The problem seems to affect only connections using a specific brand of modem, so I am imagining there is some bug in the DNS cache software on the modem.
Continued tracing my NNTSC live export problem. It soon became apparent that NNTSC itself was not the problem: instead, the client was not reading data from NNTSC, causing the receive window to fill up and preventing NNTSC from sending new data. A bit of profiling suggested that the HMM detector in netevmon was potentially the problem. After disabling that detector, I was able to keep things running over the long weekend without any problems.
Fixed a libwandio bug in the LZO writer. Turns out that the "compression" can sometimes result in a larger block than the original uncompressed data, especially when doing full payload capture. In that case, you are supposed to write out the original block instead but we were mistakenly writing the compressed block.
This week I focused on the fastpath problem and have created a separate ryu controller and connected both the internal RouteFlow OpenFlow switch, dp0, and my real mininet switch to it. This has been a good way to learn the OpenFlow specification through practical use, the most popular protocol use to control SDNs. I've also had to learn ryu an OpenFlow python library for creating OF controllers and skill up on my python.
During this I've encountered some minor issues
* Originally to connect the controller and network with a fastpath in my emulatation environment I tried to use a Linux tun device however found that I could not add this to ovs. However Linux tap devices work - I suspect the lack of a MAC address on the tun was the cause of this.
* I've have run into problems trying to using MPLS labels to hold port information, I'm unsure if this is an OVS issue, something I'm doing wrong or an issue with trying to put these through a Linux tap tunnel which I use to add a fastpath link between my network and controller. Currently the solution to this is to use VLANs which seem to be working as expected and should be easy to swap back to MPLS when I figure out my problem with that.
Currently the implementation is quite naive and results in a duplicated rule for each port on the data-plane to add the correct tag which directs traffic to the controller. This should be easy to reduce down by adding an extra table which tags the port.
The next step is to integrate this with the existing RouteFlow controller and add support for inter-switch links.
Started using mallet on the files that I have collect. Tested it on the entire directory at once and it created a .mallet file quickly and a topic model in 1 hour 30 mins, I then used it on the Bearwall logs which I unzipped and took a lot longer over 3 hours which makes me believe that it ignores the zipped files.
Then I looked at the topic keys it had generated and it had stripped out all the numbers and only kept words so looking at the topics didn't really prove to show anything useful.
So next step is to look into other programs and methods that are better suited towards log files because it would be more useful to see it grouping events together from multiple files of different applications. Though that's not to say the topic modeling the Mallet is suited towards won't be helpful I will also need to look into if there is an option to retain numbers etc. and if so re-test.
Code was written to analyse the new data set containing per destination data based on seven flow IDs and many. Once the data was downloaded from PlanetLab a run to count paths containing per destination load balancers for each mode was set underway.
The revised Megatree simulation completed and I found a corrupted address file causing the results to be incorrect. The simulation was based on an estimate of extra packets required by Megatree to find already analysed load balancers without carrying out a complete pre run Traceroute analysis.
Whilst waiting for these results I have been going over sections of my draft thesis to check for correctness and logical flow.