This week has focused on the check-paths routine for validating the IS0 simulator. I recently found that this routine was getting into an endless loop, though it did not exhibit runaway memory usage. The problem was that it was looking for a next hop and replying not found, and so on. It turned out that it was not reading missing hop identifiers from the hops-dest file of hop information, and so there were gaps in some traces when read into the hash tables. Another problem was that reading the TTL value from the log file looked for an equals sign that didn't exist, and so the whole line was read as the TTL value causing the hash table to be corrupted. Finally the simulator message string was too short to print some of the longer messages. This also resulted in some errors when analysing the trace log file, as for example some addresses were truncated and were flagged as incorrect by the program.
Short week as I was on leave on Thursday and Friday.
Continued tweaking the event groups produced by netevmon. My main focus has been on ensuring that the start time for a group lines up with the start time of the earliest event in the group. When this doesn't happen, it suggests that there is an incongruity in the logic for updating events and groups based on a new observed detection. Now the problem happens rarely -- which is good from the perspective that I am making progress but it is also bad because it takes a lot longer for a bad group to occur so testing and debugging is much slower.
Spent a bit of time rewriting Yindong's python trace analysis using C++ and libflowmanager. My program was able to run much faster and use a lot less memory, which should mean that wraith won't be hosed for months while Yindong waits for his analysis to run.
Added a new API function to libtrace to strip VLAN and MPLS headers from packets. This makes the packets easier to analyse with BPF filters as you don't need to construct complicated filters to deal with the possible presence of VLAN tags that you don't care about.
Installed libtrace on the Endace probe and managed to get it happily processing packets from a virtual DAG without too much difficulty.
I've done some basic benchmarks (for loops around operations) and made some optimisations based upon that for the libraries I've been testing. All the libraries performed well enough for my purposes. The performance of the libraries did differ depending upon the task being performed, with some outperforming the others at one task and the reverse with another task. Overall I've found ROFL to be the easiest to work with and it appears to still be in active development.
I've have now got access to the OFLOPS-Turbo source. So I will be looking at that to see what changes have been made and if it is more suitable to build on top of.
I've focused my attention back to RouteFlow again and started to fix some problems. I've investigated the issue with discovering hosts (i.e. ARP). Previously we were not sending IP traffic to the controller meaning that we did not already know about a host we would drop all traffic destined to it. I investigated using broadcast (ping, arp, dst mac broadcast etc.) methods to try find all hosts proactively however this did not work as most machines were not listening. As such I concluded we still have to send unknown IP traffic to the controller so it can perform ARP. This is a DOS vector, but should be less of a concern now with fastpath.
Manged to set up static IP for the gateway device:
required a clean install of the OS and radvd (had multiple version of radvd installed leading to unusual behavior).
Adding static config the /etc/network/interfaces file, then required an ifdown/ifup to refresh the configurations of the interface. Needed to add the interface the ifstate file in order for if* to work (/run/network/ifstate added: lowpan0=lowpan0) however this file is not persistent so changes may need to be replicated each boot.
Upon reboot the static config was missing - still looking into the source of the problem (address is assigned after a ifdown/ifup on the lowpan0 interface.)
Installed bind9 as the dns services, configured local zone and have working ipv4 local name resolution, still checks outside if local cannot resolve name (kept it this way in case the gateway needs to be connected to the internet), however it does not have working ipv6 name resolution yet.
Talked to brad about getting tagged log file data and he modified bearwall to log both packets that were allowed and packets that were blocked, these are indicated in the event in the log and in the case they were blocked the reason is indicated. So now I have a days worth of tagged firewall logs (about 11,000 lines).
I then made an application to process this data and convert it into the ARFF format so that it will be compatible to work with in WEKA. Now I need to split this into separate sets for testing, parameter tuning and evaluation.
Now I will build the application with the WEKA framework to rank this data based on classification accuracy.
Got a little bit of work done this week. Have setup freeradius3 in a new deb8 VM, have bridged device to mininet test environment and can hand out DHCP leases without too much hassle. Currently, there's no VLAN tags on this otherwise it breaks, so need to investigate freeradius sitting on multiple vlans (vlan per hop) handing out DHCP to double tagged traffic (triple if you count the HOP vlan).
Also added rest api support to the controller, will build on that once I have the VLAN task sorted.
Spent some more time looking into using embedded Chromium as part of the
HTTP test. I've managed to successfully extract all of the
navigation/resource timing information from the browser after the page
has loaded which is very useful. Getting access to headers looks like it
will require implementing my own resource handlers and processing
requests manually, but should be doable. Also, I still haven't managed
to completely decouple the browser from GTK - something is still trying
to initialise it even though there is no need for it and nothing is ever
drawn to the screen.
Tidied up some more configuration parsing and parts of the main loop in
the amplet client, removing the need for a few more global variables
that were convenient at the time of writing.
Helped Brad configure a new measurement machine to be sent out.
Reconfigured some of the existing machines to swap the management ports
around so we can test them without the reporting traffic interfering.
Tried multiple setups of CoAP servers:
libcoap has been most successful so far, the example provided functional server with time functionality and worked nicely with Copper (Cu) a Firefox extension for CoAP servers.
Can configure the address (-A) to start up with - one thing that wasn't included was the necessity to add '&' to the end in order for the command to execute (possibly something within the code)
Tested with a LAN machine (/w Firefox + Copper) and the IPv4 address of the RPi on LAN
Initial Copper screenFirst view of the CoAP server from inside Copper
Selecting 'time' and clicking 'observe' produces the following result
ObservingObserving the time feature of the CoAP Server (PUSH notifications)
Running the client example program with settings "echo 1000 | sudo ./coap-client -m put -T cafe coap://192.168.1.7/time -f -"
gave the output: v:1 t:0 tkl:4 c:3 id:18275
which is just the header of the CoAP message (no payload was defined)
Also trying to 'put' into the core or async (instead of time) gives a response of "Method Not Allowed" same as trying the operations in Copper.
Having a running example is useful and next steps will be to configure the example, or write a separate script based on the example with the functionality that is required for the projects server.
ccoap is a C based implementation of a CoAP server with examples, however running anything other than the examples (there are command line versions) fails to work (no response from the server. Also required editing of /etc/services and installing cmake to compile the code.
WebIOPi is an interesting setup, has functionality for both CoAP and HTTP servers which could come in handy later on in the project. The main feature of the project is to have a web interface for viewing the GPIO ports on the board. it also is implemented as a daemon service, which is part of what is wanted for the project server.
I've continued evaluating libraries.
I run into issues with padding using loxi, turns out this needs to be done manually and in the correct order to ensure fields are not overwritten etc.
I looked at using OFConnect again this seems to perform similar functionality to fluidbase, helps establish a OF connection. However does not include functions to build OF messages, given I'm happy with fluidbase and that there are a few bugs listed with OFConnect I've decided to skip it in favour of Revised OpenFlow Library (ROFL).
ROFL is a C++ library and supports fewer OF versions than loxi, however supports 1.0 and 1.3. The same flow mod is used for both allowing for quite nice support between versions.
I've also had a quick play with floodlight's oftest. A python library that tests OpenFlow switches capabilities and conformance to the OF standards. It's similar to OFLOPS however appears to be targeting testing capabilities rather than benchmarking performance. I.e. the tests included typically add a rule and send a packet that will match it and verify the packet is returned modified correctly. I needed to make some slight modifications to get it working such as removing, including adding some sleep to work around what I'm assuming is incorrect barrier behaviour (response before TCAM update) and some removed some specific match fields as it creates very specific matches matching every field in the packet to be sent.
Brad got the Brocade working, I've done some quick testing it appears to work but has limited support for features and lacks VLAN support. However it would still be useful to test packet in rates upon.
I have written a chapter on efficient discovery of load balancer diamond divergence points at the expense of internal diamond nodes. This used a restricted set of flow IDs. There is now also a corresponding section in the discussion chapter.
I also wrote a chapter on improving the efficiency of MDA analysis by changing the way in which the source port flow ID is chosen. The three methods were incrementing, random and sequential bit flipping. Sequential bit flipping did not appear to offer any advantage in discovering the successor set of a load balancer sooner. However it was hard to know what to expect as there are likely to be a number of different hashing algorithms used by different types of routers.