Finished prep for AIMS and hackathon. Gave a practice run of both of my talks on Tuesday and got some useful feedback. Incorporated this into the talks on Wednesday.
Flew out to San Diego on Thursday and had a quiet day to rest and recover on Friday before the hackathon on the weekend.
Have finished the implementation using InfluxDB, and have spent some time ironing out bugs. Influx Conitinuous Queries only automatically query the last couple of bins of data in real time, so when inputting old data it needs to be explicitly rolled up. Have created code to do this.
Also did a bit of a hack to get the smoke arrays behaving a bit more sensibly when there are less than 20 observations per bin. Doesn't make sense to take 20 percentiles of 3 results, so needed to just return raw results if there were less than 20 of them. Currently doing this by still rolling up the percentiles and then taking only as many percentiles as we have results. Seems to be working, but not an ideal implementation. Would be good to do this at database level but can't with InfluxDB.
Will walk Brendon through what I've done this week so someone else can revisit the code once I leave.
This week, I finished coding the netlink module as a Ryu application. This will be used as a context in the main Ryu application and all threads that need to be spawned will be spawned and managed by the Ryu Application manager; this solution resolves the issue, I encountered where the threads in the netlink module was being blocked. I tested the Ryu-Netlink application and I encountered an issue where trying to write to the netlink socket causes the socket to close, for the purpose of my work, writing to netlink (to configure or change interfaces) is more of a nice to have feature as I'm interested in getting and processing netlink broadcast messages so I have elected not debug this issue further for now, I'll use subprocess calls to ip route library if the need to make changes to interfaces arises.
This week, I'll be working on writing the RouteReceiver as a Ryu application and adding events that would notify the main Ryu application when a route has been received from the router.
Finished moving all the control connections to protocol buffers,
including sending a full set of test options to the server when the test
The udpstream test now calculates packet delay variation between
consecutive packets and reports summary statistics across the whole
stream. The test schedule (directions, sizes, etc) can now be controlled
properly on the client, which will inform the server of what it needs to
do and when.
Continued prepping for the trip to San Diego. Wrote a talk on AMP to present at AIMS, since Brendon won't be able to attend. Managed to finally settle on a project that I'll be working on at the BGP hackathon: adding useful filtering to the BGPstream software.
Met with Harris to talk about the CSC dataset and how he can go about looking for interesting events in the dataset. Wrote some example code to extract a metric from the data (syscalls per second for each major type) and added a module to NNTSC for the new metric. Hopefully Harris will be able to use that to start adding his own metrics.
Noticed that there was a lot of variation in my rtstats performance test results. It seems that the achievable packet rate for ring: seems to fluctuate from test to test, but will remain constant within a test. For example, one test run I'll get 1 million packets per second consistently for 60 seconds, the next test run I'll get 40,000 packets per second for 60 seconds. Spent a lot of time looking into this further (including an afternoon with Richard S. trying out various things), but we're still unable to account for this inconsistency.
I have completed wiring up the querying functionality for amp-web to influx.
Will now spend some time refactoring, documenting code and tidying up a few corners. Graphs and matrices seem to be doing what they're supposed to, but will try to double check that everything goes once more data is inserted and that I haven't missed any edge cases. Will also see if I can speed up the matrix query by searching for multiple streams at once.
Short week as I took two days off to move house.
Continued working on porting the udpstream test to the new amplet
client. Spent most of the time trying to combine the control protocol
from the throughput test and udpstream test into something nicely
generic that can be used with future test servers, instructing them on
when and how to send test flows. Also started to convert the control
channel to use protocol buffers so that interoperability is much easier.
Last week, I worked on resolving the limitation I encountered with Ryu controller that causes threads to block while waiting for an event to occur. The solution I came up with, involves writing the netlink module as Ryu application and run it as a context object within the Ryu application. This would allow Ryu manage thread synchronisation between netlink object and Controller objects.
I intend to finish working on that this week.
I've continued writing sections of the paper, I've been working through describing a way to avoid Packet-Ins and Outs.
I've also worked on the patch panel manager. Last week I had started on a simple ryu app using the REST API which I could link two ports together with. I've worked on web interface, for this I've opted to use angular, and so far it has been working well. It seems to be very well suited to displaying lists of JSON object which matches with my REST api output. This simple web interface allows two ports to be linked together and manage these links in general. I still have to finish implementing some things such as loading and saving configurations etc, and other features such as addingSPAN ports.
This week I began investigating how easy it was to store data from influxdb on an external disk, and found that Influx has put out a new update (0.10.0) in Beta which provides functionality to backup and restore data. Combining this with Retention Policies means that we will be able to easily send data to a backup disk and have it drop off the main storage system after a given amount of time.
I have begun to implement InfluxDB into the AMP system, and have got the dataparsers sending data to influx for every test except traceroute, while still using PostgreSQL for information about streams. I have also added code to automatically generate continuous queries and retention policies on setup. Plans now are to get client queries using the InfluxDB data.