Updated some of the signals used with the amplet client to provide
better management - as well as being able to reload configuration from
disk, it can now force a refetch of remote schedule files with a SIGUSR2.
Also made sure that all children (tests, servers, etc) have their
signals unblocked and the signal handler restored to the default.
Libwandevent sets all these in the main process, which was being
propagated to the children and causing some unexpected behaviour. The
init scripts now try to kill the entire process group of the amplet
client, which means children should now get the signal too.
Renamed server processes in ps so that it was obvious what task they
Refactored some more of the repeated server code out of the
udpstream/throughput tests so they are now a lot cleaner. Moved some of
the test server control message code around so that it was grouped
together in a sensible place.
Continued making progress with my unidentified mice flows in libprotoident. Added a whole pile of new rules, mostly for various Chinese apps again. Have probably done enough now that I can draw a line under this and start writing the paper itself; there are a few obvious patterns that I would like to identify but this has consumed a lot of time already.
Answered a handful of questions from 513 students -- mostly intelligent ones, so I'm reasonably confident about how the class is going overall. Due date is this coming Friday, so we'll know for sure soon enough.
I don't have much to report this week, I've been working on FastPath for Rhea and I hope to get it completed before the next week.
Spent some time updating unit tests to work properly with the new
watchdog and control API. Improved checks to make sure that only valid
control messages are being parsed. Other small fixes to make sure that
errors are caught and reported properly.
Started refactoring the test control connections to use an SSL BIO so
that exactly the same code paths can be used to read and write control
messages whether SSL is in use (amplet, standalone tests) or not
(standalone tests), which has removed/simplified a lot of code. Also
figured out how to properly do non-blocking IO when the BIO functions
behave differently to normal read/write.
Went with Shane to visit Lightwire on Thursday and had a discussion
about how we can make event detection, measurements, graphs etc work
better for them.
Helped finish off the funding proposal in the first half of the week.
Continued working with libprotoident. This week I gave up on the elephant flows and started looking at the mice flows. Found some interesting stuff; the highlight being a huge number of flows on TCP port 80 that seem to be associated with the Baidu web browser. The behaviour of these flows is particularly odd: connect to server, send a FIN with seqno N, retransmit FIN a few times, send a non-FIN packet with 1 byte of payload (0x00) and seqno N-1 (incredibly invalid TCP behaviour!), server sends a RST. End result is > 150,000 flows over a week on port 80 with a single outgoing byte of payload.
Added some filters on the Endace probe to see if we can find people doing this traffic on campus, as the Baidu browser is pretty well-known for having a tendency to leak all sorts of private data back to its masters. Found multiple staff PCs that appear to be doing this sort of traffic, so Brad and I will try to prepare a report for ITS next week.
Met with Nathan at Lightwire on Thursday afternoon re: AMP and netevmon. Came away with plenty of ideas and suggestions for improvements we can make and hopefully we also helped Nathan understand parts of our system better as well. The good news is that netevmon seems to mostly be picking up valid events, but even so the number and frequency of these events can be overwhelming so we need better control over what events are shown to the user.
Last week, I was able to convert routes to OpenFlow rules and install the rules on the switch. The rules installed on the switch now make it possible for hosts on different networks connected to the switch to communicate without packets getting to the gateways of each network.
This week, I'll be working on writing the code to setup fastpath between the OF switch and the Rhea virtual switch. The aim of fastpath is to have a direct link between the OF switch and the virtual switch and have certain types of packets like ICMP being forwarded directly to the virtual switch mapped ports rather being sent to the controller which then examines the packet and sends it to the virtual switch.
Finished adding the ability to set DSCP bits for all the amplet tests
individually as well as globally. Slightly tidied up the way the global
options are turned into individual test options now that there are a few
more of them.
Tidied up the management connections to try to reuse the existing SSL
connection that started the server, rather than always expecting a
separate connection (as is sometimes the case when run standalone). As
part of this, added SSL support to the standalone tests, so now they can
be run standalone with/without SSL, or using it to connect to a normal
Reworked the way watchdogs worked to make sure they will properly
monitor new server threads, or remotely scheduled tests. The central
watchdog management has now been replaced by a timer inside each
server/test process that will ensure the test completes on time.
Worked on the next MBIE funding proposal document. Still got a fair way to go so this will probably eat up a lot of next week too.
Continued trying to identify the remaining Unknown applications in the Waikato Sept 15 traces. Only managed to identify one new protocol (Xunlei Accelerated) but this did account for 14G of unknown traffic on TCP port 8080 so that has gotten rid of the biggest outstanding quantity of unknown traffic. The rest are looking like they might get the better of me -- it's almost all Chinese in origin and I can identify the parent company (Tencent, CERNET, Taobao etc) but actually figuring out which of the myriad of apps these companies own is mostly just trial and error at this stage.
Received feedback for the paper, I've worked a little bit on some of this.
Filled out my PhD progress report. Had a talk with Richard about this he is concerned that my proposed approach might take to long before I start tackling the key issues. As such this is likely to be discussed more next meeting.
Styled the patch manager I've been working on with bootstrap and added some basic documentation ready and other tidying. It now has a name OFCupid and is available on github https://github.com/wandsdn/OFCupid.
Had some interest in libtrace with the DPDK 2.2 library (currently support up to 1.8), it seems that they are working on updating the code to support this.
I'm away for the next three weeks.
Decided to start doing systemd scripts properly and wrote a service file
for the amplet client. Also slightly tweaked the debhelper scripts that
are run to make sure the client doesn't start without configuration and
end up reporting errors to systemd. Had to officially split the Debian
directories now for Wheezy and Jessie as they are starting to diverge
Started work on adding the ability to set the differentiated services
bits in the IP header for all of the AMP tests. This can be set at a
global level, or on a per test basis. So far only the icmp test will
obey the setting, I'll update the rest of the tests next week.
Spent some time trying to remove an unnecessary extra control connection
for tests involving servers started by a remote amplet client. It looks
like I should be able to reuse the connection used to start the server
as the ongoing control channel, but I'm not quite sure how to make this
work best with standalone tests (that expect the server to already be
running, and don't currently encrypt anything). I should be able to tell
if I have a secure control connection or not and take the appropriate
actions, but a bit more planning is required.