Libprotoident is a library that performs application layer protocol identification for flows. Unlike many techniques that require capturing the entire packet payload, only the first four bytes of payload sent in each direction, the size of the first payload-bearing packet in each direction and the TCP or UDP port numbers for the flow are used by libprotoident. Libprotoident features a very simple API that is easy to use, enabling developers to quickly write code that can make use of the protocol identification rules present in the library without needing to know anything about the applications they are trying to identify.
Started looking at the most common patterns in my example sysdig logs. It's pretty obvious that we can easily recognise some low-level actions based on the sequence of system calls and produce models that can be used to identify them. For example, loading a .so shared library will generally result in the same sequence of system calls (with some minor variations) and therefore that can be expressed as a finite state machine.
Developed FSMs for four low level actions: loading a .so library, loading a python module, receiving a typed character via ssh and reading a modprobe config file. Implemented the SSH action as code so I can now find and replace those sequences in the logs with a single SSHCharInput action.
Helped Brendon install NNTSC, ampy and amp-web packages on one of our existing deployments on Thursday. We ended up with a problem where NNTSC would not return query data to the web-site and it took a lot of time (and debugging) to find the source of our problem: incongruous versions of psycopg2 in pip vs the debian package.
Started prepping a libprotoident release. libprotoident is moving to an LGPL license so I've had to replace the blurb at the top of every source file. Been working through the usual pre-release testing and ChangeLog updating.
Spent Wednesday at the Honours conference. I thought all of our students presented well and gave good accounts of their work so far.
Worked on the camera-ready version of my IMC paper. Managed to add some nice content to address the reviewer feedback we got, only to find that I had been using a font size that was too small (i.e. the default font size that every previous IMC has ever used). Unfortunately, switching to the bigger font size would mean I would have to remove almost all the new content I had added, so I'm hoping the PC chairs will change their mind and revert back to the old font size.
Wrote the basic architecture for a provenance log parsing library that can be used with both live progger records and sysdig log files. This will replace the old progger-central which I had written as a hacky PoC which was in danger of becoming production code otherwise.
Got my script to extract common patterns from Sysdig logs working reasonably well. Took a few attempts to get some nicely formatted output that contains all the information I should need to track down what actions are causing the repeated patterns.
Spent a fair bit of time helping CROW get a handle on the Endace Probe, what it can do and how it might fit into their research goals.
Listened to our 520 student practice talks on Thursday. The projects themselves are pretty good -- just the usual issue of the students underselling just how much actual work they had put in to the development side of their project.
Got NNTSC and amp-web working with the sysdig data that Harris gave me, so we have a simple proof-of-concept. After talking with Harris some more, he is interested in finding patterns in the syscall logs that are "predictable" so that we can build models of known specific actions on a system (e.g. opening a file with vim, starting a python interpreter etc). Started working on a script to identify common patterns in the sysdig logs so that we can get an idea as to what these patterns look like and how hard they will be to recognise and identify.
Continued tracking down unknown traffic patterns with libprotoident. Managed to nail one pattern that had been bugging me for a long time: the Baidu Yun P2P protocol. Also added rules for YY, Overwatch, Zoom TCP and NetCat CCTV.
My IMC paper on unexpected traffic on well-known ports was accepted, which is great news. Spent Monday going over reviewer feedback and thinking about what revisions I need to make for the camera-ready version.
Continued working on integrating STRATUS with NNTSC. Spent way too much time trying to figure out why my data was not being inserted into the Influx database -- turns out the timestamp for the test data I was using was too old for the default retention policies so it was being automatically discarded. Fudged the test data times to be more recent and it finally worked.
Added file operations metric support to ampy and amp-web so we can now look at simple graphs of open frequency data. Found some scalability issues with our modal dialogs in cases where the number of possible options for a dropdown is very high, so I've gone back and added pagination support to all modal dropdowns so they only load 30 or so options at a time. This had some interesting flow-on effects, especially for the latency modal dialog which had a lot of custom code for populating the tabs for the different latency metrics. I think I've ironed out all of the extra wrinkles now.
Spent a little more time with the July traces to track down some more unknown protocols. Added a rule for the Netcore vulnerability scan (which happens a lot!) and updated rules for a lot of (mostly game-related) protocols.
Started working on integrating some of the STRATUS metrics into NNTSC so that we can explore using time-series based event detection to highlight potentially interesting file interactions. Going forward, I'm going to be splitting my time 50:50 between STRATUS development and WAND research work -- existing research might progress a bit slower as a result.
Continued poking at unknown flows in the July trace data. Added protocols for Final Fantasy XIV and Facebook Messenger. Noticed that we are still having issues with the vDAG pipe on the probe that services wdcap dropping packets so our captures are sometimes missing packets. Moving IP encryption off onto wraith seems to have helped with this, but is not an ideal solution.
Short week after taking leave on Monday and Tuesday.
Spent most of my remaining week looking at some new captures I took using the upgraded Probe. The main aim was to see whether there were any new protocols that libprotoident should be able to identify. Managed to find a handful of new protocols: Facebook Zero, Forticlient SSL VPN and Discord, as well as made some improvements to the rules for existing protocols (including the AMP throughput test!).
Most of my time was actually spent unsuccessfully hunting down what appears to be a new Chinese P2P protocol, which is a shame because it was contributing a very large amount of unknown traffic in my sample dataset.
Using BSOD on the live traffic feed also allowed me to spot a student that was doing vast quantities of torrenting on the campus network (which Brad reported to ITS) and our WITS FTP server being hammered with tons of download attempts from China. Fair to say, we've gotten some good milage of the upgraded Probe already.
Fixed a couple of outstanding bugs in amp-web. Should be ready to push some new packages out to skeptic and lamp early next week now.
Finished up the first release version of the event filtering for amp-web and rolled it out to lamp on Thursday morning. Most of this week's work was polishing up some of the rough edges and making sure the UI behaves in a reasonable fashion -- Brad was very helpful playing the role of an average user and finding bad behaviour.
Post-release, tracked down and fixed the issue that was causing netevmon to not run the loss detector. Added support for loss events to eventing and the dashboard.
Released a new version of libprotoident, which includes all of my recent additions from the unexpected traffic study.
Marked the last libtrace assignment and pushed out the marks to the students.
After what seems like forever, I've finally managed to put together a new libprotoident release that includes all of the new protocol rules I've developed over the past couple of years. This release adds support for around 70 new protocols, including QUIC, SPDY, Cisco SSL VPN, Weibo and Line. A further 28 protocols have had their rules refined and improved, including BitTorrent, QQ, WeChat, Xunlei and DNS.
The lpi_live tool has been removed in this release, as this has been decommissioned in favour of the lpicollector tool.
Also, please note that libflowmanager 2.0.4 is required to build the libprotoident tools. Older versions of libflowmanager will fail the configure check.
The full list of changes can be found in the libprotoident ChangeLog.
Started writing up a short paper on the unexpected traffic analysis I've been doing for the past few weeks. Made decent progress -- I've got a mostly complete draft, just missing a conclusion and an abstract.
Spent a decent chunk of Thursday dealing with the fallout from upgrading influxdb to 0.11 on prophet. This broke most of our existing rollup tables, as the data type that we were now inserting (int) was no longer compatible with the data type that we apparently used to insert (float). Compounding matters was influxdb's lack of visibility into what data types are associated with any given column. Ended up trashing and re-creating the database (somewhat by accident) which fixed the problem, but not an ideal solution if we ever roll this out in production.
513 assignment was due at 5pm on Friday, so dealt with a few final queries from students. 20 submissions in the end, so a bit of marking to do next week.
Continued making progress with my unidentified mice flows in libprotoident. Added a whole pile of new rules, mostly for various Chinese apps again. Have probably done enough now that I can draw a line under this and start writing the paper itself; there are a few obvious patterns that I would like to identify but this has consumed a lot of time already.
Answered a handful of questions from 513 students -- mostly intelligent ones, so I'm reasonably confident about how the class is going overall. Due date is this coming Friday, so we'll know for sure soon enough.
Helped finish off the funding proposal in the first half of the week.
Continued working with libprotoident. This week I gave up on the elephant flows and started looking at the mice flows. Found some interesting stuff; the highlight being a huge number of flows on TCP port 80 that seem to be associated with the Baidu web browser. The behaviour of these flows is particularly odd: connect to server, send a FIN with seqno N, retransmit FIN a few times, send a non-FIN packet with 1 byte of payload (0x00) and seqno N-1 (incredibly invalid TCP behaviour!), server sends a RST. End result is > 150,000 flows over a week on port 80 with a single outgoing byte of payload.
Added some filters on the Endace probe to see if we can find people doing this traffic on campus, as the Baidu browser is pretty well-known for having a tendency to leak all sorts of private data back to its masters. Found multiple staff PCs that appear to be doing this sort of traffic, so Brad and I will try to prepare a report for ITS next week.
Met with Nathan at Lightwire on Thursday afternoon re: AMP and netevmon. Came away with plenty of ideas and suggestions for improvements we can make and hopefully we also helped Nathan understand parts of our system better as well. The good news is that netevmon seems to mostly be picking up valid events, but even so the number and frequency of these events can be overwhelming so we need better control over what events are shown to the user.
Worked on the next MBIE funding proposal document. Still got a fair way to go so this will probably eat up a lot of next week too.
Continued trying to identify the remaining Unknown applications in the Waikato Sept 15 traces. Only managed to identify one new protocol (Xunlei Accelerated) but this did account for 14G of unknown traffic on TCP port 8080 so that has gotten rid of the biggest outstanding quantity of unknown traffic. The rest are looking like they might get the better of me -- it's almost all Chinese in origin and I can identify the parent company (Tencent, CERNET, Taobao etc) but actually figuring out which of the myriad of apps these companies own is mostly just trial and error at this stage.
Continued working away at the Unknown traffic from my libprotoident port study. Added new protocols for Telegram Messenger and Kuguo, as well as improved DNS (especially TCP DNS) and NTP matching. I still have a bit more Unknown traffic to identify before I'd be comfortable putting the results in a paper, but we're getting closer.
Gave my 513 lectures this week. Looking forward to seeing how the class get on with my assignment.
Met with Ryan Jones who is doing an Honours project that will use netevmon to try and find events in the CSC data. Gave him access to the code and a few hints to start out, but I imagine I'll have to dedicate some more time to this over the course of the year.
My fixes to Andy's InfluxDB code seems to be resulting in consistent and correct bins being stored in the rollup tables. Threw netevmon at the development system to see if it can cope, which it seems to be doing OK. There's still a bit of a concern around long-term memory usage, but I'll see how that pans out over the next couple of weeks.
Spent the rest of my week concentrating on finishing up JP's summer study on unexpected traffic on typically open ports. Managed to improve a few existing rules to recognise more traffic, as well as add new rules for QQ video chat and what appears to be a C&C covert channel for some Chinese malware using UDP port 53. Started framing up a paper for IMC based on this study.
Did some final prep work for the libtrace lectures and assignment for 513.
Finished up the implementation chapter of the libtrace paper. Added a couple of diagrams to augment some of the textual explanations. Got Richard S. to read over what I've got so far and made a few tweaks based on his feedback.
Spent a decent chunk of time looking at Unknown UDP port 80 traffic in libprotoident. Found a clear pattern that was contributing most of the traffic, which I traced back to Tencent. Unfortunately Tencent publishes a lot of applications so that knowledge wasn't conclusive on its own.
My initial suspicion was that it might have been game traffic so I downloaded and played a few popular multiplayer games via the Tencent games client, capturing the network traffic and comparing it against my current unknown traffic. No luck, but then I had the bright idea to look a bit more closely at video call traffic in WeChat (a messaging app). Sure enough, once I was able to successfully create two WeChat accounts and get a video call going between them, I started seeing the traffic I wanted.
Also added rules for Acer Cloud and OpenTracker over UDP.
Started writing some content for the parallel libtrace paper. Managed to churn out an introduction, a background and a little bit of the implementation section.
Fixed a couple of bugs in netevmon prior to the deployment: crashing when trying to reconnect to a restarted NNTSC and some confusing event descriptions for changepoint events.
Finished setting up a mobile app test environment for JP. I've configured my old iPhone to act as an extra client for 2-way communication apps (messaging etc.). So far the environment has already been helpful, as we've managed to identify one of the major outstanding patterns as being used by the Taobao mobile shopping app.
Finished up the demo for STRATUS forum and helped Harris put together both a video and a live website.
Spent a bit of time trying to fix some unintuitive traceroute events that we were seeing on lamp. The problem was arising when a normally unresponsive hop was responding to traceroute, which was inserting an extra AS transition into our "path".
Rebuilt DPDK and Ostinato on 10g-dev2 after Richard upgraded it to Jessie so that I can resume my parallel libtrace development and testing once he's done with his experiments.
Installed and tested a variety of Android emulators to try and setup an environment where JP and I can more easily capture mobile app traffic. Turned out Bluestacks on my iMac ended up being the most useful, as the others I tried either lacked the Google Play Store (so finding and installing the "official" apps would be hard) or needed more computing power than I had available.
Played around with getting netevmon to produce some useful events from the Ceilometer data and updated amp-web to be able to show those events on the dashboard. Some of our existing detection algorithms (Plateau, BinSegChangepoint, Changepoint) worked surprisingly well so we should have something useful to demo at the STRATUS forum on Friday.
Helped Brendon get netevmon up and running on lamp. There were a few issues unfortunately, mostly due to permission issues and R being terrible, but managed to get things running eventually. Spent a bit of time fixing some redundant event groups that we observed from the lamp data which were a side-effect from the fact that a group of traceroute events can be combined with both latency increase and decrease events. We also worked together to track down some bad IP traceroute paths that were being inserted into the database -- new amplets were not including a 'None' entry for non-responsive hops which NNTSC was expecting so an 11 hop path with 6 non-responsive hops was being recorded as a 5 hop contiguous path. Updated NNTSC to recognise a missing address field as a non-responsive hop.
Gave JP a crash course in libprotoident development so he can get started on his summer project.
Spent the early part of my week reading over Dan's and Darren's revised Honours reports and offering a final batch of suggestions.
Continued poking at libprotoident and the unknown traffic on various Web ports. Finally managed to get Blade and Soul (a Chinese MMO) installed and running and was able to confirm that it was responsible for some of my unknown flows.
Started turning my attention towards our STRATUS research this week. Initially, we are going to look at general metrics that we can extract from cloud infrastructure and see if any of our existing event detection techniques are useful for finding anomalous behaviour. For a start, we are using data collected by the Ceilometer module on the Waikato OpenStack instance. Spent some time bringing Harris up to speed on NNTSC and netevmon so that he can experiment with the data within our system. In the meantime, I'm going to take a closer look at the data that we've collected to see which series will be most suitable to focus on in the short term.
Gave more details about our STRATUS work / goals to the designers who will be producing a poster about our research for the upcoming STRATUS forum.
Also played with a service called ThisData which claimed to offer something similar to what we have envisioned from STRATUS. ThisData is certainly pretty, but doesn't really seem to offer much more than daily revision control for your cloud data.
Spent a fair chunk of my week proof-reading, first a document responding to questions about the BTM project, then Dan and Darren's Honours reports.
Tracked down and fixed a bug in parallel libtrace where ticks were messing with the ordered combiner, causing some packets to be sent to the reporter out of order. Also managed to replicate and fix the memory leak bug that was causing Yindong's wdcap on wraith to invoke the OOM killer.
Continued poking at unknown port 443 and port 80 traffic in libprotoident. Most of my time was spent trying to install and capture traffic from various Chinese applications that I had reason to suspect were causing most of my remaining unknown traffic, with mixed success.