User login

Blogs

02

Sep

2011

The 520 honours conference was on Wednesday. Overall I really enjoyed the day. I think my presentation went well. I ended up winning runner up best project award.

I had two interviews this week. One of which was down in Wellington where Braden works. Was cool to catch up with him and see how their company works.

29

Aug

2011

While double checking the quality of my datasets and looking for reasons
that would be preventing SMTP flows from matching (or helping non-SMTP
flows to match) I noticed that the direction flag was being inconsistently
set. This meant that many flows weren't contributing data to transitions
that were reachable (they would be discarded for being in the wrong
direction). Regenerated some data to start testing again.

Also noticed that even when I threw a lot more data into the mix I was
getting transitions that had a lot of variability - I was expecting more
data to give me more consistent distributions. Added some more checks to
try to prevent making stupid choices when merging nodes/transitions, which
didn't actually cause much to change. Many of the transitions I'm unhappy
with are actually a result of the clustering algorithm, so I'm now back
trying to make that generate more sensible clusters.

29

Aug

2011

Spent first half of week sick at home. Wednesday afternoon was spent running a photoshoot for lightwire which I really enjoyed taking photos for, had some great models and came out with some great shots at various locations.

Spent Thursday and Friday finishing the web interface for COMP518 which all works now and it's great to see some of the students have started their assignments.

29

Aug

2011

Finished fixing up my longitudinal study graphs, although I did discover that the direction tagging was the wrong way around for one of the datasets. Have re-run that analysis and updated the graphs accordingly.

Returned to my evaluation of other trace processing libraries. Managed to write a libnetdude program that replicated the results produced by the other programs, although it still cannot read from any sort of pipe without segfaulting. Also had to write my own IPv6 protocol plugin because libnetdude does not provide one. Tested it with an uncompressed pcap trace - it was the slowest of all the C libraries, despite not needing to decompress.

Started working with the python library, Scapy. Annoyingly, Scapy does not provide any mechanism for getting the header at a specific layer - instead you have to check for the existence of a specific protocol header that you're interested in. Scapy has also proved to be incredibly slow - I cannot believe anyone would use it for analysing anything except the most trivially small trace sets.

28

Aug

2011

Prepared a presentation for the Wednesday practice run. Got lots of helpful feedback from which I pretty much remade a new presentation from scratch. Just need to practice it lots before this Wednesday and it should go well.

Brendon helped me build up an example (real) VLAN for Karen. I hope to get something working before the presentation on Wednesday so I can make it part of my demonstration.

25

Aug

2011

Got to the bottom of the object extraction issues - some old code in the
program made some assumptions that only valid packets would be seen (or
that they would be checked for in a function that had moved to the
library) and was failing on a packet with a TCP data offset smaller than
the minimum allowed. Generated a lot of object trace data for a number of
consecutive traces in a recent ISP data set as well as a few from
different locations/times to use as testing data for the state machine.

Now that I have the state machine generation working consistently I was
able to run a bit of data through it. Initially I'm using an arbitrary
30 minute period from an ISP trace for my training data and comparing it
to some of the following 30 minute periods and the other object traces
mentioned. Accuracy of identifying SMTP flows is consistently high, though
the number of false positives is more than I would like. Looking closer,
most (~90%) of these flows are either POP3 (very similar to SMTP) or
various short lived unknown flows that just happen to match the right
packet sizes.

22

Aug

2011

This week I read a chapter from each of two text books and started on my
literature review by recording key points from several papers. These
latter are recorded in a bibliography database, as well as a text file
with reference details followed by the key points for each reference.

Some preliminary results were produced from the scamper traces from
Matthew. These where counts of paths containing load balancing for UDP
and ICMP MDA traceroutes.

22

Aug

2011

Updated most of the webpages for the longitudinal study to include new graphs for the ISP data - http://www.wand.net.nz/~salcock/longitude/ . There are still a few missing or broken graphs, but most of it is there now.

Started developing the libnetdude version of the scan analysis program. Seems libnetdude doesn't support reading from stdin, which is going to make reading my compressed ERF trace tricky...

At home sick from Tuesday - Friday.

19

Aug

2011

I added the ability to generically add overlays to my network map. A function is provided with the visualisation options, the graph nodes and links and the canvas to draw on. I need to do some testing to make sure this is the best and easiest way to do this. I am going to firstly look at overlaying VLAN data on to the karen example.

The honours conference is coming up on the 31st of August, so I have been starting to think about what is going to go into my presentation. There was talk about presenting to wand as a practice with the others at some point before the say.

16

Aug

2011

Updated the object extraction and protocol identification used to create
my input traces to use the new libraries Shane wrote for those tasks. I've
noticed a few small differences, with extra objects now being detected
that were previously missed. Some unusually formed packets from an ISP
trace also seem to be giving it trouble, which I'll need to dig deeper to
solve before I can start generating useful object traces.

Figured out the root cause of the deadlocks I was seeing in the threaded
tree merging code. Threads were holding on to too many locks (that seemed
safe to hold) while still waiting for others. Most locks were dropped if a
thread couldn't lock all the states possibly involved in merging but a
couple of locks on core states were sometimes still being held. These had
to go as well and now that the locks are an all or nothing package it
seems to work well. Speed is good and Helgrind is happy with everything.