User login

Blogs

05

Sep

2011

Managed to spend most of my week listening to talks:
* Matthew's practice talks on Tuesday
* Honours conference on Wednesday
* Matthew's first interview talk + more practicing on Thursday
* Matthew's second interview talk and the other competing candidates on Friday

Also, I attempted to improve the performance of my scapy program without any success. It still takes over 2 days to process a single day of Waikato trace!

Started working on a Libtrace paper version 3 - this one is going to be a lot longer and more explicit in describing the issues with trace processing that are solved by libtrace.

Reviewed a paper on modeling YouTube traffic for Computer Communications. Most of the paper replicated the analysis I had done in my own YouTube paper and the eventual model struck me as being overly simplistic, so I was not able to recommend it for publication.

04

Sep

2011

I adapted a scamper driver to collect data for three mda traces, which
were based on probes of packet type TCP, UDP and ICMP. 100 prefixes were
collected from routeviews and used to run these traces.

Next a program was written that counted the traces containing load
balancing, and the results were collected using this procedure.

I collected some more papers to review, based on the citations of some of
my current papers for reading.

02

Sep

2011

The 520 honours conference was on Wednesday. Overall I really enjoyed the day. I think my presentation went well. I ended up winning runner up best project award.

I had two interviews this week. One of which was down in Wellington where Braden works. Was cool to catch up with him and see how their company works.

29

Aug

2011

While double checking the quality of my datasets and looking for reasons
that would be preventing SMTP flows from matching (or helping non-SMTP
flows to match) I noticed that the direction flag was being inconsistently
set. This meant that many flows weren't contributing data to transitions
that were reachable (they would be discarded for being in the wrong
direction). Regenerated some data to start testing again.

Also noticed that even when I threw a lot more data into the mix I was
getting transitions that had a lot of variability - I was expecting more
data to give me more consistent distributions. Added some more checks to
try to prevent making stupid choices when merging nodes/transitions, which
didn't actually cause much to change. Many of the transitions I'm unhappy
with are actually a result of the clustering algorithm, so I'm now back
trying to make that generate more sensible clusters.

29

Aug

2011

Spent first half of week sick at home. Wednesday afternoon was spent running a photoshoot for lightwire which I really enjoyed taking photos for, had some great models and came out with some great shots at various locations.

Spent Thursday and Friday finishing the web interface for COMP518 which all works now and it's great to see some of the students have started their assignments.

29

Aug

2011

Finished fixing up my longitudinal study graphs, although I did discover that the direction tagging was the wrong way around for one of the datasets. Have re-run that analysis and updated the graphs accordingly.

Returned to my evaluation of other trace processing libraries. Managed to write a libnetdude program that replicated the results produced by the other programs, although it still cannot read from any sort of pipe without segfaulting. Also had to write my own IPv6 protocol plugin because libnetdude does not provide one. Tested it with an uncompressed pcap trace - it was the slowest of all the C libraries, despite not needing to decompress.

Started working with the python library, Scapy. Annoyingly, Scapy does not provide any mechanism for getting the header at a specific layer - instead you have to check for the existence of a specific protocol header that you're interested in. Scapy has also proved to be incredibly slow - I cannot believe anyone would use it for analysing anything except the most trivially small trace sets.

28

Aug

2011

Prepared a presentation for the Wednesday practice run. Got lots of helpful feedback from which I pretty much remade a new presentation from scratch. Just need to practice it lots before this Wednesday and it should go well.

Brendon helped me build up an example (real) VLAN for Karen. I hope to get something working before the presentation on Wednesday so I can make it part of my demonstration.

25

Aug

2011

Got to the bottom of the object extraction issues - some old code in the
program made some assumptions that only valid packets would be seen (or
that they would be checked for in a function that had moved to the
library) and was failing on a packet with a TCP data offset smaller than
the minimum allowed. Generated a lot of object trace data for a number of
consecutive traces in a recent ISP data set as well as a few from
different locations/times to use as testing data for the state machine.

Now that I have the state machine generation working consistently I was
able to run a bit of data through it. Initially I'm using an arbitrary
30 minute period from an ISP trace for my training data and comparing it
to some of the following 30 minute periods and the other object traces
mentioned. Accuracy of identifying SMTP flows is consistently high, though
the number of false positives is more than I would like. Looking closer,
most (~90%) of these flows are either POP3 (very similar to SMTP) or
various short lived unknown flows that just happen to match the right
packet sizes.

22

Aug

2011

This week I read a chapter from each of two text books and started on my
literature review by recording key points from several papers. These
latter are recorded in a bibliography database, as well as a text file
with reference details followed by the key points for each reference.

Some preliminary results were produced from the scamper traces from
Matthew. These where counts of paths containing load balancing for UDP
and ICMP MDA traceroutes.

22

Aug

2011

Updated most of the webpages for the longitudinal study to include new graphs for the ISP data - http://www.wand.net.nz/~salcock/longitude/ . There are still a few missing or broken graphs, but most of it is there now.

Started developing the libnetdude version of the scan analysis program. Seems libnetdude doesn't support reading from stdin, which is going to make reading my compressed ERF trace tricky...

At home sick from Tuesday - Friday.