User login

Blogs

19

Aug

2011

I added the ability to generically add overlays to my network map. A function is provided with the visualisation options, the graph nodes and links and the canvas to draw on. I need to do some testing to make sure this is the best and easiest way to do this. I am going to firstly look at overlaying VLAN data on to the karen example.

The honours conference is coming up on the 31st of August, so I have been starting to think about what is going to go into my presentation. There was talk about presenting to wand as a practice with the others at some point before the say.

16

Aug

2011

Updated the object extraction and protocol identification used to create
my input traces to use the new libraries Shane wrote for those tasks. I've
noticed a few small differences, with extra objects now being detected
that were previously missed. Some unusually formed packets from an ISP
trace also seem to be giving it trouble, which I'll need to dig deeper to
solve before I can start generating useful object traces.

Figured out the root cause of the deadlocks I was seeing in the threaded
tree merging code. Threads were holding on to too many locks (that seemed
safe to hold) while still waiting for others. Most locks were dropped if a
thread couldn't lock all the states possibly involved in merging but a
couple of locks on core states were sometimes still being held. These had
to go as well and now that the locks are an all or nothing package it
seems to work well. Speed is good and Helgrind is happy with everything.

16

Aug

2011

I am just settling in to my new research program, which is on the topic of upscaling the measurement of black holes in the Internet.

I have launched straight in to analysing some traces collected by Matthew Luckie. This is to determine if all per flow load balancing routers determine path using the same set of fields.

I have also bought a couple of text books for the courses that I am doing informally, and I have started reading the appropriate parts of them.

I am also getting my literature review procedure organised. I want to make notes in my bibtek file, however comments are automatically overwritten by the bibliography program that I use. Further, notes are printed with the references, however these problems were solved by creating a new unknown field name, which is neither discarded nor printed.

15

Aug

2011

Determined that the reason libpcap was outperforming libtrace when running the scan analysis was because we were CPU-bound rather than IO-bound. This meant that the faster IO of libtrace was not providing enough gain to cancel out the overhead of the libtrace function calls (compared with the direct pointer manipulation I was doing in my pcap program).

As a result, I decided to also test the libraries by doing a simple packet/byte count for each TCP and UDP port which would turn out to be IO-bound instead. In this case, libtrace was much faster than libpcap. Also implemented the two analyses using libcoral and ruby-libtrace. Libcoral was both slower and required more LOC than libtrace for both tests. Ruby-libtrace required less code for the port count (but more for the scan study, as I needed to write bindings for the flow management library I was using) but was waaaayyyy slower to run.

Finally finished running the longitudinal analysis on the various ISP traces and started working on adding the resulting graphs to my webpages. Decided that the ISP C time series graphs would be best done by plotting each year separately with an X-axis defined by the date minus the year, e.g. http://www.wand.net.nz/~salcock/longitude/graphs/icmp/icmp_in_ispc.png . This, of course, involved reworking a decent chunk of my graph generation scripts...

12

Aug

2011

This week I worked on getting my Karen example tidy enough for Richard to present to the crew down in Wellington on Thursday. This involved making a drag & drop position editor to make it easier to make changes to node positions. I fixed up the CSS layout to make it look more like a dashboard. I also did some cross browser testing and it seems to work okay on most latest version browsers. There were some exceptions, but mostly just performance issues.

I have made the Karen demo public here:
http://wand.net.nz/~jo26/network-map/examples/karen-detail.html

Any feedback is most welcome :-D

09

Aug

2011

Short week as I was away in Wellington on Thursday/Friday.

Worked with Helgrind on improving the threading in the SMTP state machine
construction to clear up a couple more race conditions that were present.
Looks to be really close to getting a clean run through with large numbers
of flows processed using a few threads, in fairly short time.

Started looking at what needs to be done to use the newer object
extraction and protocol identification libraries to create my object
trace files before exploring a range of traces from different times and
locations. Will be interesting to see how applicable results are across
different times/places.

09

Aug

2011

Spent most of this week developing the web frontend for COMP518 cyber security assignments. It's coming along quite nicely and I'm attempting to keep it secure keeping in mind the target audience.

Since everyone was in Wellington this week I got to run the NZPC programming contest marking on Saturday. Took a bit to get my head around Judge Judy, but eventually got it all sorted.

08

Aug

2011

Short week this week, as I was in Wellington on Thursday and Friday.

Managed to get Bro running and producing results that I could replicate with a libtrace program. Found that Bro was tracking TCP state incorrectly - it would often describe a TCP flow as both established and closed correctly when, in fact, no SYNs were observed at all. Reported the bug to the Bro team and decided to use my state classifications from now on.

Wrote a libpcap program that was equivalent to the libtrace program to compare the performance of the two. Surprisingly, the "zcat | dagconvert | libpcap" run was quite a bit faster than the "libtrace" equivalent. Profiled the libtrace program and managed to find a couple of opportunities for speeding things up, mostly through increased caching. The libpcap program is still slightly faster now, but the gap has closed significantly.

05

Aug

2011

Created a basic ISP for 514 this week. PPP client connecting to a NAS running PPPoE and authenticating via another server running freeradius.

I have also been working on getting a cleaner example of my network maps for Richard to show Karen next time he visits. I have been altering the overview plots to show and additional view for each deeper level. This will make it more useful when you zoom in further.

My health has been pretty bad lately. I have a flu for a week and a half, then crushed my hand on Sat and finally have been diagnosed with tonsillitis today :-S

02

Aug

2011

Rewrote the main section of the SMTP state machine code responsible for
merging states and minimising the machine. It now performs a recursive
traversal of the tree and is smarter about which states are available for
merging at any point. I improved the time taken to process 30 minutes of
an ISP trace from 25 hours to 3 and a half hours using a single thread.
Almost got all the locking sorted to be able to run successfully and
consistently with multiple threads which will hopefully increase
performance further. Initial tests with small numbers of flows and two
threads showed it taking 1/3 less time again.

Also got some useful comparisons from the first few ISP traces I've been
testing against, now that the program runs significantly faster. Using
approximately 7000 SMTP flows from a 30 minute trace as training data I
can identify about 97% of the SMTP flows in the following 30 minute
period. Unfortunately there is still quite a large number of false
positives, though the majority of those are POP3 flows which are very
similar to SMTP. Will need to see how these compare when looking at data
further away in time and location and/or with better object/protocol
identification using the new libraries.