User login

Blogs

28

May

2011

Had a play around with the google maps api. I need to visualise exports from a given country (starting with NZ) to all other countries for a group project. I turned a list of countries in to longitudes and latitudes using the google geolocation api. Was rather easy - just loop through all of the countries and make an api request in a rate limited way. Then, using export data from stats New Zealand, I drew weighted lines from NZ to all other countries. The resulting shape kind of looks like a butterfly.

View image

This is only the first iteration. The next step is to draw paths through the ocean that split off into countries. Similar idea to Minard's famous French wine exports drawing:

View Image

I also made a start on my 520 interim report, polished off two assignments and made a couple improvements to my network map.

24

May

2011

Managed to have a busy week this week with 5 unrelated software problems causing outages on different servers throughout the week that needed attention to fix. Erg was the first to die on Saturday which I brought back Monday morning followed by wagyu having a short outage on Tuesday. Warlock got into an unhappy state twice requiring a reboot to bring it back from the dead. Also the VPS hosting the KAREN wiki that I look after for REANNZ decided to go down as well on Thursday which came back after a fsck and a restart.

Looking forward to a "quiter" week next work catching up on uni assignments.

24

May

2011

I have made Tony's suggested changes to my introduction. Tony, Bill, Sally Jo and I have been meeting regularly to revise my thesis and improve parts of it as much as we can before I submit on Tuesday 31st.

23

May

2011

Spent some time getting the state machine generation code to read in a
machine from a previously output dot file so that the same machine can be
quickly reused to run different traces. Added extra reporting on spam
counts etc per link so that these can be used by programs later in the
chain when generating graphs of the paths spam/ham take through the
machine. This will hopefully let me run a few large traces through the
machine once and then use that data to test and evaluate others in a
fraction of the time.

Started investigating which MTAs the clients in my traces were using to
see if there were any interesting patterns. Approximately 20% of clients
accepted my test connection and 94% of those gave me something useful in
the banner or help message to identify them. Am currently waiting on
another run through of the trace with extra reporting identifying the MTAs
involved so I can compare between them.

Worked on getting the WRAMP simulator up and running on a more modern
version of wxWindows and with a newer compiler. Lots of search and
replace later it seems to be working fine. Most of the issues were with
wxWindows no longer accepting a good old fashioned char* as a string and
needing to convert everything to a unicode capable wxString.

23

May

2011

Arrived back at work on Tuesday afternoon and caught up with everything that had gone on in my absence. Spent the rest of the week doing little odd jobs. Fixed a couple of libtrace bugs that had come up while I was away. Created a trac for BSOD. Fixed some byte-ordering issues in libprotoident and released version 2.0.1. Filled in some of the missing documentation on the libprotoident wiki.

On Friday, I wrote a quick tool for live reporting of protocol usage using libprotoident. Thinking we could use the live stats to make some sort of pretty on-going graphs of the Waikato traffic.

20

May

2011

I made it easier to move between nodes in my network map, while zoomed in. You can now move along edges to the destination node. I also implemented some more edge types such as double arrows to indicate when edges are connected in both directions.
Read about 5 papers on efficiency related to topology inference from comp513. It was interesting to link theory I have learned in my data mining paper (information theory) to traceroute efficiency. Still have 3 more sections to write for that assignment (Alias resolution, ISP mapping and a section of my choosing).
Planned out a visualisation for my info vis paper that will display the exports of a given country with respect to the rest of the world.

19

May

2011

Libprotoident 2.0.1 has been released!

This release fixes a number of bugs in 2.0.0, as well as adding support for new application protocols and improving the rules for many existing ones.

The full list of changes is described in the libprotoident ChangeLog.

Download libprotoident 2.0.1 here!

18

May

2011

I have created Trac sites for both the libprotoident and BSOD projects, so it is now possible to file tickets to report bugs or request features for either of these projects through the Trac system, rather than having to contact me directly.

The Trac sites also feature wikis which I intend to use to provide more extensive documentation for these projects, e.g. explanations of the protocols supported by libprotoident. At the moment, this is a work in progress but hopefully will get fleshed out over time.

The BSOD trac: http://wand.net.nz/trac/bsod/
The libprotoident trac: http://wand.net.nz/trac/libprotoident

17

May

2011

Attended the ICT 2011 conference in Ayia Napa, Cyprus. Unfortunately,
most of the talks were a bit outside my areas of expertise and featured
way too many mathematical symbols, so I didn't learn too much that is
relevant to us. Gave my own talk in the final session of the last day to
a very small audience, but seemed to go ok and even got a few questions
(unlike most other talks I attended).

Spent the remainder of the week in Cyprus on holiday.

17

May

2011

Successfully got the state machine generation running across ISP traces,
fixing a few bugs that the new dataset exposed along the way. Took the
machine that was generated using the ISP data and ran it with the older
data with known spam status to see how they compared (quite similar).
Again, it is quite clear what is spam after the point it is rejected by
the mail server but the distinction is much less clear prior to that.

Started to work on reading the machine back in from the output dot graph
files so that a pre-built machine can be used to run against any object
trace without having to rebuild the machine every time.

Spent some time working on documentation about embedding R in C code in
response to an email query I got. I've been tinkering with this off and on
for a while and should blog about it when it's more complete.