The Emulation Network is designed to be used as a testbed where people can try out ideas in an environment that is predictable and fully controlled. It aims to be a versatile system that will allow users the freedom to experiment without worrying about interfering with a production network, or going through the trouble of setting up their own.
Worked with Shane to try to get a new FreeBSD image built for the
emulation network. FreeBSD is still pretty terrible to deal with and seems
to have updated some disk partitioning/formats to ones that frisbee
doesn't fully understand. For now it's creating a raw disk image, but if
we do another from scratch it would be good to see if we can use some
slightly older options.
Changed the way the allowable time difference between events in a group is
calculated for LPI events to shorten it if the events were significantly
different. Some suboptimal groupings were being made because the events
were slightly similar and nearby in time but not related.
Worked with Nathan to test his program operating on smokeping rrd files
and feeding the results into the anomaly detection process. Had to put
together a module to deal with specifically with smokeping data, and think
I now have a good understanding of how everything fits together, where
data storage needs to be etc. Waiting on more data to arrive and the
fetching of historical data to work to properly fill the training buffers
now before events will be detected.
I'm still not happy with the event groupings that I'm getting, so spent
some time looking further into the literature for ideas. A little bit of
this is caused by my test data fetching program presenting information
outside chronological order (Nathan should hopefully have his one working
soon and I can use that) but a lot is still due to making bad grouping
decisions. Found a few interesting ideas about comparing events for
similarity, but most require more attributes to compare on. Thinking I
should start investigating other attributes that I can add to the events -
information about paths etc.
Wrote up a blog post
our adventures with NDT, hopefully that will be of interest to a few
people. Might be worth looking deeper into it to see if we can track down
what was causing some of the performance issues.
Spent some time looking at various emulation network issues. update-grub
is sometimes using the wrong root partition, so running it often results
in an unbootable machine.
Fixed the http2 test in AMP to properly share the DNS cache between
simultaneous connections which means it no longer performs unnecessary
lookups for the same name. The sharing interface in libcurl actually works
Tried to build new amplet packages including the recent changes, but ran
into some problems with libraries when building in my lenny buildroot.
Autoconf/make is meant to build a particular binary with an extra library
that the rest don't need, but this doesn't make it through to the Makefile
Jamie put together new RJ45-DB9 serial connectors for the emulation
network, so I created some sensible minicom configs for all the machines,
should be just as easy to use now as the old system with the Cyclades
terminal server was. Also set up udev on my linux image to force a
consistent order of the network interfaces that matches the way they are
Finally got frisbee working properly on the emulation network. The new
ramdisk image missing useful drivers for disks meant that any imaging was
actually taking place on the ramdisk. Having drivers in the ramdisk for
all the network cards meant that the old way of determining the interface
address would no longer work (all the interfaces were given the same
Had a couple of good meetings with Nathan and Anthony about their honours
projects and some of the requirements for what they are doing.
Had more trouble with the emulation network than expected. Built a custom
ramdisk using proper tools (mkinitramfs etc) and a default Debian kernel
rather than using entirely custom ones. This worked fine, but after
installing an old image the machines would refuse to boot. Turns out that
the disks used to be part of a RAID and didn't have a useful MBR (and
frisbee didn't fix this). Built a master machine to create images from,
built a new Squeeze image and am now trying to convince frisbee to send
the entire image. I'm thinking a new version of frisbee may be required.
Tried to track down why the http2 test was performing DNS queries multiple
times for the same hostnames. Even with all the DNS cache sharing options
set in libcurl it will repeat requests, unless I force IPv4 only. A vague
line in a changelog looks like this might be fixed, but too recently for
the change to make it into Debian.
Finished putting together basic historical usage data for KAREN, it shows
a nice up and to the right trend.
Spent some time working on adding a latency test to AMP that will perform
DNS lookups and then test to the (possibly multiple, possibly changing)
addresses resolved. This can hopefully be used to give a bit of insight
into some of the google services. In doing this I found and fixed a few
more small bugs in AMP that were showing up with newer compilers etc on
the emulation network.
The Cyclades terminal server in the emulation network won't be getting
it's firmware upgraded - it now refuses to boot at all. Bringing it into
the lab and having a closer look at it doesn't really show any options on
how to improve the situation.
Put together a bit of pydoc documentation for the new emulation network
set up stuff. It's not complete but it covers most of the user facing
functionality and is very easy to use.
Spent some time working with Chris and William to get scripts in place and
working, testing various bits and trying to provide some helpful
Recent software upgrades have meant that the imaging and configuration
program for the emulation network (that was based around ns-2 years
ago) needed to be recompiled to work with new libraries etc. Decided a
better solution was to finish the python version I started writing last
year to do the job. It's fairly simple now to install and configure
machines again, and hopefully won't be too hard to add in support for
virtualised machines as well.
In doing so I found some issues with the Cyclades terminal server used
in the emulation network - it kernel panics after a short amount of
uptime, sometimes while booting, and sometimes fails self memory checks
while booting. There aren't any obvious user serviceable parts inside (no
flash that we can replace), so going to try updating the firmware and see
if that improves the situation.
Also spent some time trying to track down unusual ipv6 results that were
showing some sites as having no connectivity to other sites that they
normally should. Both Waikato and Victoria started failing to reach test
destinations at the same time last month, which I'm trying to track down.