Improved KAREN Weathermap
This project attempts to deploy Joel Oughton's Improved Network Visualisation 520 project on KAREN as a replacement for the existing mostly static weathermap. It will also extend Joel's work to include features present on the existing weathermap, such as pop-up graphs showing recent performance trends, and incorporate the feedback received from KAREN when they were presented with Joel's initial prototype.
Advances in web browser technologies have allowed for highly interactive, cross platform applications that have not been fully utilised in the computer network industry. The aim of this project has been to use these technologies to produce an improved implementation of a network map that effectively communicates topology and performance data to its users. The development of a map has involved assessing HTML5 technologies, identifying important features of network maps, researching relevant information visualisation theory and developing working prototypes. Representing computer networks in particular can be diffcult when the number of nodes and links is large. For this project, a network mapping tool called NetMapJs was implemented to address weaknesses in current tools. Visualisation techniques were identified and used that were effective in addressing the scalability of displays, such as semantic zooming and layout grouping. This, along with an expressive way of displaying network
bandwidth, has resulted in an improved presentation of network map displays. To evaluate the implementation, network maps for two operational networks were developed which received good feedback from the engineers in those organisations.
Spent some more time reading bits of honours reports before they were
Updated addressing the KAREN AMP machines so they would continue to work
with recent network changes. In the process of doing so, discovered that
CFEngine would no longer update certain sites and spent quite a while
trying to debug it. It was failing to authenticate server keys properly,
which was fixed by forcing it to refetch the (exactly the same,
identical) key. Not impressed that it is acting flakey over something like
Started work on a new implementation of AMP using some of the ideas we've
been talking about. Currently I'm working on a reimplementation of
measured using libwandevent. At this stage it can read the old format of
schedule file and creates a new timer event for each one, runs a dummy
function when the time arrives and reschedules itself afterwards.
Worked with Shane to try to get a new FreeBSD image built for the
emulation network. FreeBSD is still pretty terrible to deal with and seems
to have updated some disk partitioning/formats to ones that frisbee
doesn't fully understand. For now it's creating a raw disk image, but if
we do another from scratch it would be good to see if we can use some
slightly older options.
Changed the way the allowable time difference between events in a group is
calculated for LPI events to shorten it if the events were significantly
different. Some suboptimal groupings were being made because the events
were slightly similar and nearby in time but not related.
Worked with Nathan to test his program operating on smokeping rrd files
and feeding the results into the anomaly detection process. Had to put
together a module to deal with specifically with smokeping data, and think
I now have a good understanding of how everything fits together, where
data storage needs to be etc. Waiting on more data to arrive and the
fetching of historical data to work to properly fill the training buffers
now before events will be detected.
Sat down with Shane and went over what we need to do to get our event
detection programs integrated. The protocol used between data fetching,
detection and eventing needs to be updated slightly as there is more
information needing to be shared and magic numbers from testing to be
changed into real data. Started work on updating the protocol to match
what is required and updating the database schema to match.
Integrated the path change detection into the main detection code and
updated the main code path to deal properly with the slight differences
between a wider variety of data - traceroute, latency, byte counts, etc.
Did a bit of maintenance on the NZ AMP mesh and KAREN weathermap as well -
updated some addresses and got IPv6 addresses for the Citylink and
Netspace AMPlets which is neat.
Tidied up some more of the AMP documentation to make it more friendly to
others installing it, made some of the Debian package scripts a bit
smarter and worked through closing a few of the more simple outstanding
tickets. Fixed a whole lot more errors on the PHP side of things in
Worked with Chris to try to recreate in SQL some of the ways we use RRD in
the weathermap. This seems to be coming along - it is functional and
pretty quick but just needs tweaking to make sure that it is fast when
dealing with large amounts of data.
This week I worked on a new star layout algorithm that deals with the "outer" edges so that the "inner" edges will not overlap with the outer ones. The main idea is to rotate all of the inner nodes until it finds a solution where it does not overlap. It this solution fails, it will run another method which will reduce the arc between each nodes in the circular group.
The only problem with this algorithm now is that it does take a while to compute (approximately 4 seconds upon page loading). I also spent about a day documenting all the code I've been working on.
I've attached some images showing the star layout before and after.
This week I continued to fix bugs that were occurring on the weathermap. One particular annoying bug that occurred was the zooming inverse bug that flip all edges and nodes when zoomed out too fast. Turn out this was to do with the scaling going negative when zooming in/out too quickly.
I had to make minor changes to the JIT library (visualisation library) to fix it. I also made some minor changes to the weathermap such as the change of cursor styles, expandable height for graph view, and having a useful message when there is no graphs added.
This week I've been debugging a lot. Found out that some of the edges have had their data switched around. E.g. node A to node B had node B to node A's data. This was a very subtle bug as only some of the edges have this.
After a few days of debugging, it turns out that the arc tangent passed in to draw the edge was sometimes 180 degrees off because the method used (Math.atan) only returns a value between -pi/2 and pi/2. This cause the edges to 'flip' around (edge drawn reversely) which is the reason why some of the data was switched around.
I've also been experimenting different ideas with the 'hidden edge problem'. Currently, I've just hard-coded in some edges for the top level only, so that the edges are not hidden. I'll attempt to find a better solution later on.
This week I spent my time changing the way the traffic graphs are viewed. Previously, to view the graphs, an edge would have to be clicked and that will take you to a different page which shows the graph. Now I've changed it so that when an edge is clicked on, a graph will pop up as a widget without going to a new page.
The new graph widget can be removed, or closed so it can be re-opened later. I also started to debug some of my server-sided scripts as there were some minor bugs in them.
This week I modified the colour scheme of the weathermap so that it has the KAREN colours. I also added some logos and the traffic load legend on the weathermap. When I changed colour scheme, I noticed that most of the edges on the KAREN weathermap had very small ultilisation bands (since the network utilisation is very low) and therefore the weathermap looked very dull (i.e not much colours).
After talking to Richard and Mark Apperley, we decided to re-design the traffic utilisation visualisation a bit so that the edges have 5 bars. Each different level of bars will have a different colour associated with it along with a unique utilisation indicator. For example, 1 bar will indicate the utilisation from 0% - 1% for a link and the bar will be blue; 2 bars will indicate 1% - 10% and the colours of both bars will be green, and etc.
This was a good suggestion since we cannot really make fine distinctions between say 6% and 7% using the original linear traffic utilisation visualisation method and it also added some colours to the weathermap.
The second part of the week involved me adding features/functions that I was missing on my graphs such as y-label, display options, ability to choose max/average, and etc. I also got Brendon to hook up my graphs with live data; so now clicking on any edge will take you to some graphs.
There are still some minor bugs (mostly my php script that is returning the data) with the graphs which will need some work after the break.
This week I continued working on the interactive traffic graphs for the weather map. I managed to make it so that when an edge is clicked, it'll take you to a separate page so that the traffic graphs can be viewed.
Currently however; there are only traffic graphs for the link from Hamilton to the University of Waikato because the graph data is not hooked up with live data (the Hamilton -> UOW data is from the new RRD files that Brendon fetched me this week).
I also re-worked some of my scripts that are being used to get the utilisation data so that it will work nicely with the scripts that will be used to fetch live data for the graphs. I'm almost finished with the scripts that will be used to fetch live data for the traffic graphs, and hope that it'll be up next week.
Kept looking at generating better data for Chris to use in his maps. Wrote
some simple code to try to remove destinations that are obviously
international using latency measurements and hostnames. I err on the side
of caution though so there are still a few included that are quite likely
Got William set up with live access to json weathermap data so that he can
start testing his new weathermap in a more realistic fashion.
Some different visualisations of my smtp state machine graphs show some
nodes with a huge number of links, all at various stages of the protocol
that it probably shouldn't be able to transition between. These tend to
link to another node with a large number of transitions that also links
back, so tight loops are formed with many transitions. Direct loops should
be prevented from forming (they are being checked for at least), but it's
possible that check is missing a case or the loops are being formed as
side effects of merging others states. Need to keep wading through the
step by step merging output to see how this is coming about.
This week I managed to create PHP scripts to fetch data from RRD files to generate the appropriate time series plots for the graphs I'm producing. I also spent a good amount of time dealing with the way the graph is zoomed and panned (e.g. the change of the time labels, loading new "parts" of the graph when zoomed or panned, and etc).
I used aggregation for the plots when the user zooms out since there will be too many plots if this is not done, making the graphs slow and almost impossible to interact. Aggregating the plots proved to be quite hard and some more work needs to be done next week.
I also managed to hook up NetMapJs with live karen data. The live weather map looks good in most parts, though there are some missing edges on the top level. For example, an edge is missing from the Sydney to Auckland (when viewed from top-level). Once zoomed in, the edge appears.
The reason for this is because Sydney is connected to another group node (AKL_international) that is a parent to Auckland.
I still need to tidy up some code and slightly optimise the way the edges are updated (needs quite a bit of refactoring).
Karen Weather Map (NetMapJs version)
Note: For the Karen network, utilisation is very low, so most of the edges will show a very smallish length of green. Also, the current colour scheme used is not the same as the current Karen weather map.
This week I spent most of my time completing the php scripts that will be used to generate the necessary JSON/php configuration files for the NetMapJs version of the Karen Weather Map. The data in the configuration files that defines the map is based on the Karen topology data stored in a database, which is used to generate the current Karen Weather Map configuration files. This proved to be a little more challenging than expected since some of the information in the current karen weather map is hard-coded.
After seeing the graphs Chris made with a small number of destinations and
a single testing site, I found a few more sources of addresses that are
used within New Zealand (though not necessarily assigned to NZ
organisations) and started generating a new test data set. The new dataset
will have data for 28,000 /24s from four sources once collection
completes. Will need to have a look and see if this resolution is
Also put together a bunch of sample data and databases for William to use
with the new weathermap. It mimics the KAREN set up, so hopefully he can
use that to write scripts to deal with getting the data into the right
Spent some time picking up the smtp state machine documentation again,
trying to get some images of example trees to show the merging process.
I continued working on the php scripts I started last week after getting Brendon to install the missing RRD-php module. I then spent a good amount of time getting NetMapJs to update based on the data in the example RRD file I received last week.
I finished off the week by mucking around with some RRD files to get an understanding of the RRDTool. I also created some php scripts to fetch data from the aforementioned files, but it turns out the RRD-php extension support was not installed.