User login

Brad Christensen's blog

28

Jan

2014

Continued working on splitting out IPv4 and IPv6 data on the matrix. Cells in the matrix are now diagonally split in two by default to show IPv4 on the left and IPv6 on the right. I've added the option to show only IPv4 data or only IPv6 data, though clicking on a cell will still take you to the graph for IPv4/IPv6 irrespective of the view you're in. I tweaked the severity colours and added a legend for them that changes based on the current test type (relative/absolute latency, loss, hop count).

More refactoring in the matrix, fixes for matrix popovers and more general IE8 fixes, particularly to get modal dialogs working. Added helpful tooltips to radio buttons for aggregation in the modals to explain what they do.

Changed the colour of event lines on smokeping graphs with only one series. The series would be drawn in black which would clash with the grey of event lines, so I set event lines to be the same colour as their event markers in this situation.

Finished up the week by adding some performance improvements to smokeping graphs that hugely improve software rendering of the canvas (particularly in Firefox on Linux). The canvas is always antialiased so rounding our drawing coordinates to nearest integers prevents unnecessary antialiasing and results in smokeping summary graphs being drawn instantly (which is mainly where our problem was). They aren't noticeably reduced in quality because the diagonal lines connecting points are still antialiased, so we benefit greatly from this (if anything I think the sharper edges look better). I also rounded the drawing coordinates on event markers and the rainbow traceroute graphs for drawing clarity, and I separated out some of the data processing that was in the smokeping drawing code which will mean that in future if we still require better performance, it would be possible to move more of the code into a separate thread.

21

Jan

2014

More work on improving the consistency and quality of JS and CSS in the AMP web interface this week. I fixed lots of minor bugs to get everything working in IE8, starting with the most basic ECMAScript dialect differences such as removing extraneous trailing commas and polyfilling some non-existent functions, then moving on to the URL handling code, which previously only took into account how URLs are rewritten in HTML5 browsers (which can write over the current URL). HTML4 browsers such as IE8 and below cannot rewrite the URL so they instead append as a hash/fragment a URI relative to the base URL (where the base URL is that of the page first directly accessed, i.e. typed into the address bar). HTML4 and 5 are both now handled by checking for the existence of a hash and resolving the fragment if it exists, otherwise resolving the base URL. I broke my work on the history code down into a few stages, first tackling the matrix, graph pages, then modal dialogs. I fixed up the rest of the history state handling code afterwards to restore state while moving forwards and backwards between pages, which hadn't been hooked up before.

Another common issue worth mentioning is that some browsers, particularly IE8 and below require explicit object.hasOwnProperty() checks when iterating through object properties with code like for (var x in y) { do things }, which would become for (var x in y) { if (y.hasOwnProperty(x) { do things } }. Instances where these checks weren't happening created several not easily detectable bugs, because IE will additionally iterate through the object's prototype (and debugging IE is a nightmare).

To finish off the week I ripped apart the matrix and removed its dependence on the rather bloated DataTables JS library, added documentation and did some much needed refactoring and tidy up, then started working on splitting up cells to show IPv4 and IPv6 stats.

13

Jan

2014

This week I wanted to dedicate some time to cleaning up the fragmented amp-web interface to improve the consistency of CSS and markup across the site, and to remove unnecessary JS libraries, in the process determining which libraries best suit our purposes in cases where features overlap.

As a first step I included Bootstrap's CSS globally and rewrote the rest of the global stylesheet around it, restructuring hacky CSS that relied on (inline) markup. I would prefer to include only the Normalize reset and Bootstrap's class-based CSS rather than the base CSS that styles other elements, which I might investigate at some point, but for now everything works fairly well. Including Bootstrap globally broke the Matrix, whose CSS definitions overlapped with Bootstrap's (so I fixed this temporarily by renaming the affected classes).

So after breaking the matrix, I spent a lot of time cleaning it up (albeit mostly because I wasn't aware it existed in the first place). The Matrix used jQuery UI to display tooltip-style popups, which I replaced with a similar feature (popovers) in Boostrap. This took a bit of time and more rewriting than I'd expected as the JS for instantiating each is very different (particularly as the popovers are intended to appear on click rather than on mouseover), but it worked eventually and I managed to streamline some of the Matrix code in the process. I also replaced the matrix's jQuery UI tabs with custom ones, and that allowed me to remove jQuery UI and its CSS, the JS library cssSandpaper (which had been used for backwards compatibility that wasn't really relevant), and its dependency libaries cssQuery, EventHelpers, sylvester and textShadow.

I added a CSS hack to fix graphical glitches that were sometimes produced when rendering rotated text (the matrix headers). It seemed to only occur on Voodoo, but as well as preventing the flickering issue, the fix also looks to have improved legibility on all platforms.

Finally I spent some time integrating the traceroute map with the latest changes and updated it to use real data. It was interesting to see what a difference this made to the summary view, whose highly aggregated data is no longer useful for representing unique paths or for being able to see where paths change. Will have to look at how to best address this next.

17

Dec

2013

As per discussing with Richard and Brendon, I aimed this week to produce a more fully connected version of the traceroute graph, for the purpose of visualising the networks represented by interconnected routes more accurately. I knew that in order to do this I needed to abandon my existing tree structure and instead represent the data as a graph, so I spent some time reading about drawing algorithms for directed graphs, which led me to reading further into hierarchical graphs, which appeared to be exactly what I was looking for. Hierarchical graphs are typically drawn according to Sugiyama's framework, which breaks the process down into 5 stages, each requiring the selection of an algorithm meeting the requirements of application.

Fortunately I came across a Javascript library called Dagre, which is used together with Graphlib to lay out coordinates for hierarchical graphs and usually with a rendering library such as D3 or JointJS to draw the graph. I wrote some quick rendering code to draw the graph to the Flotr canvas instead and it worked excellently, so I continued and quickly produced a desirable graph and mouseovers.

Of course, multiple stages of the layout process are NP-hard or NP-complete, so drawing these graphs is slow. Dagre would hang the browser for potentially several seconds before anything could be drawn to the canvas.

As such I investigated Javascript web workers (essentially threads, which work in the most recent browsers) and separated the data processing + layout step into a worker, resulting in a much more responsive experience, however the graph is now blank for a few seconds before the data processing catches up and draws something, which I would like to improve next week.

06

Dec

2013

I've been creating the traceroute map this week (the first half spent initially coding and the second half spent fixing it to work with a much larger real data set). Instead of trying to port the existing PHP or RaphaelJS (SVG library) code, I decided it would be easier to roll my own for Flotr2. I've had a fair amount of success so far, and my graph now looks like this image:

Traceroute map

You can also hover over a path to highlight the entire path and show information about it, and hover over an individual hop to highlight all hops to the same host.

Colours represent unique hosts and there are certain conditions governing path divergence and convergence so that it remains clear to the human eye which path is which. Implementationally the data structure used is a tree in which each node is a path who has zero to many branches. Therefore one constraint is that a path will only ever join up with its root node (the same path it diverged from) and it will currently only ever diverge and converge a maximum of once each. Each alternative path is also drawn on a new line for clarity. I think these constraints help to strike an effective balance between accurate network representation versus visual complexity.

29

Nov

2013

Further to my simple event marker implementation (showing "signpost" markers above the plot area of a graph, along with the vertical lines currently used to mark events), I set out to determine the best way to draw these markers and lines so that they would be clearly visible but not so much as to obscure or draw attention away from the plot. One particular challenge was to be able to guage the severity of the event at a glance, which is currently done by colour (yellow/orange/red) but to avoid conflicting with colours used by series on the plot. One solution for this I tried was to prevent graph styles using any colours in a particular hue range (so as to avoid yellow-red which would be used by the events) but due to both the events and plot being in colour it still made it confusing to tell which were which.

After much very scientific experimentation in the field of colour perception I found that plots drawn with the same saturation or lightness appeared to be related, and changing the hue divided this related group further; it was easiest to tell event markers apart from data when one of these groups was drawn with a different level of saturation, regardless of hue. The obvious conclusion therefore was to draw one group at 0% saturation and the other at 100%, for maximum differentation. As such, the data plot is drawn at 100% saturation with any hue, the event lines are drawn grey and the event markers are still drawn with colour where they will not detract from the plot.

For the rest of the week I have been looking at reproducing (to some extent) the AMP traceroute map in Cuz using Flotr2. I've been familiarising myself with old code and researching alternative representation methods, such as directed graphs, as the data we want to visualise is not necessarily a tree, although we want to view the data in a tree-like form and we may not want the nodes to be as interconnected as they would be in a graph. While I've been thinking about this I've created a graph type that can draw static plots while moving through time and I'm going to start drawing things in it next week.

22

Nov

2013

I spent this week continuing work on the rainbow graph, implementing the following features:

* Mouse tracking (no conflict with mouse tracking for events)
* Option to measure latency instead of hop count
* Optional minimum height for points/bars to give improved readability at the expense of a small degree of accuracy
* Out of order data points will be set to the minimum height and stacked on top of previous points (e.g. in the case of a traceroute where the second hop has a latency coincidentally less than the first hop)
* Improved caching for more efficient traversal of data points in many cases, particularly mouse tracking

I began looking at how events could be better marked on graphs given that as lines, they may currently be easily confused with data points depending on the graph type and density of points. I have been pursuing the idea of drawing event markers above the plot area (along with the existing lines) and I think this seems to be an effective way to tell at a glance where events fall. I've implemented this using a plugin, which can draw directly to the canvas as opposed to only the plot area that is normally accessible to a graph type. A convenient side effect of this is the ability to draw events either behind or in front of the data points because plugins can intercept beforedraw and afterdraw (Flotr) events. Events are currently drawn behind data, but the rainbow graph is one example of where they may be more appropriately drawn in front. In this implementation I also rewrote how events are processed to favor caching, and addressed a minor bug in one of Flotr's internal plugins.

18

Nov

2013

This week I learned some Python, reading and working through some of Brendon's exercises with the AMP REST API and then moved on to looking at graphs in Cuz. I have since been working on creating a "rainbow" traceroute graph type for Cuz with JS and Flotr2, which is now mostly completed aside from some minor tweaking and checking to ensure the graph will still draw correctly when given errors or null data etc. The library's lacking documentation made coding feel like a slow process so I intend to document my files fairly thoroughly starting next week.