User login

Weekly Report -- 7/10/2016

10

Oct

2016

Influx on a skeptic workload was still not performing as well as I'd like, so I've decided to take a different approach. I removed all of the pre-aggregation that we were doing for graph drawing as this was requiring too much computing power for little real gain (i.e. most of the aggregated data would never be used). However I have added matrix-specific aggregations (i.e. minutely and hourly aggregations of the data columns used by the matrix) to try and speed up the matrix loading. These aggregations only look at very recent data and are retained for a short period of time so they are much less onerous to perform.

Initial testing seems to suggest that the matrix loads about 3 times faster than the old postgres system and memory usage by Influx seems to be not ridiculous so hopefully this will prove workable in the long term.

Continued working on my suffix-tree for finding repeated sequences in syscall traces. Initial results are promising but there are a couple of outstanding issues to deal with, mostly related to recognising two sequences as being variants or shifts of one another. If I can do that, I might be able to get close to automatically creating a FSM for those common sequences.