User login

Meenakshee Mungro's blog

22

Jan

2015

Implemented and tested the final features of the event database - updating groups and events as new detections are added to an event. This meant updating the DS score, magnitude, timestamp of the most recent detection, detection count and detectors that fired for each event. The group that the recently updated event belongs to is also updated: the end timestamp of the group and the severity (the highest probability across all events is used). Spent a lot of time testing since I kept double-guessing the correct addition and updating of events. Shane also plans on deploying the eventing code so that it'll be possible to see event groups, etc on the graphs, so this will be useful for debugging too.

Shane needed some accuracy values of DS with and without using magnitude as another source of input for use in his presentation. Started the rerun of the sample dataset against eventing with magnitude enabled and disabled.

15

Dec

2014

Spent those last 2 weeks implementing and testing event grouping and writing those events to a Postgres database on Prophet. Shane also finished collecting ground truth and generating an updated set of probabilities including the newest detectors (BinSegChangepoint and SeriesModeDetector), so I worked on calculating the accuracy of the new probabilities. Shane also calculated a series of probabilities for using the magnitudes as another source of input for Dempster-Shafer by calculating the probability that a magnitude from a certain bin ( e.g. 0-10, 11-20, 21-30, etc) is significant, a FP or undecided. I added those values to the DS calculations and ensured that the magnitude probs would only be used as a source of input once, i.e. the DS score is calculated for the detectors that had fired at the time, and then the magnitude is combined with the DS score.

Because of previous experience with indexing by streams, we decided to test the use of a separate table for each stream. I implemented the creation of new tables as they are encountered, adding events to each stream's tables and started working on updating the appropriate event after a detection occurs. Testing was quite sluggish, so will need to bug Brad about running Postgres on spectre instead of prophet.

02

Dec

2014

Rewrote some of the eventing script to handle multiple streams as input and store their data independantly since it had only ever been tested with a single stream at a time. Shane introduced some new magnitude calculations and I ran anomaly_ts with the ground truth series as input to get the new magnitude scores. Plotted different versions of those values and were reasonably satisfied with the results, although the slow duration/high magnitude/FPs were still there. Took a break from playing with magnitudes and moved on to event grouping. Spent some time rethinking the grouping strategy and started working on event grouping by target with multiple series.

26

Nov

2014

Spent the last 2 weeks working on integrating event magnitudes into eventing. Tested a simplistic magnitude that used the latency changes reported by some of the detectors (Plateau, Mode, Changepoint) and the results looked somewhat promising. In theory, significant events would have a high magnitude and insignificant events would have a lower magnitude, but this wasn't always the case. Because a lot of the insignificant events had big changes in latency but were very short in duration, their magnitudes were high.
Took a break from the magnitude and had a chat with Shane about the next step of the project - storing the events (group of detections) and their probabilities of significance into a database. We also discussed event grouping of different streams and the criteria for grouping: by source or target Generated an initial schema, and spent a good amount of time on reading up on non-relational DB solutions since there were concerns that relational wasn't the optimal option for this scenario.
Shane implemented 2 new detectors(BinSegChangepoint and SeriesModeDetector) and I added them to eventing, currently using bogus probabilities for Demspter-Shafer until the new detectors have been tested fully and are thought to be free of bugs. Testing showed that the addition of two new detectors improved the accuracy of detecting significant events, but there was also a very small increase in FPs.
Spent some time testing magnitudes as a secondary condition for an event being significant.

04

Nov

2014

This week, I focused on using the magnitude of a change as another indication of severity because the results of using AMP-specific probabilities were not as favourable as we expected (this can be explained by the fact that DS does not work on events where only a single detector has fired). Spent several hours looking for relevant literature, but the papers I looked at were not especially helpful or completely unrelated. I took a break from reading papers and moved on to graphing the relationship between magnitude and severity. I used the Plateau detector's absolute and relative latency changes and the TEntropy-Stddev detector's TEntropyChange as metrics and plotted them against the severity score of the matched ground truth group. Found that TEntropyChange was useless for this purpose, but the absolute latency change showed promise: there was an easily identifiable threshold for identifying significant events but there were also some outliers that were less desirable (insignificant events with a high magnitude of change).

22

Oct

2014

Spent the first couple of days rewriting/restructuring the eventing script since it was a real abomination of a script (atleast the functions had been well documented/named so it was not too painful). Also rewrote the probabilities script so that each time series subtype (e.g. AMP ICMP/rrd Smokeping) would be a separate module and have its own sets of probabilities. This also makes it easier to add new modules later on. Using the AMP-specific probabilties, I re-ran anomalyts using the original series used for the ground truth and got a list of event groups and their significance ratings. Then, I attempted to match the output produced by the eventing script (i.e. event groups and their significance probability) and the original manually classified ground truth. In theory, most of the detectors' behaviour should have been very similar to those found from the ground truth since they are using the exact same latency values, but for some reason there were missing/additional events. This was expected behaviour for the Changepoint/HMM Detectors, but there were some differences with detectors that relied on the Plateau algorithm (Plateau, TEntropy-Stddev, and TEntropy-Meandiff detectors). Spent the remainder of the week comparing events from the two sources and flagging those that needed to be investigated.

14

Oct

2014

Started working on using a data fusion method(Dempster-Shafer) with AMP-specific values. The AMP ground truth dataset collected as part of my masters project was used to calculate a set of probabilities for each detector based on the variability of the time series during a particular event group. These probabilities might not be final because of the introduction of a new variability (MultiModal), so i will need to collect more data later on, especially for MM streams. Wrote a script that parsed a time series log file and sent the latency values to anomaly_ts so as to test anomaly_ts against the ground truth data since it is no longer in the active database. I also generated graphs leading up to and 2 hrs after each event in the ground truth to determine if any of those streams were multimodal, but no luck so far. This also means that the probabilities for Noisy and Constant will remain unchanged.

03

Apr

2014

Once the eventing script had been tested properly, I moved on to including the option of producing a CSV file out of the results.
Afterwards, I wrote a Python script to read in the manually classified events (ground truth) and the results from the eventing script. Then, the entries are compared and matched to produce a list containing the following info: ts event started and severity score(from ground truth), fusion method probabilities (from eventing results, which include DS, bayes, averaging and a newer method which counts the number of detectors that fired and allocates an appropriate severity score), and finally the timestamp at which point each fusion method detected that the event group is significant. This is useful to determine which fusion method performs best (e.g. fastest at detecting significant events, smaller number of FPS(ground truth says not significant, but fusion method detects significance), etc.
The script performs better than I expected after testing (e.g. 46 event groups from the ground truth and 42 matched event group/probability results from the eventing script when tested with Google's stream). The remaining unmatched events will need to be manually sorted out, so hopefully the script will perform as well on AMP data.

25

Mar

2014

Spent a fair bit of time finishing my detector probability script and making it look less awful. Then, spent the res of the week updating the eventing script to use the new detector probabilities and also updated the initial Sig and FP probabilities used by the Bayes fusion method. Then, added options to the eventing script to allow outputting the values of the different fusion methods, detectors and event grouping methods. The remaining time was spent testing the new changes and fixing a couple of minor bugs.

18

Mar

2014

Last week was a bit slow, spent ages double-checking the values in the spreadsheet and making sure the correct ell ranges and formulas were being used. Found quite a few mistakes, so it wasn't entirely a waste of time. Then, spent the rest of the week updating the detector probabilities script with the new values. This took forever too since there are several detectors and two methods, each with their own probability values and event grouping, etc.