User login

Stephen Eichler's blog

11

Sep

2014

I started on preparing the slides for the internal PhD conference. I made some new slides for Megatree and prepared some graphs from the CAIDA Doubletree data.

I made a repository for the PAM version of the churn paper to share with Richard. The graphs titles have been removed and put in the text of the paper instead. It currently needs to be shortened by one more page. I also found some journals which would take the full length paper and some that would do so after it had been presented at PAM. These are a possibility.

I downloaded the black hole detector data and analysed it for black holes in load balancers. This has also had the address files updated and then been restarted.

I found a few more bugs in the non event driven Doubletree simulator, so this has been restarted after carrying out the fixes. I am gradually getting a good data set for the three simulation types: Doubletree, CAIDA Doubletree and Megatree.

03

Sep

2014

After discussion with Richard I made initial steps to put the churn paper into PAM submission format. Initially this resulted in 22 pages and the limit is 10 pages. After removing two sections and related discussion, along with graph captions and implementing the use of sub-caption formatting the result was 11 pages.

An alternative to the severe restrictions of PAM is the "International Journal of Computer Networks & Communications" which has a limit of 20-25 pages and is published bi-monthly. I will need to find out more about this Journal and its suitability.

In the non event based double tree simulator, at present the method used for many sources to make use of local stop sets runs for excessively long periods of time. It is already making use of text file lists of source addresses that occur more than once, to limit array sizes of these values. So far I have not thought of other ways to make further improvements to the run times. Fortunately, the benefits of local stop sets are expected to be small with only a few runs occurring from each source. Furthermore it may be possible to examine the savings in a few thousand cases for varying numbers of traces from a given source between 2 and 12, and predict the overall savings for hundreds of thousands of cases.

28

Aug

2014

Summaries from the three runs of the black hole in load balancer detector have been compiled. Each run covers about 5000 load balancers and a total of 6 very short lived cases have been seen, along with one case of repeated detection and 6 cases of long lived discontinuity. In each case the non load balancer nodes remained unchanged before and after the hourly Paris Traceroute runs.

I have been getting some Doubletree results for small numbers of vantage points using non event based simulation. This is useful for debugging purposes as the turn around time is quick. Now that debugging is completed for most of the analysis modes, I have started some full length runs which should take about two weeks in most cases. I am still deciding if I can do 1, 2 and 10 stage analysis for many to few probing. If I can find a way to make it run in a short enough time, I will set that going as well.

20

Aug

2014

Some proof reading of the churn paper has been carried out and a few minor modifications have been made.

I have limited the number of vantage points used by the non events driven Doubletree simulators to five. This facilitates short run times and the opportunity to debug the programs, as do some previous measures to this end as well. These included using an array of arrays instead on a single very long array, and generating a data structure containing probe data once instead of once for each analysis call. I am starting to get some good results now. The full size runs still take about two weeks, however not a lot of memory is used so Wraith can run a lot of them simultaneously. Also because Spectre scratch is down I have moved all my analyses to Wraith.

06

Aug

2014

Work has been carried out on the introduction of the churn paper. In particular the beginning of the introduction now states the intent of the paper and explains the relevance of the following topics that are discussed. Version control file permissions have also been set up to allow Richard Nelson to commit changes to the paper.

The next round of black hole detector data has been downloaded and processed. This time there have been some transitory black holes and some that persisted longer than the data collection. The next round of data collection has been started.

With the MDA data based Internet simulator for Megatree, the control packet analysis using many sources to few destinations where each source occurs more than twice has been upgraded. In the current run each window of sources receives a full set of global load balancer data as it begins to run Megatree. This is slightly wasteful as some nodes have already received some of this data, so a more complex upgrade has also been developed where each source node only receives the same global set data once. To achieve this a pre-run run to identify the sources in each window is carried out, as the program cannot otherwise know in advance what these will be when running only a single run. The data are stored as lists of grouped source addresses in text files.

30

Jul

2014

The Doubletree simulation using the same data files as the event based simulator from Tony McGregor took 2 weeks to run. It turned out that the stored value for number of probes sent was incorrectly recorded as zero in all cases, so this value was determined using the hop information recorded in the traces. Four runs were started using this fix, to run four categories of simulations using this data. It should once again be a two week cycle.

The data from the Megatree simulator is now quite good, however snapshots for stepwise increases in source count are still incorrectly calibrated. Further steps have been taken to remedy this and another run to gather this data has been initiated.

The black hole detector has finished another cycle and so further post collection analyses are required. Steps are being taken to confirm that the transitory black holes found, withstand the scrutiny of warts dump data evaluation manually.

24

Jul

2014

I have continued to analyse the results from the fast mapping black hole detector. It looks like counting hop distances in the MDA traces with more than one node is the easiest way to determine if a given stop point from Paris traceroute is inside a load balancer. A small number of transitory black holes have been found in the first lot of load balancer black hole data from Planetlab.

Some of the non event based simulators have long run times now. I need to see if there is a way to speed them up, perhaps using pre compiled lists of addresses to avoid the need for sorting. However the sources window analysis is running in a short time because it only uses the many set where the sources occur more than twice. Last time I ran this I found a bug in the calculations so I have fixed that and set it running again. At this stage I can't work on the event based simulator because I need the results from these former simulators first, and also they cannot fit on the same computer at the same time.

16

Jul

2014

The results from the black hole in load balancer detector on Planetlab were downloaded using an automated process. Warts analysis was also carried out in an automated fashion. The results were promising as once again there appear to be black holes occurring in load balancers. In some cases these are very transitory i.e. Occurring in only one Paris trace. I probably need to find a way to start counting these different discontinuity scenarios in this ICMP probed data.

I have written code to analyse windows of many sources. In particular I am only using sources that occur more than twice. A window is a number of sources that run simultaneously, before another group of the same number run. This means that the Doubletree and Megatree analyses running from a given source are likely to see come of the same nodes or load balancer more than once. This is like the situation that we expect to see in Atlas and so it is a desirable simulation. An advantage of these modes is that they are likely to finish fairly quickly.

There are a number of non event based Doubletree and Megatree simulations running that are taking some time. Hopefully they are bug free so that they will only have to run the once. I have saved some time by writing files of the 'many' set destination addresses that occur more than once and more than twice. This avoids a lot of sorting when each address is added to a data array, one by one, when this information is used in analysis.

09

Jul

2014

The black hole detector is running and will need to have the data sets downloaded soon. This collects load balancer data at the beginning and end of the first drivers run and at the end of the second drivers run. The second driver is only triggered for a given destination address, when Paris traceroute runs between the MDA load balancer Traceroute runs stop short of the destination. Because I am running the detector on 15 Planetlab nodes I am looking for ways to automate repeated download and analysis tasks.

Because Spectre and Wraith were shut down on Saturday, I restarted the scamper warts analysis simulator jobs that I had running. This however allowed me to adopt some recent changes to these analyses. In particular I have been adding analysis of windows of many sources. In this mode it is not always possible to get local results as I am not able to use the same sources repeatedly on a consistent basis. These many sources are actually the destinations in the warts data sets, as the data is used in reverse direction.

The scamper warts analysis based Internet simulator for Doubletree has been adapted to the data set that the event based Internet simulator uses. This was collected by Caida with Paris Traceroute using ICMP probes. It is currently being debugged and once this is done a run will be carried out.

It now time for my six monthly report. This has been drafted and loaded onto the website. Once my chief supervisor has had a look at it, it will submitted.

02

Jul

2014

I spent a few days setting up the black hole in load balancer detector on Planetlab. Because there are two scamper drivers used, these are required to write and read the same file, and this did not work initially because of the permissions that are set automatically for a root owned process. I eventually changed from a linux managed service to using a script where the drivers were no longer root owned processes, and could conveniently open the files that they had created. The black hole detector looks for Paris Traceroute traces that are shorter than the original MDA traceroute and runs further Paris data collection from the second driver. I intend to run 20000 destinations from 15 sources every week or so.

The Doubletree simulator that uses MDA data as a source of simulated Traceroute data ran successfully, and was then extended to cover more factor combinations and restarted.

The Megatree simulator that uses the same data has had some many to few cases built in and is running again. It has run for somewhat longer than before, nearly a week so far. It seems likely that it will not be run in full but on selected factor combinations as they are developed and debugged.

Two programs to count destinations in warts data have been written and set going. One uses the MDA data and the other the CAIDA traceroute data from the website. These count destinations of traces including if the destination is repeated and how often this occurs.