A remote ATM network monitoring system
Ian Graham, Murray Pearson, Jed Martens, Stephen Donnelly and John G Cleary
Department of Computer Science
University of Waikato
Private Bag 3105
phone +64 7 838 4136
fax +64 7 838 4155
corresponding author: Professor Ian Graham
This paper describes a low-cost ATM monitoring system, hosted by a standard PC. The monitor can be used remotely. returning information on ATM traffic flows to a central site. The monitor is interfaced to a GPS timing receiver, which provides an absolute time accuracy of better than 1 msec. By monitoring the same traffic flow at different points in a network it is possible to measure cell delay and delay variation in real time, and with existing traffic. The monitoring system characterises cells by a CRC calculated over the cell payload, thus special measurement cells are not required. Delays in both local area and wide-area networks have been measured using this system. It is possible to measure delay in a network that is not end-to-end ATM, as long as some cells remain identical at the entry and exit points. Examples are given of traffic and delay measurements in both wide and local area network systems, including delays measured over the Internet from Canada to New Zealand.
ATM; performance measurement; Internet performance; network delay; quality of service
Areas of interest
Testbeds and measurements; high speed networking; network traffic analysis and performance measurement.
At present, measurement of the performance of wide area ATM networks is a complex and expensive business, requiring equipment costing in the region of $100,000 installed at each monitoring site. Traditional methods of network performance monitoring also depend on the introduction of measurement cells into the network, which disturbs the very conditions that are being measured.
This paper describes a low-cost and non-invasive method of measuring traffic flow, cell delay, cell delay variation and cell loss in ATM networks, including networks that are geographically large. It is also possible to make delay measurements on networks that are not end-to-end ATM, such as the Internet, if the encapsulation of IP over ATM is the same at both ends of the measured portion of the network.
The technique requires the installation of PC-based ATM monitors at measurement sites in the network, each monitor returning information on the cells it receives back to a central analysis site. If this return flow of information can be carried on a separate network, then the measurement process itself imposes no extra traffic on the network being measured.
The non-invasive method of measuring cell delay relies on the correlation of traffic at different sites by the identification of individual cells in existing traffic. The measurement systems calculate a CRC over the payload of each ATM cell received. This acts as a cell signature, which does not change as the cell moves through the network. If these cell signatures, and arrival times, are returned to a central site the cell flows can be matched and the time delay calculated for each matched cell.
Figure 1. Remote monitor overview
The accurate clock required for cell delay measurement is provided by a GPS time receiver at each measurement site. These receivers have a time accuracy (relative to UTC) of about 100 nsec, and cost less than $1,000 per unit. An overview diagram of the remote monitor system is shown in figure 1.
2 The remote ATM monitor
The remote ATM monitor is a low-cost PC-based cell capture device. Cells are captured and their arrival times recorded by a specially designed network interface card, know as the Dag, which is described in [1, 2]. This card consists of an ATM physical layer interface, a large FPGA, an ARM RISC processor, and a PCI bus interface.
The card is linked into the ATM network by an optical coupler or resistive matching network which non-intrusively duplicates all the cells. The monitor hardware and software are capable of capturing cell headers, payload CRCs and arrival times at full OC3 rates, that is about 300,000 cells per second. However, the volume of data that can be sent back to the central analysis site is limited by the bandwidth of the return link. The Dag can filter cells by VPI and VCI, so that only subsets of the cell flow are returned, and data compression techniques in the PC are used to reduce the return data flow to a minimum.
Serial cell data received from the ATM link is assembled into cell buffers
that contain both the cell payload and header. At the start of each cell
a time-stamp is generated from the local 4 MHz clock, and this is also written
into the cell buffer. During cell reception a 32-bit CRC is calculated over
the cell payload, this CRC is written into the cell buffer when a complete
cell has been received. The on-board processor can read all of the cell
data, or just a sub-set, such as cell header and arrival time-stamp. The
heart of the cell capture card is a large FPGA (10,000 gate equivalents),
which can be programmed from the PC. An outline diagram of the normal FPGA
configuration for cell arrival time measurement is shown in figure 2.
Figure 2 Dag FPGA standard configuration
The GPS time receiver, a Trimble Palisade , produces a time pulse each second, synchronised to UTC with an accuracy of 100 nsec. Shortly after each time pulse an ASCII time information packet is output by the receiver, this gives the UTC time and date for that second and other receiver management information; this time packet is received directly by the PC through a serial port. At the leading edge of the time pulse the local clock is loaded into the GPS register in the FPGA, and an interrupt is generated to the on-board ARM processor. This processor can in turn interrupt the PC and copy the GPS time register value into PC memory. The on-board clock signal, unfortunately drifts by many microseconds in the one second intervals between GPS pulses. However, careful interpolation allows this drift to be corrected to give a final accuracy of about 250 nsec compared to UTC.
The remote monitor software running in the PC acts as a server which can connect to an analysis client by TCP/IP. Thus the client may be geographically remote from the monitor server, making its connection over the Internet. Also, an analysis client may connect with two or more remote monitors, in order to correlate cell streams measured at different points in the network.
The monitor software receives buffers of cell data - cell header, time stamp and CRC - from the Dag at intervals of one second. These data are compressed and transmitted to the client together with the GPS register value and information from the GPS time packet that will allow the client to correct the cell time-stamps to UTC. These corrections are not made in the monitor. The monitor software can also receive cell filter requests from the client, and load these into the Dag. The monitor has been designed to run unattended, and has been implemented under both DOS and LINUX.
3 Cell payload CRC and cell matching
In order to measure cell delay we need to be able to identify cells at the entry and exit points of the network we are testing. Conventionally this is done by injecting special test and measurement cells into the network, these cells have some characteristic that distinguishes them from cells in the normal traffic. However, this technique has two disadvantages. Firstly, an active device is required to inject new cells into the network, secondly the very injection of cells into a network may change the parameters that we are trying to measure. Therefore it would be better if we were able to use the existing traffic in the network for measurement. To do this we need a way of characterising cells in the existing traffic flow, and this must be based on the cell payload, as the cell header will change as the cell moves through switches. It is not practical to use the entire cell payload; there would be too high a volume of traffic back from the monitor to the matching station, matching would be slow as all twelve words from each cell would have to be matched, and the network user might rightly be concerned about security if their traffic was being copied and re-transmitted. Thus we have investigated the used of a 32-bit CRC calculated over the cell payload to characterise the cell. This has the advantages of being easy to calculate in hardware, of reducing the data to be transmitted to the analysis site from 48 bytes to 4, and of being an irreversible operation, in that the cell payload cannot be recovered from the CRC.
We have chosen to implement the AAL5 CRC32 in our hardware. This CRC is calculated in parallel with cell storage in the buffer as each 32-bit word is received, and the CRC is transferred to the cell buffer at the end of the cell time. The CRC transfer adds only one cycle of the FPGA clock (24 MHz in this implementation) to the time taken to receive a cell.
In order to match one cell stream against another in the analysis software, and thus measure delay through a network, each payload CRC from the cell stream at the entry to the network is compared with CRCs recorded at the exit point. This comparison is carried out for exit cells within a time window relative to the timestamp of the cell recorded at the entry point. The size and position of this window depends on the circumstances of the measurement.
For example, if we are measuring delay through a switch that has a minimum delay of 20 msec, the window will initially be set to start at say time 0, and may run forward to 1 msec. If we are measuring the delay between two points on the Internet separated by a light-speed travel time of 40 msec the window will start at 40 msec, and go forward to about one second. The width of the window is a compromise between increasing the probability of correctly matching cells delayed for a very long time, and reducing the probability of a false match.
False matches can occur if two cells with the same payload CRC are transmitted during the width of the time window. To study the distribution of CRC values we have recorded several million cells from NFS traffic between a workstation and its file server. We have found that the overall number of duplicate CRCs is very small, less than one per cent of the total. However, the important statistic for judging whether or not cell matching will be accurate is the number of duplicates that occur within the time window used for matching. Investigations into this statistic are still proceeding, using recordings of different types of network traffic.
However, at present we conclude that the AAL5 CRC is suitable for cell matching, and that there is a very low probability of false matches for delays in the region of 0 - 1 second.
In this section we present a selection of the types of measurements that we have been able to generate using the system. Many more experimental results will be found on our Web site .
The present version of the analysis software runs on a LINUX system, and is written in a mixture of Java and C. Its main functions are to set up communications with the remote monitors, to set up filter parameters, and to request the return of data on filtered cell streams. The sofware can also display graphs of traffic and delay histograms in real time.
The cell timing information in the received data stream is corrected to UTC, using calibration information also carried in the data stream. The corrected data can be written to disk, or may be matched with a cell stream from another monitor to calculate delay and delay variation statistics. Development of analysis software is continuing, with a web-based version as a high priority.
ATM traffic measurements
An initial requirement of the system was that it should be capable of collecting ATM cell arrival times over long, continuous periods, to enable us to carry out statistical analysis of ATM traffic patterns. An example is shown in figure 3, where NFS traffic between a SUN server and a workstation has been recorded. Figure 3a shows a record over a period of about two hours, displayed as counts within 40 second bins. Figure 3b shows the same traffic, but with 1 second binning, for a period starting at about 3900 seconds into the record.
Figure 3a/b Traffic measured on a NFS server to workstation link
Switch delay measurements
To demonstrate the timing accuracy and resolution of the system measurement s were made of time delay for cells passing through an ATML Virata switch. The cell stream used for these measurements was again NFS traffic from a server to a workstation. Figure 4 a shows the delay histogram for the switch under the relatively lightly loaded condition produced by typical NFS traffic.
Figure 4a Cell delay through a lightly loaded switch
Figure 4b Cell delay under heavier loading
In figure 4b a heavier switch loading has been simulated by sending the same data twice through the switch. These diagrams illustrate that time resolution of the system, the 2.75 msec quantisation of delays due to slotting of cells in the OC3 frame is clearly visible.
Delay measurements over OPERA.
Measurements were made over an approximately 400 km path from the University of Waikato, in Hamilton, to a Telecom New Zealand site in Wellington, New Zealand. The main part of the circuit was carried over E3 links (34.5 Mbps) which are part of the OPERA experimental trial being run jointly by Telecom New Zealand and a consortium of New Zealand universities and Crown Research Institutes .
The traffic flow for the experiment was a copy of live NFS traffic flowing
between a Sun server and a Sun workstation. The traffic was copied by creating
a multicast circuit within the ATM Ltd Virata switch that connects the workstation
to its server (figure 5).
Figure 5 Experimental setup for Hamilton -> Wellington delay measurements
The histogram in figure 6 shows a typical result, where 196,346 cells were matched in an experiment lasting several minutes.
Figure 6 Histogram of Hamilton to Wellington cell delays over OPERA
Intercontinental Internet delay measurements
The remote monitor system has recently been used to measure delay over the Internet between the universities of Calgary (Alberta, Canada) and Waikato (Hamilton, New Zealand) a great circle distance of 12,000 km, corresponding to a time of flight of 40 msec at the speed of light.
As only a unidirectional route could be established, from Calgary to Waikato, the measurement were made on UDP traffic. The test traffic was generated by a workstation in the computer science department at Calgary, and routed via WNet to another workstation on campus. This workstation acted as a gateway to the Internet, and the traffic then followed a path across the Pacific to Australia and onwards to New Zealand. Within Waikato University the traffic was routed into our ATM LAN and re-encapsulated as classical IP over an ATM PVC.
Most measurements were made on UDP packets containing 400 bytes of data,
which were encapsulated as 10 ATM cells per packet. It was observed that
the CRCs of the first and last cells in each UDP packet could not be matched,
although all intermediate cells were matched. When UDP/IP is encapsulated
by AAL5 and split up into the cells the first cell contains the UDP and
IP headers. Most of the fields in these headers remain constant as it passes
through the IP network, however the time to live (TTL) field is updated
as it passes through each router. The change in the TTL field affects the
payload CRC of the first cell, and also the AAL5 CRC in the last cell, and
thus the payload CRC of that cell. Figure 7 shows the measured results for
approximately 680 UDP packets, for which 5412 cells were matched.
Figure 7 Histogram of delays over the Internet from Calgary to Waikato
In this paper we have described the design of a simple and low-cost system for making cell arrival time measurements on ATM systems. Despite its low cost the system achieves very high time resolution and absolute accuracy, and is capable of a range of measurements that would otherwise require expensive test gear.
Work is continuing at Waikato on higher performance versions of the monitor hardware and software, and on the introduction of a system for monitoring uni-directional Internet delays in the Pacfic Rim/Asia region..
We would like to thank the New Zealand Public Good Science Fund, Telecom New Zealand, and the New Zealand Lotteries Board for their generous support of this project.
1. Graham, Ian D and Cleary, John G Cell level measurements of ATM traffic in Proceedings of the Australian Telecommunication Networks and Applications Conference, 1996, Monash University, Melbourne, pp 495-500.
2. Graham, Ian D, Pearson, M, Martens, J and Donelly, S, Dag - a cell capture board for ATM measurement systems. http://www.cs.waikato.ac.nz/Pub/Html/dag.html
3. see Palisade description at: http://www.trimble.com/cgi/products.cgi/pd_om008.htm
4. The Waikato ATM group pages at http://atm.cs.waikato.ac.nz/atm/
5. OPERA information site at http://www.opera.net.nz/
6. Wnet is described under the WurcNet pages: http://www.wnet.ca/