WITS: Waikato VIII

Trace Format ERF, captured using a DAG 3 card.
Volume on Disk 624 GB
Number of Traces 99
Capture Start (Local) Thu Apr 7 12:00:01 2011
Capture End (Local) Sat Nov 5 13:00:00 2011
Total Duration 86 Days, 17 Hours, 19 Minutes and 50 Seconds
Packets Captured 27,870 million
Total Traffic 16,576 GB
Contiguity Mostly discontiguous, but there is a small series of traces that are contiguous.
Snapping Method Packets truncated four bytes after the end of the transport header, except for DNS
Rotation Policy Daily rotation at Midnight UTC. Also rotate on AES key change.
Anonymization IP addresses anonymized using Crypto-Pan AES encryption.
Download Link

This is a selection of packet header traces captured at the border of the University of Waikato network. The traces were captured using a single DAG 3 card and the WDCap trace capture software. The version of WDCap used was version 3.1.2 and the Libtrace version was a subversion build between 3.0.7 and 3.0.8.

The capture point was located between the University's network infrastructure and the commodity Internet. This allowed access to all the traffic that was coming into and exiting the University. However, no internal traffic would have been observed and captured by our capture point. The capture machine performed all the anonymization and truncation before exporting the packets via the network to a second machine. That machine was also running WDCap which would read the packets off the network and write the traces.

Each trace file is named using the following format: yyyymmdd-HHMMSS-[code].gz. The time and date refers to the time in UTC when the first packet in the file was captured. The code refers to the event which caused the previous file to be closed and this new file to be created. Note that new codes have been added in this edition of WDCap.

Codes of interest for this traceset are as follows:

  • 0 - Rotation boundary was reached
  • 1 - Encryption key was changed
  • 4 - The capture process has been restarted
  • 5 - Client reconnected after disappearing without notifying the server

Regular file rotation (code 0) occured daily at Midnight (UTC).

Packet records are truncated four bytes after the end of the transport header except in the case of DNS traffic, which is snapped twelve bytes after the end of the transport header. This means that many packets will contain a small amount of user payload - enough to perform some rudimentary layer 7 analysis without seriously threatening the privacy of the network users. ICMP packets which are truncated after any IP and transport headers that may be present in the packet payload.

The IP addresses contained within the packets have been anonymised using Crypto-Pan AES encryption, which is a prefix-preserving anonymisation method. This means that unanonymised IP addresses that were on the same subnet will also be identifiable as on the same subnet when the addresses are anonymized. We change the encryption key once a week on Sunday midnight (local time). This key change causes a file rotation, with a rotation code of 1.

The TCP and IP checksums have also been validated and anonymized. If the checksum was correct, it has been replaced with 0. If the checksum was incorrect, it has been replaced with 1.

Unlike some previous Waikato datasets, this dataset was not intended to be entirely contiguous. Instead, we have saved one day's worth of trace per week during the capture process as well as occasionally retaining a fortnight's worth of traces. There was a bug in the trace-retention script, however, that meant that the first trace after a key change was not saved so most of these fortnightly blocks also have a gap in them.

If you require a large contiguous block of traces, your best bets are the 20111021 to 20111104 series (but even that has a potential break on 20111027) or the 20110601 to 20110620 series.

The recommended method for processing these traces is to use Libtrace, which we have developed. There are a number of tools included with libtrace such as a packet dumping utility, a trace format converter (for example, to convert to pcap), a trace splitting/filtering tool and a few statistic generators. We suggest you examine the Libtrace Wiki for more details on the Libtrace tools and the library itself.

Name Local Start Time Duration Total Packets Compressed Size
20110407-000000-0 Thu Apr 7 12:00:01 2011 24:00:00 331 million 7,715 MB
20110414-000000-0 Thu Apr 14 12:00:01 2011 24:00:00 331 million 7,664 MB
20110421-000000-0 Thu Apr 21 12:00:01 2011 24:00:00 177 million 3,927 MB
20110428-000000-0 Thu Apr 28 12:00:01 2011 24:00:00 252 million 5,666 MB
20110505-000000-0 Thu May 5 12:00:01 2011 24:00:00 344 million 7,986 MB
20110510-000000-0 Tue May 10 12:00:01 2011 24:00:00 369 million 8,593 MB
20110511-000000-0 Wed May 11 12:00:01 2011 24:00:00 400 million 9,434 MB
20110512-000000-0 Thu May 12 12:00:01 2011 24:00:00 372 million 8,743 MB
20110513-000000-0 Fri May 13 12:00:01 2011 24:00:00 295 million 7,081 MB
20110514-000000-0 Sat May 14 12:00:01 2011 12:00:01 141 million 3,309 MB
20110515-000000-0 Sun May 15 12:00:01 2011 24:00:00 352 million 8,000 MB
20110516-000000-0 Mon May 16 12:00:01 2011 24:00:00 417 million 9,860 MB
20110517-000000-0 Tue May 17 12:00:01 2011 24:00:00 402 million 9,427 MB
20110518-000000-0 Wed May 18 12:00:01 2011 24:00:00 369 million 8,764 MB
20110519-000000-0 Thu May 19 12:00:01 2011 24:00:00 367 million 8,663 MB
20110520-000000-0 Fri May 20 12:00:01 2011 0:25:38 9 million 219 MB
20110521-000000-0 Sat May 21 12:00:01 2011 12:00:00 135 million 3,143 MB
20110522-000000-0 Sun May 22 12:00:01 2011 24:00:00 307 million 7,064 MB
20110526-000000-0 Thu May 26 12:00:01 2011 24:00:00 388 million 9,006 MB
20110601-000000-0 Wed Jun 1 12:00:01 2011 24:00:00 366 million 8,477 MB
20110602-000000-0 Thu Jun 2 12:00:01 2011 24:00:00 349 million 8,153 MB
20110603-000000-0 Fri Jun 3 12:00:01 2011 24:00:00 295 million 6,888 MB
20110604-000000-0 Sat Jun 4 12:00:01 2011 12:00:01 126 million 2,890 MB
20110604-120001-1 Sun Jun 5 00:00:01 2011 11:59:59 91 million 2,193 MB
20110605-000000-0 Sun Jun 5 12:00:01 2011 24:00:00 222 million 5,241 MB
20110606-000000-0 Mon Jun 6 12:00:01 2011 24:00:00 406 million 8,817 MB
20110607-000000-0 Tue Jun 7 12:00:01 2011 24:00:00 416 million 9,559 MB
20110608-000000-0 Wed Jun 8 12:00:01 2011 24:00:00 360 million 8,481 MB
20110609-000000-0 Thu Jun 9 12:00:01 2011 24:00:00 336 million 7,824 MB
20110610-000000-0 Fri Jun 10 12:00:01 2011 24:00:00 321 million 7,334 MB
20110611-000000-0 Sat Jun 11 12:00:01 2011 12:00:00 135 million 3,036 MB
20110611-120001-1 Sun Jun 12 00:00:01 2011 12:00:00 80 million 1,935 MB
20110612-000000-0 Sun Jun 12 12:00:01 2011 24:00:00 282 million 6,468 MB
20110613-000000-0 Mon Jun 13 12:00:01 2011 24:00:00 334 million 7,782 MB
20110614-000000-0 Tue Jun 14 12:00:01 2011 24:00:00 347 million 7,978 MB
20110615-000000-0 Wed Jun 15 12:00:01 2011 24:00:00 372 million 8,574 MB
20110616-000000-0 Thu Jun 16 12:00:01 2011 24:00:00 356 million 8,199 MB
20110617-000000-0 Fri Jun 17 12:00:01 2011 24:00:00 295 million 6,921 MB
20110618-000000-0 Sat Jun 18 12:00:01 2011 12:00:01 147 million 3,422 MB
20110618-120001-1 Sun Jun 19 00:00:01 2011 11:59:59 88 million 2,113 MB
20110619-000000-0 Sun Jun 19 12:00:01 2011 24:00:00 295 million 6,844 MB
20110620-000000-0 Mon Jun 20 12:00:01 2011 24:00:00 352 million 8,241 MB
20110623-230233-4 Fri Jun 24 11:02:33 2011 0:57:27 19 million 443 MB
20110701-000000-0 Fri Jul 1 12:00:01 2011 13:46:36 183 million 4,260 MB
20110708-000000-0 Fri Jul 8 12:00:01 2011 24:00:00 225 million 5,339 MB
20110712-000000-0 Tue Jul 12 12:00:01 2011 24:00:00 340 million 8,089 MB
20110713-000000-0 Wed Jul 13 12:00:01 2011 24:00:00 357 million 8,481 MB
20110714-000000-0 Thu Jul 14 12:00:01 2011 24:00:00 348 million 8,298 MB
20110715-000000-0 Fri Jul 15 12:00:01 2011 24:00:00 266 million 6,175 MB
20110716-000000-0 Sat Jul 16 12:00:01 2011 12:00:01 104 million 2,463 MB
20110717-000000-0 Sun Jul 17 12:00:01 2011 24:00:00 302 million 5,995 MB
20110718-000000-0 Mon Jul 18 12:00:01 2011 24:00:00 407 million 8,671 MB
20110719-000000-0 Tue Jul 19 12:00:01 2011 24:00:00 397 million 8,467 MB
20110720-000000-0 Wed Jul 20 12:00:01 2011 24:00:00 395 million 9,169 MB
20110721-000000-0 Thu Jul 21 12:00:01 2011 24:00:00 365 million 8,466 MB
20110722-000000-0 Fri Jul 22 12:00:01 2011 24:00:00 263 million 6,159 MB
20110723-000000-0 Sat Jul 23 12:00:01 2011 12:00:00 129 million 2,983 MB
20110729-000000-0 Fri Jul 29 12:00:01 2011 24:00:00 299 million 6,976 MB
20110805-000000-0 Fri Aug 5 12:00:01 2011 24:00:00 280 million 6,505 MB
20110812-000000-0 Fri Aug 12 12:00:01 2011 24:00:00 287 million 6,639 MB
20110819-000000-0 Fri Aug 19 12:00:01 2011 24:00:00 266 million 6,083 MB
20110826-000000-0 Fri Aug 26 12:00:01 2011 24:00:00 225 million 5,170 MB
20110902-000000-0 Fri Sep 2 12:00:01 2011 24:00:00 247 million 5,642 MB
20110909-000000-0 Fri Sep 9 12:00:01 2011 24:00:00 257 million 5,921 MB
20110910-000000-0 Sat Sep 10 12:00:01 2011 12:00:00 132 million 2,987 MB
20110911-000000-0 Sun Sep 11 12:00:01 2011 24:00:00 307 million 6,974 MB
20110912-000000-0 Mon Sep 12 12:00:01 2011 24:00:00 364 million 8,359 MB
20110913-000000-0 Tue Sep 13 12:00:01 2011 24:00:00 370 million 8,533 MB
20110914-000000-0 Wed Sep 14 12:00:01 2011 24:00:00 403 million 9,366 MB
20110915-000000-0 Thu Sep 15 12:00:01 2011 24:00:00 398 million 9,356 MB
20110916-000000-0 Fri Sep 16 12:00:01 2011 24:00:00 293 million 6,655 MB
20110917-000000-0 Sat Sep 17 12:00:01 2011 12:00:00 120 million 2,670 MB
20110918-000000-0 Sun Sep 18 12:00:01 2011 24:00:00 281 million 6,363 MB
20110919-000000-0 Mon Sep 19 12:00:01 2011 24:00:00 373 million 8,437 MB
20110920-000000-0 Tue Sep 20 12:00:01 2011 24:00:00 377 million 8,417 MB
20110921-000000-0 Wed Sep 21 12:00:01 2011 2:10:17 53 million 1,227 MB
20110922-000000-0 Thu Sep 22 12:00:01 2011 24:00:00 376 million 8,476 MB
20110923-000000-0 Fri Sep 23 12:00:01 2011 24:00:00 306 million 6,916 MB
20110930-000000-0 Fri Sep 30 13:00:01 2011 24:00:00 273 million 6,098 MB
20111007-000000-0 Fri Oct 7 13:00:01 2011 24:00:00 289 million 6,701 MB
20111014-000000-0 Fri Oct 14 13:00:01 2011 24:00:00 310 million 7,031 MB
20111021-000000-0 Fri Oct 21 13:00:01 2011 24:00:00 293 million 6,664 MB
20111022-000000-0 Sat Oct 22 13:00:01 2011 11:00:01 134 million 3,077 MB
20111022-110001-1 Sun Oct 23 00:00:01 2011 12:59:59 97 million 2,277 MB
20111023-000000-0 Sun Oct 23 13:00:01 2011 24:00:00 217 million 4,912 MB
20111024-000000-0 Mon Oct 24 13:00:01 2011 24:00:00 353 million 7,867 MB
20111025-000000-0 Tue Oct 25 13:00:01 2011 24:00:00 415 million 9,519 MB
20111026-000000-0 Wed Oct 26 13:00:01 2011 24:00:00 401 million 9,105 MB
20111027-000000-0 Thu Oct 27 13:00:01 2011 21:31:54 328 million 7,619 MB
20111027-213205-5 Fri Oct 28 10:32:05 2011 2:27:55 58 million 1,342 MB
20111028-000000-0 Fri Oct 28 13:00:01 2011 24:00:00 275 million 6,269 MB
20111029-000000-0 Sat Oct 29 13:00:01 2011 11:00:01 133 million 2,989 MB
20111029-110001-1 Sun Oct 30 00:00:01 2011 12:59:59 84 million 1,905 MB
20111030-000000-0 Sun Oct 30 13:00:01 2011 24:00:00 309 million 6,917 MB
20111031-000000-0 Mon Oct 31 13:00:01 2011 24:00:00 356 million 8,144 MB
20111101-000000-0 Tue Nov 1 13:00:01 2011 24:00:00 338 million 7,879 MB
20111102-000000-0 Wed Nov 2 13:00:01 2011 24:00:00 339 million 7,564 MB
20111103-000000-0 Thu Nov 3 13:00:01 2011 24:00:00 313 million 7,012 MB
20111104-000000-0 Fri Nov 4 13:00:01 2011 24:00:00 264 million 6,203 MB