Research into the Viability of Service-Provider NAT

Jump to Inbound Sessions

Outbound Sessions

Methodology

A session was defined as outbound (or outgoing) if the first packet observed for that session was sent from within the ISP's network. In the case of TCP, this initial packet must also have the SYN flag set. For UDP, we do not have the luxury of knowing which packets mark the beginning of a connection which may result in a few in-progress flows at the start of traceset being classified incorrectly. The overall effect of this should be relatively small, though.

Outbound sessions for protocols other than TCP, UDP or ICMP are only counted if a response was observed from the external IP address. This was primarily to allow us to count the number of subscribers that are doing legitimate traffic using these protocols. Bidirectional traffic is not required for outbound TCP, UDP or ICMP sessions to be counted. Many TCP sessions, for instance, were an outgoing SYN which was responded to with a RST. These sessions were all included in the final counts.

The following rules were used to govern when a session should be expired:

The term "Active Subscribers" will be used often when discussing the results. For an IP address to count as an Active Subscriber, either a new session must have been created originating from that address during the designated time period or there must have been at least one unexpired session associated with that address at the beginning of the time period.

Another term worthy of definition is "Peak Sessions". This is the largest number of unexpired sessions associated with a subscriber at a single point in time.

Finally, the time period used for all this research was 30 minutes. All per-period counts were reset whenever a time period ended. The new session counter was reset to zero, while the peak session counter was set to be current number of unexpired flows.

Results

Active Subscribers

This graph shows the number of Active Subscribers within each 30 minute time period. "Active" is defined above but to reiterate, an Active Subscriber either creates a new session within the time period or has an outstanding unexpired session left over from the previous session. Note the very clear difference between day and night. There is also a notable variation between morning and evening most days. Nothing totally unexpected here, though.

Here we use a slightly different definition of "Active". This time only subscribers that create a new session during the period are counted - pre-existing unexpired sessions are ignored. Ultimately, we get the same variations as in the previous graph, just with slightly smaller numbers. Note that this is only time we use this definition of "Active". All subsequent graphs are based on the Active Subscriber values in the first graph.

This is simply a count of the unique subscriber IP addresses observed over time. Not particularly useful, in and of itself, but the final count of 18185 is handy for producing many of the subsequent graphs.

New Sessions per 30 Minute Period

Mean number of new sessions created by a subscriber within the given 30 minute period. Note that this average is across ALL observed subscriber IPs, including subscribers that weren't active within that time period and subscribers who had not been observed at all yet. The mean is slightly higher in the afternoon/evening than between midnight and midday, although the mean number of new UDP sessions per time period doesn't appear to vary much at all throughout the entire traceset.

The median number of new sessions created by a subscriber within the given 30 minute period. As above, this includes all observed subscriber IPs within the entire traceset. Since most IPs are not active within any particular time period, the median often turns out to be zero and, in fact, turns out to be never more than 2. Overall, probably not a very useful measurement.

This graph is also the mean number of new sessions created per subscriber within the 30 minute period. However, this time the mean is calculated across Active Subscribers only. The most interesting aspect of this graph is that the peaks are actually between midnight and 6am. This is because there is a much smaller number of active subscribers during that time period, but those subscribers are probably primarily "heavy users" such as gamers and torrenters. Gamers would account for the increase in the mean number of new UDP sessions.

The median number of new sessions created by an active subscriber within the 30 minute period. Removing all the inactive subscribers means that we have a median that isn't zero! Interestingly, despite the large mean we saw in the previous graph, the median for the overnight periods is much lower than during the day. This implies that the high overnight means we saw were due to a relatively small proportion of extremely heavy users. This idea is strengthened by the change in scale between the mean and median graphs - the highest mean is close to 700 new sessions, the highest median is closer to 70. Finally, again note how static the numbers for UDP are. This is probably strongly influenced by machines doing solely NTP and maybe a little DNS.

Peak Sessions per 30 Minute Period

This graph shows the mean of the peak session counts for all subscribers (including inactive subscribers) within each 30 minute time period. As the mean is calculated across all subscribers, it is reduced by the inactive subscribers who would have a peak session count of zero. The mean only tends to vary slightly between day and night. The most interesting feature of this graphs is that UDP has a much higher mean than TCP. This suggests that when people do use UDP they tend to do it all at once.

The median number of peak sessions observed across all subscribers. Once again, the median across all subscribers results in very low numbers as the majority of subscriber IP addresses are inactive within each time period.

This is the mean number of peak sessions for active subscribers only for each 30 minute period. Once again, we see the accentuated mean between midnight and 6am caused by the increased proportion of "heavy users" overnight. UDP continues to demonstrate higher peaks than TCP. Overall, the numbers shown on this graph and the previous peak mean graph are much higher than might have been expected. This graph alone suggests that it is no big deal for users, particularly the bigger ones, to be have over 80 sessions active simultaneously.

This graph shows the median number of peak sessions for active subscribers only within the time period. The medians are again much smaller than the means - half the subscriber base has no more than 15 sessions active at any given point. This shows that the heavier users are in a distinctly separate class to most other users and will be very difficult (if not nigh-on impossible) to accommodate using service-provider NAT.

This graph breaks the number of peak TCP sessions per active subscriber down into percentiles. The aim is to show how the mean number of peak sessions is accentuated by a very small subset of subscribers who operate at levels far beyond what most other subscribers are. For each line on the graph, that percentage of subscribers fall at or below that number of peak TCP sessions. Note the massive range in which the top 1% of subscribers fall, compared to the remaining 99% (the Y-axis is also a logscale!).

Another percentile-based graph, this time for peak UDP sessions per active subscriber. The gap between the top 1% and the rest is not quite as large for UDP, although still significant. The shapes of the lines between 85 and 95 percent inclusive are quite interesting. My guess is that this might be once again related to gaming, possibly due to game server browsers opening UDP connections to multiple servers within quick succession.

Cumulative New and Peak Session Counts

I have graphs that show the cumulative changes in new and peak session counts per subscriber. I've decided not to include them on this page because they don't have a lot of impact on the viability of service-provider NAT. The curious can look at the graphs directly by going to this location and clicking on the images that have names beginning with "total". The graphs probably do have some academic interest, but including them here would distract from the more important results contained in other graphs.

Jump to Outbound

Inbound Sessions

Methodology

A session was defined as inbound if the first packet observed for that session originated from outside the ISP's network. In addition, an outgoing packet had to be observed in response to that initial inbound packet. For TCP sessions, this response had to be a SYN ACK. For UDP, there were no such restrictions on the response packet. This means the UDP counts will not be as accurate as the TCP counts, due to previously expired outbound UDP sessions continuing after a long delay with an inbound packet. Such behaviour would create a new inbound session because the original outbound session had long since expired. Nevertheless, the TCP numbers alone are sufficient for reaching a significant conclusion.

Note that we have ignored ICMP and all other protocols aside from TCP and UDP when measuring inbound sessions.

All other aspects of the methodology were the same as they were when counting outbound sessions, including rules for expiring flows and the definitions of "active subscribers" and "peak sessions".

Results

Perhaps the most pertinent result from investigating inbound connections is one that I don't have a graph for. Over 40% of the observed subscriber IP addressess accepted at least one incoming TCP connection! Adding in the slightly less accurate UDP counts and the ratio grows to over 60%. If customers are going to be placed behind a NAT box, port-forwards need to be created to allow incoming connections. This becomes an inefficient proposition if a significant proportion of subscribers want to accept connections and this is exactly what these results show.

Server Port Usage

This file is a list of the top 50 most commonly used TCP server ports by customers. The percentage value for each port is the percentage of inbound sessions that are destined for that particular port. Most importantly, there is no one single dominant port - rather the port usage appears to be spread across many high numbered ports. This does suggest that peer-to-peer services are the most likely source of accepted inbound sessions. The presence of well-known peer-to-peer ports 4662 and 6881 at the top of the list reinforces this idea. Most subscribers tend to change the default port that their file-sharing programs use to avoid rate-limiting by their ISP.

This graph illustrates the number of unique server ports used by each subscriber that accepts incoming TCP connections. Subscribers that do not accept a single TCP connection are not taken into account in this graph. 80% of subscriber-run servers use no more than 5 different ports, 90% use no more than 10. However, the distribution is very heavy-tailed, with a few subscribers using more than 1000 different server ports.

Active Subscribers

These graphs are the same as the "Active Subscribers" graphs shown earlier, except inbound sessions are being counted rather than outbound. Again we see diurnal variations in terms of the number of subscribers that are active throughout the day, although the shape is nowhere near as smooth and distinctive as the one we observed for outbound sessions. Also, the total active subscribers over time graph shows the total number of IPs that accepted some form of inbound TCP or UDP connection.

New Sessions per 30 Minute Period

These are also replicas of the graphs seen earlier, expect we've counted inbound rather than outbound sessions. I've included these more because they may be of academic interest rather than because they illustrate a specific point - the sheer number of inbound connections and the number of subscribers accepting them is the far bigger issue here.

Peak Sessions per 30 Minute Period

Again, these graphs are mostly of academic interest.

Cumulative New and Peak Session Counts

These graphs are probably of very little interest so I won't clutter up this document with them - instead they can be found here. The image files beginning with "total_" are the ones that are not already on this page.