[NCLUG] Re: DSL Throttling or General Congestion?

[John L. Bass]

Hi DJ,

Bandwidth and latency are two very different beasties, and they are NOT 
co-dependent. Consider that an information stream to the moon band back 
can exist at 100mbps both up and down in a pipeline that is roughly 
385,000,000 meters each way. The speed of light is roughly 299,729,458 
m/s, so the round trip delay (flight time latency) of 2.58 seconds, with 
data happily streaming along at 100mbps.

The same problem exists on the ground, with packets flying coast to 
coast, or across the oceans, where the speed of light introduces flight 
time latencies of 50-200ms round trip, happily in a steam pipelined at 
the transport technology/modulation speeds. This is why the tcp protocol 
introduces a sliding window of packets which are streamed between 
destinations so that there are multiple packets in flight to match up 
the latency to the length of data in a streaming pipe, so that round 
trip delays do not limit bandwidth (ACK delays).

Most wide area network solutions (dsl, cable, Canopy wireless, WiFi 
wireless, etc) have a usable internal modulation bandwidth of 1-6 mbps, 
and typically have some modest speed CPU for a router at each hop. A 
typical 1500 byte packet at these speeds has a flight time latency of 
about 80us for 15 miles,  but a modulation latency of 8ms (at 1.5mbps). 
Each router will add routing delays and queuing delays per hop, of 
typically 3-60ms.

At the leaf end, where modulation rates of a T1 are 1.5mbps, and TCP has 
a stream of 64 1500 byte packets in flight, there will be a queue delay 
at the T1 of 64*8ms = 512ms per active TCP connection with 64KB windows. 
It gets worse, with newer TCP protocols that are willing to stack up 10 
times that, or with non-flow controlled UDP streams.

This is why P2P is horrible on a network, as each client/server will 
start up dozens of TCP connections, each with a large window, and 
saturate the slowest medium's router with hundreds of packets in the 
queue. On the leaf end of a DSL or wireless connection that has a 
1-5mbps feed, this causes considerable queuing latencies.

But the worst of it, is that with high queues, there will be times when 
the router buffer memory gets full, and it discards packets. This packet 
loss causes the sending end to shorten the window size by a factor of 2, 
meaning that connection has half the packets in flight. Several lost 
packets will shorten the window size to just a few packets, or even one, 
meaning that you only get a few packets per ACK round trip time. This is 
what REALLY slows down a network connection, big time.

Use a tool like mtr (matt's trace route) to monitor the latency and 
packet loss across the internet, and you will quickly see where the 
latency and packet losses stack up.

Have fun,
John




DJ Eshelman wrote:
> Nothing like reviving a near-dead thread with a nice long wordy email...
>
> John put this well, I think- DSL/Cable are 'oversold' networks, where 
> they hope that not everyone will be wanting to be on at the same time, 
> and usually they are very correct.
> Generally speaking, it works well enough for most home users, but as 
> the problem increases, I think we'll see more and more of 'fiber to 
> the home' kinds of services.  I'm not a network expert by any means 
> but because of some extreme issues with this on the business side of 
> things, I've been doing a lot of research lately.
>
> Bottom line:  A lot of people confuse speed capability 
> (download/upload speed) with *latency* as the reason why things are 
> 'slow'.


>
> EVERY TIME I've had an issue with DSL or Cable, I can track it down to 
> line latency of 200 ms or more.  That is usually caused by too many 
> hops before the CO, QoS, line congestion and just plain outdated 
> designs in the TCP/IP protocol itself.  So, not necessarily 
> 'throttling' of the connection, but definitely has that effect.  Any 
> throttling that will be done is more to prevent these issues from 
> causing timeouts than to actually squash the speed itself.  Add to 
> that the sheer number of compromises along the way for 'download 
> speed' reasons and you get latency WAY beyond a leased line.
> Want proof?  Next time you're having an issue with 'slowness', go to 
> http://www.speakeasy.net/speedtest (flash required) and run the test.  
> You'll probably find the line itself is running at normal speeds, but 
> your ability to actually download much of anything is nuts.  Ping 
> tests to tier one providers will usually confirm this latency exists, 
> and you'll see some crazy fluctuations during peak times.
>
> It's also why VPN connections are so difficult to use practically, but 
> what I'm finding is that I can't always convince my clients to put in 
> a leased circuit (a p2p T-1 from Fort Collins to Greeley, for example, 
> typically has an 8ms latency whereas an internet-based DSL connection 
> averages 70-180ms (cable was worse at 130-2000ms, testing both from a 
> Level3 Internet T-1 line).  Add the overhead of either IPSec or PPTP 
> for your VPN and you've got major latency issues.  And in my world- 
> having to deal with Windows/CIFS for the majority of what my clients 
> are doing- my problems are huge because CIFS is an extremely chatty 
> protocol that goes over TCP/IP for most of it's work.
>
> So now to why I'm even addressing this to the group:  I've been trying 
> to find open source solutions to get around these latency issues 
> (packeteers and such) and haven't had much luck.  Anyone else out 
> there dealing with these issues/have any ideas?