[NCLUG] Re: Connectivity problems to ebay, (and possibly others)

[Mehgan Laveck]

John,

In response to your last post:

> I don't know which end the packets are being lost from, but rather believe
> they are being lost on the way to ebay, the other side of Level3 and
> Cogent.

I would like to try to get Level3 to set up some logging on their network
to verify that 1) they are getting the request, and 2) Ebay is responding
to that request.  If you have a source IP that seems to be failing more
often than not, let's get the ball rolling on that.
>
> This theory is based on FRII's forced routing change from Level3 to Cogent
> for one dest IP which changed the src/dest pairings for that dest IP. This
> then strongly suggests the problem exists at the Denver NAP, or ebay (with
> a small nagging question about the last FRII router in the path).

The testing I mentioned above should clear up this question.
>
> The only viable theory I have why this changed src/dest pairings is that
> the TTL changed because of the number of hops. This theory is also
> consistant with another theory that the observed randomness of paired
> src/dest connection failings exists because of changes in the timestamp
> bits, the only other uncontrolled variable in IP packet data.
>
> If losses were due to the replies from ebay, that route would remain the
> same, thru Level3 and one would expect the problem not to be affected by
> the forced routing change.

The only way I can think of to verify issues with TTL or something
corrupting the data stream is to set up packet sniffing.  It's going to be
interesting sniffing traffic on a link that is pushing upwards of 70Mb,
but I will open a ticket with Juniper to see if there is any way to filter
a mirror.
>
> The theory of a forced web cache remains in question, but as Jeff noted,
> my understanding too is the CWX is not behind FRII's web cache.

FRII's web cache is only in place for people who choose to us it.
>
> In addition, there is this free variable regarding problems which appeared
> at nearly the same time, with web pages for ads also timing out ... mostly
> for doubleclick url's, and partly due to DNS lookup failures returning
> SERVFAIL for zones that I can reliably access all the servers for from my
> desktops. If these are related to the ebay problem, then that points to
> the Denver NAP routers, or possibly other services co-located with the
> same infrastructure as the Denver gear serving ebay. However, so far the
> traceroutes for all doubleclick services I've found don't have routes
> terminating at the Denver NAP (which doesn't rule out a hidden ATM cloud
> between the visible end points and Denver). So there might be two or three
> separate problems.
>
> The most definiative failure would be a Non-FRII hosted machine, which
> would move the finger to point clearly at the Denver NAP or Ebay. The
> second most definiative test would be a completely clean (1-254) Class C
> outside FRII, which passes a couple dozen connections to all four
> destination IP's at ebay, with routes thru Level3 or Cogent terminating in
> Denver, which would point the finger directly at FRII.
>
I have posted to NANOG to see if we can find anyone outside of FRII's
network experiencing the same problem.

--
Mehgan Laveck - Network Engineer
Front Range Internet, Inc.
mehganl at frii.net - (970) 212-0725