[NCLUG] [OT] Re: Question about IP forwarding

Mon Apr 26 15:54:12 MDT 2010

I believe at this point it's clear that this isn't a linux problem, so I'll mark this thread off-topic, but here are the results from a wireshark capture on the 192.168.234.1 interface when I attempt to connect to a node 192.168.235.119 (this node isn't in my diagram, but it's just a linux node like the one pictured):

10.50.10.1 --> 192.168.235.119 TCP 51106 > ssh [SYN] Seq=0 Win=8192 Len=0 MSS=1380 WS=2
192.168.235.119 --> 10.50.10.1 TCP ssh > 51106 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460
10.50.10.1 --> 192.168.235.119 TCP 51106 > ssh [RST] 51106 Seq=1209360052 Win=0 Len=0
10.50.10.1 --> 192.168.235.119 TCP [TCP Port Numbers reused] 51106 > ssh [SYN] Seq=0 Win=8192

So, it appears to me as if the workstation initiates a connection, the worker node acknowledges it, but the break down happens when the workstation replies with an RST.

I've never seen those packets before, but my googling indicates that it's the TCP equivalent of "WTF?" . So the machines are talking, but apparently the replies are coming back in Mandarin or something.

The next packet seems to be the workstation trying to reinitiate the connection. This continues ad infinitum (or actually, until the ssh client times out).

-- Marcio

________________________________
From: Marcio Luis Teixeira <marciot at yahoo.com>
To: Northern Colorado Linux Users Group <nclug at lists.nclug.org>
Sent: Mon, April 26, 2010 3:00:11 PM
Subject: Re: [NCLUG] Question about IP forwarding

Oh, yeah, a picture would be good. I didn't know how to include one, but I guess ASCII would work. Hope this turns out ok:

            Internet
               |
           Cisco ASA
           10.50.0.1
               |
     +---------+------------+   10.50.0.0/255.255.240.0
     |                      |
  10.50.10.1            10.50.2.10
Workstation             Linux NFS
(gw: 10.50.0.1)        192.168.234.1
                            |
                            |   192.168.234.0/255.255.255.0
                    +-------+-------------+-- ...  
                    |                          
              192.168.234.100    
              Diskless Node #1
             (gw: 192.168.234.1)

I also fumbled some of the IP addresses in my original e-mail, so this should clear things up. So, in short, I want the workstation to be able to secure shell into Diskless Node #1. This works *great* if I configure a static route on the workstation as such:

   route add -net 192.168.234.0 netmask 255.255.255.0 gw 10.50.2.10

However, if I set up a static route on the Cisco instead (plus disable NATing to that destination), then the workstation is able to ping the Diskless Node #1, but not secure shell into it.

I tried the "ping -v 192.168.234.100" on the workstation, and I saw nothing different. What am I supposed to be seeing if there is an ICMP redirect going on?

-- Marcio

________________________________
From: Bob Proulx <bob at proulx.com>
To: nclug at lists.nclug.org
Sent: Mon, April 26, 2010 1:50:42 PM
Subject: Re: [NCLUG] Question about IP forwarding

Marcio Luis Teixeira wrote:
> worker nodes to PXE boot Fedora. The NFS server has two NICs: one
> NIC connects to our LAN (10.50.x.x/16) while the other NIC connects
> to a private switched LAN where all the worker nodes are connected
> (192.168.234.x/24).

A picture of everything would be great.  A ascii picture would be
file.  But I think I am following your description okay just the same.

> addresses. My first attempt at this was to enable IP forwarding on
> the cluster's NFS server and then go to individual workstations and
> set up a static route to the 192.168.234.x, using the NFS server as
> the gateway, ...

All good.  An important point that you didn't mention is that (I
assume) the worker nodes all have the NFS server as a gateway router
too, right?  Then all of the packets have a route that works and all
enter and exit on different interfaces.

> The problem has been to scale this beyond one workstation. On our
> LAN, we have a Cisco ASA 5510 at 10.50.0.1 -- all our workstations
> have that as their default gateway. I figured it would be a simple
> matter to set up a static route for the 192.168.234.x subnet on the
> Cisco device and that the Cisco would receive packets for the
> 192.168.235.x subnet, pass them on to my NFS server, and all would
> be good. Only that it isn't.

I have run into problems whenever a packet's route causes it to enter
and exit _on the same nic_.  AFAICT you are expecting the cisco to
pick up the packets and then retransmit them on the same wire.  That
will probably trigger additional spoofing rules and will take
additional configuration to work around.

The easiest solution would be if you could attach your 192.* private
network to another interface on the cisco.  Then everything should be
in a normal space and work normally.  Of course you might not have a
spare interface available.  But if you can you should try that because
I think all of your problems would go away at that point.

> I'm now able to ping the worker nodes just fine,

I think what you should be seeing is an ICMP redirect message from the
cisco to your individual workstation.  This sets up a cached temporary
route (until it times out) for the IP address.  Your first ping packet
goes to the Cisco as you would expect.  The Cisco says, that IP
address is on your network and it is over there, you don't need me to
route for you.  It replies to the workstation with an ICMP redirect
message.  Subsequent ping packets are transmitted from the workstation
to your nfs server router due to this dynamic route.

IIRC 'ping -v' with the -v option will show this redirect happening.
But that may be ping specific and there are many different
implementations.  I don't recall which ones do or don't.  But
regardless you should be able to see this wth tcpdump/wireshark.

> but SSH sessions hang before even reaching the login prompt.

That is what I would expect when a Linux machine is being used as a
router and packet would both enter and exit on the same device.  I
forget the details but more configuration is usually needed to enable
this.

Or I am wrong on that and the problem is that the source IP-PORT and
destination IP-PORT quad (which makes up the virtual connection) isn't
preserving the IP-PORT relationship when going the route through the
cisco.  For example if masquerading is in effect in the path (perhaps
on the nfs server router) then one direction may be doing source or
destination network address translation but not the opposite
direction.  This would break the virtual connection.  Because your
route may be taking asymmetrical paths it may be setting up this
condition along the way.  But when going through the single nfs server
router it may have been doing the nat both ways.

Because the path from the worker nodes is gatewayed to your nfs server
router in all cases, right?  But from your workstation would be
workstation, cisco, nfs-server-router, worker node in one direction
but not in the other direction.  If I followed that.  An asymmetrical
route may or may not be okay but the address port quad must be a
reasonable virtual circuit.

You would need to run tcpdump on the workstation as well as on the nfs
server and see what packets are passing there too.

I am just guessing and brainstorming for you.  It is tedious to track
with tcpdump/wireshark but you should be able to verify and understand
the source and destination ip-port happenings and be able to tell if
this is happening or not.

> I did some wiresharking on the NFS server and noticed that the first
> SSH packet appeared to come in fine, but wireshark colored
> subsequent ones red and black and seemed to say something or other
> about sequence numbers being fubar.

Look specifically at the source and destination addresses.  I am
really suspecting the problem is there.

> One of my ideas was that packets *into* the private network were
> being sent to the Cisco and forwarded to the NFS server, but return

Maybe.  I suspect an ICMP redirect should be occurring.

> packets were bypassing the Cisco and being delivered directly from
> the NFS server to the workstations (because both the NFS server and
> the workstations have NICs in the 10.50.x.x subnet).

That should definitely be happening.  Because the destination address
is on the same network and the host has a device there.  Therefore it
will route them there directly.

> I don't know enough about TCP/IP to know if this would be the
> problem, although I have an inkling that I may be incorrect in my
> assumption that I can have two devices acting as routers on the same
> network, and have one router "hand off" packets to another.

Each route segment is independent of other route segments.  They don't
know about each other.  They don't coordinate.  (Not talking about BGP
here or anything.)

> So, my question is, am what I am trying to do feasible and am I
> approching it the correct way?

Someone with more insight than myself will surely pop in with the
critical piece of information to help.

Bob
_______________________________________________
NCLUG mailing list      NCLUG at lists.nclug.org

To unsubscribe, subscribe, or modify 
your settings, go to: 
http://lists.nclug.org/mailman/listinfo/nclug

_______________________________________________
NCLUG mailing list      NCLUG at lists.nclug.org

To unsubscribe, subscribe, or modify 
your settings, go to: 
http://lists.nclug.org/mailman/listinfo/nclug