[NCLUG] Multiple nics same subnet

[Sean Reifschneider]

On Thu, Mar 02, 2006 at 06:13:07PM -0700, Tim Kuhlman wrote:
>http://lartc.org/howto/lartc.rpdb.multiple-links.html#AEN298

This won't work at all, it's meant for the case where you have multiple
upstreams to the public Internet.

Brian's response is almost right, but you need to list the remote host IP,
not the local host IP in the route:

   route add <REMOTE HOST 1> dev eth0
   route add <REMOTE HOST 2> dev eth1
   [...]

This assumes that the clients are directly connected on the same subnet as
the server.

However, you need to be careful that you don't end up with the remote
clients using a single one of the servers IP addresses for sending
continued traffic to.  This could happen in some cases, and will cause
traffic from the clients to default to going over the same pipe.

You could also just set up specific subnets as aliases for each of the
interfaces and effectively do point-to-point routing using this alternate
set of IPs between the machines for this communication.  This would more
completely separate them, but could require that get get both ends to bind
to the particular IP address for that alternate network block, instead of
just using the system's default IP as the source address for outgoing
packets.

You could probably make it simple and not require different IP addresses
for each interface by adding specific ARP entries on all the machines
involved and on the server specifying the static ARP entries for each
client on a different interface.  Then on each client you would associate
the server's IP address, but listing a different server MAC address for
each client.  This would probably require that all interfaces have the
server's IP address bound to it.

You *MIGHT* be able to get by with just using equal cost multi-path routing
in the kernel.  Just list multiple routes for your subnet for each
interface:

   route add 192.168.1.0/24 dev eth0
   route add 192.168.1.0/24 dev eth1
   [...]

ECMP in Linux only works on a per host basis though, and some of your
clients may end up conflicting based on how the kernel picks the paths.

You also could do this with the fwmark routing and routing based on source
IP, but this can be tricky to set up.  I've done it, but I always have to
look it up whenever I do it.  It's pretty subtle and the iproute2
documentation, frankly, sucks.

Many of these solutions may require that rp_filter be disabled for many of
the interfaces, for obvious reasons.

What you really want is probably the "bonding" driver.  Depending on the
capabilities of the switch you are connecting to, you can get a single
interface that has 4x the throughput of a single interface, or can at least
balance the load incoming and outgoing based on MAC address.  See the
in-kernel documentation for more information about this.  Particularly if
you have a switch that can do link aggregation...

Thanks,
Sean
-- 
 Dear Santa, all I want for Christmas is your list of girls who were naughty.
Sean Reifschneider, Member of Technical Staff <jafo at tummy.com>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability
      Back off man. I'm a scientist.   http://HackingSociety.org/