Ethernet-layer NAT (userspace queueing for ebtables)?

[Gabriel Somlo]

Hi,

I find myself in need of a Layer-2 (Ethernet) NAT solution to work
around "cloudy" vmware's restrictions on promiscuous mode for guest
network interfaces.

For context, I'm running a network simulator (think gns3 or CORE),
and need to bridge simulated nodes to the outside of the host VM,
so they can interact with other host VMs sharing a VMWare vswitch
(also, think of the simulated node as the default gateway for the
LAN that's bridged to the outside vswitch):

-----------------------------
VMWare guest running sim    |
                            |
simulated node,             |
default gateway             |
-----------                 |              ------------
          |                 |   vmware     | External VM
     eth0 + -- br0 -- ens32 +-- vswitch ---+ using in-sim
  Sim.MAC |          VM.MAC |              | dflt. gateway
-----------                 |              ------------
-----------------------------

VMWare will prohibit traffic in/out of the simulator VM if the
dst/src (respectively) MAC address doesn't match the ens32 MAC.

So I thought I could support *one* simulated node being bridged
to the outside if I had ebtables do layer-2 NAT:

  ebtables -t nat -A PREROUTING \
        -i ens32 -d <VM.MAC> -j dnat --to-destination <Sim.MAC>

  ebtables -t nat -A POSTROUTING \
        -o ens32 -s <Sim.MAC> -j snat --to-source <VM.MAC>

This leaves me with the (tricky, as it turns out) problem of rewriting
ARP payloads in the requests and replies originated by the simulated
node (which is also the default gateway for the entire bridged LAN).

ARP packets (requests *and* replies) issued by the simulated node will
have the layer-2 ("outer") mac address translated by ebtables as per
above, but the "inner" (*content*) source mac address will still be
<Sim.MAC>, and I need a way to rewrite that to <VM.MAC>.

I've been looking for a way to punt Ethernet frames to userspace and
rewrite the relevant payload (src. mac) field there, but there doesn't
seem to be a QUEUE target in ebtables (unlinke in iptables).

ebtables seems to have an "arpreply" target extension, but 1. I couldn't
get it to work, and 2. it would only at best take care of the simulated
default gateway's arp replies (by generating them on its behalf to the
portion of the LAN outside the simulation). It would *not* take care of
answering any arp requests issued by the simulated defautl gateway, which
are open ended and depend on what's actually connected on the "outside",
through the vswitch.


Another thing I've considered and sort-of abandoned was to force VM.MAC
on the simulated node's eth0, e.g. with:

  ip link set dev eth0 address <VM.MAC>

but if I do that, then br0 will develop a strong reluctance to actually
forward frames destined for <VM.MAC> across the bridge from the outside,
and again I've tried disabling learning, setting the expiration to 0,
and generally anything I could think of to force br0 to "always flood",
like a hub, but unsuccessfully so far.


If you're still with me after all this, any advice, clues, or pointers
to things I've missed or should look into would be MUCH appreciated!

Thanks much,
--Gabriel