Ethernet-layer NAT (userspace queueing for ebtables)?

[Gabriel Somlo]

On Thu, Mar 30, 2023 at 06:30:55PM -0600, Sean Reifschneider wrote:
> A bridge only typically floods a few packet types, like ARP and multicast. 
> This is called a "Learning Bridge" and is what Linux implements.  You can
> bridge an eth interface to a tap device and it will forward on the appropriate
> traffic.  I'm pretty sure there are sysctls you have to enable as well, but I
> don't remember what they might be.  Maybe "proxy_arp"?

The network simulator has containers with veth pairs bridged together
to "link" containers to each other, and bridge some container's links
to the host VM's interfaces, such as e.g. ens32 in my example.

With on-prem equipment under our control we'd enable promiscuous mode,
and any other VM on the same vswitch as ens32 would see the simulated
container as its default gateway.

Those VMs should realistically appear to their users to be on a lan
with a default gateway -- realism and separation of responsibilities
precludes me forcing everyone to participate in a shared VPN scheme
(which, for the purpose of this exercise, would count as "contrived' :)
 
> What is the higher level goal here?  I'm guessing that you are trying to link
> the networks in two locations so that all VMs in multiple places can seamlessly
> talk to each other as if they are on a single LAN?  If so, consider just
> putting them all on Tailscale and using the tailnet addresses to talk.  I have
> over 100 machines on tailscale and it's working quite well.  It's a mesh VPN,
> so all the nodes establish connections directly to eachother, on a single
> segment.

In order to encapsulate all the craziness and complexity inside this
machine that's under my control, I'm now considering digging into the
linux bridging sources to find a way to force "hub-like" behavior,
i.e. forwarding ethernet frames *in spite* of what the bridge "fdb"
says about 'permanent' mac addresses associated with bridge ports
(like e.g. ens32's "official" mac address). That would allow me to
have the same mac address on multiple ports of a bridge, and *still*
have traffic to that mac address forwarded through the bridge -- that
would be perfect, too bad it's not something that appears I could just
configure with a setting...

Thanks to everyone for the brainstorm, if there's anything else I
could do without involving a VPN and "client-visible" changes, I'd
love to hear about it!

Cheers,
--Gabriel