Rob,

It's an unfortunate and currently-unavoidable implementation detail.

Remember that the system routing table and the encap routing table are separate on amprgw. The system routing table is in the kernel; the encap table is in user space. Packets arriving at amprgw which are for hosts listed in the encap hosts table are diverted to the encap router; all other packets are left to the kernel to route.

On the other hand, packets leaving (originating at) amprgw are first looked up in the system routing table and if a route is found (as it would be for your bgp-advertised subnet), they are sent out the system Ethernet interface and not diverted into the encap router. Packets with a destination not in the system routing table are tested to see if they're in the encap list, and if they are, they're sent to the encap router to be forwarded. The rest are dropped. The encap list currently has 19000 host entries in it.

I have experimented with forcing 44.137.40.2 into the encap router with an explicit route to the loopback interface. This seems to work; the pings go out a tunnel to 89.18.172.156. I've left this route in place for now so you can repeat your tests and see if you do indeed get replies over the tunnel the way you expect. Please let me know.

* * *

If we weren't doing per-host routing, but instead doing per-subnet, things could be different. We'd have to have some 620 route table entries, with 440 tunnel interfaces. That's because FreeBSD doesn't have the ability to have ONE tunnel interface with multiple endpoints the way that Linux does on your gateway. FreeBSD shares this 'limitation' with most commercial routers, such as the Cisco and Mikrotik.

Amprgw was implemented first on Solaris, later FreeBSD, long before Linux was anything but a gleam in Linus Torvald's eye.

It could be argued that I should scrap FreeBSD on amprgw and start over with a Linux host. I don't know enough about the intricacies of Linux IP handling to be at all confident of being able to do that. I don't want to take the system down, do a complete reinstall, and hope to get it working while everyone depending on it waits, disconnected. Maybe someday. - Brian

On Thu, Jul 13, 2017 at 12:37:39AM +0200, Rob Janssen wrote:

...

Yes, BGP overrides encap on amprgw.

Is there a rationale behind that or is it just because of some implementation detail?

In our gateway, all routes are in the same table but at different distance (metric). Direct/BGP routes have higher precedence than IPIP routes, when for the same subnet. But, a more specific route (smaller subnet) always has precedence over a less specific one.