Re: [44net] Help setting up a BSD-based gateway? - 44net

29 Jul 2016

[Apologies for the repost: I meant to have a useful Subject:
header and accidentally omitted that earlier.]

[Forgive the top-posting: I include the full text of what I'd
written several weeks ago below for refresher and reference.]

To recap, I've set up IPENCAP tunnels to AMPRNet networks on a
Ubiquiti EdgeRouter Lite running OpenBSD: there is one gif(4)
interface per tunnel and routes are kept in a dedicated routing
table (like Linux, OpenBSD supports multiple routing domains on a
single system); firewall rules are used to go from one routing
domain to another.

The biggest piece missing was a daemon to handle receiving 44net
RIP packets and use that data to maintain tunnels and routes.  I
thought about porting one, but decided of write my own instead.
It has been running for a few weeks now on my node and while it's
still not quite "done" it seems to work well enough that I decided
it was time to cast a somewhat a wider net and push it up to
GitHub for comment from others.

A couple of quick notes on implementation:

1. The program maintains a copy of the AMPRNet routing table in
   a modified PATRICIA trie (really a compressed radix tree).
   Routes are expired after receiving a RIP packet.
2. A similar table of tunnels is maintained.
3. Tunnel interfaces are reference counted and garbage collected.
   A bitmap indicating which tunnels are in use is
   maintained.
4. The program is completely self-contained in the sense that I
   do not fork/exec external commands to e.g. configure tunnels
   or manipulate routes.  That is all done via ioctls or writing
   messages to a routing socket.

There is more to do; I'm sure there are a few bugs.  I'd also like
to query the system state at startup to initialize the routing and
tunnel tables.  Exporting and/or parsing an encap file would be
nice.  Logging and error checking can, I'm sure, be improved.

It's about 1200 lines of non-comment code, compiles down to a 28K
MIPS64 executable (stripped).  The code is at
https://github.com/dancrossnyc/44ripd

Please have a look and let me know what you think!

        - Dan C.

On Fri, Jun 17, 2016 at 5:46 PM, Dan Cross &lt;crossd(a)gmail.com&gt; wrote:
...
  On Tue, Jun 14, 2016 at 12:51 PM, Dan Cross
&lt;crossd(a)gmail.com&gt; wrote:
      I'm trying to set up an AMPRNet gateway
at home and am running into
 some problems.  Has anyone successfully configured a BSD-based gateway
 that would be willing to give me some pointers?

   [snip]

     It seems like what I want to do is configure a gif(4) interface
 for tunnel traffic, but my attempts at doing so all seem to fail,
 and documentation for setting up an IPENCAP tunnel is related to
 setting up IPsec gateways; my attempts at transliterating from the
 examples for e.g. Linux and Cisco et al have failed.

     If someone has gone down this road before and has a working
 setup, that would be tremendously helpful.  If someone could send
 me output from 'ifconfig -a' and/or 'netstat -rn -f inet' and
 possibly some 'tcpdump' output, I could probably muddle through the
 rest.  If there are any caveats in setting up a 'pf' based firewall,
 that would be helpful as well.  If not, I suppose my next step will
 be to reinstall RouterOS on the ERL, try and get everything configured,
 and then see if I can replicate under BSD. 
 Folks,

     I just wanted to do a quick followup to this for the archives.
 I am now successfully transferring traffic between my 44net subnet
 (44.44.107.0/24) and the rest of the world using my OpenBSD-based
 router.

 A quick recap of my setup:
 1. I have a Comcast business class circuit.
 2. I have a dedicated static IP address to use as an endpoint for
    44net traffic.
 3. My comcast router is just a router; no NATing, no firewalling.
 4. My (non-comcast) edge router is a Ubiquiti EdgeRouter Lite
    running OpenBSD 5.9.  It has three ethernet interfaces:
    a. cnmac0 is the external interface connected to Comcast's network
    b. cnmac1 connects to my internal network
    c. cnmac2 is my internal gateway to 44.44.107.0/24.

     On OpenBSD, tunneling interfaces for IPENCAP are provided by
 'gif' pseudo-devices.  Unlike Linux, it appears that one creates a
 separate 'gif' interface for each tunnel, but one seems able to
 create an arbitrary number of such interfaces: I created a thousand
 as a test.  I'm sure there is a limit but it seems sufficiently
 high that practically routing AMPRNet traffic won't run up against
 it.  (Again, if someone knows of a different way to configure a
 single 'gif' interface so that it could support multiple tunnels,
 I'd be happy to know about it).  In other words, don't worry about
 scalability because you are creating a separate 'gif' interface for
 each tunnel to another AMPRNet site.

     On AMPRNet, the UCSD gateway *will not* pass traffic for an
 IP address that does not have a corresponding entry in the AMPR.ORG
 domain.  Also, 44.0.0.1 does not respond to 'ping' from 44/8 IP's.
 Caveat emptor as one tries to test: make sure you have DNS entries
 for your addresses and try pinging something other than 44.0.0.1
 or you'll suffer contusions banging your head against a desk trying
 to figure out why nothing appears to work....

     Once I had a tunnel up to UCSD, I found that I could ping my
 44.44.107.1 machine from a host on my internal network, but not
 from arbitrary machines.  This was interesting; it turns out that
 hosts on my internal network get NAT'ed to another IP address on
 the small subnet I got from Comcast (through another, completely
 separate router -- not comcast's router but another ERL).  What was
 happening was that as I ping'ed 44.44.107.1 from e.g. my laptop,
 ICMP echo request packets got NAT'ed to this other address and
 routed over to amprgw.sysnet.ucsd.edu and tunneled back to the
 external interface of my AMPRNet gateway.  The gateway accepted the
 encapsulated ICMP echo requests (I have a PF rule that explicitly
 allows ping) and forwarded them across the tunnel interface where
 they were unencapsulated; the IP stack saw that the result was
 addressed to an IP address on a local interface (i.e., they were
 for the router) and generated an ICMP echo response packet with a
 *source* address of 44.44.107.1 and a *destination* address of the
 external address of my other router (that is, the address the ICMP
 echo request was NAT'ed to).  This matched the network route for
 my local Comcats subnet and so my AMPRNet router realized it could
 pass the packet back to my other router directly.  It did so and
 the other router happily took the packet, matched it back through
 the NAT back to the original requesting machine (my laptop) and
 forwarded it: hence, I got my ping responses back.  But note that
 the response was not going through the tunnel back to UCSD: it was
 being routed directly through the external interface.

     Now consider what happens when I tried to ping 44.44.107.1 from
 a different machine on some other network.  The ICMP echo request
 packet gets routed through the UCSD gateway and tunneled back to
 my gateway as before, but since responses don't go through back
 through the tunnel, the response packet matches the default route
 of my gateway and get's forwarded to comcast's router.  Comcast
 would look at it, see that 44.44.107.1 wasn't on one of it's known
 networks that it would route floor, and discard the response.  Oops.

     The solution was to set up a separate routing table in a different
 routing domain specifically for AMPRNet traffic, and tie the two
 together using firewall rules.  In the AMPRNet routing table, I
 could set my default route to point to the UCSD gateway, so any
 traffic sent from one of my 44.44.107.0/24 addresses that doesn't
 match a route to a known tunnel gets forwarded through
 amprgw.sysnet.ucsd.edu.  With that in place, I could ping my gateway
 from random machines.  This must seem obvious to a lot of folks
 here, but it took me a little while to figure out what was going
 on.  Things are working now, however.

     So far I have encountered two other caveats: I decided to
 configure two tunnel interfaces statically at boot time: 'gif0'
 goes to the UCSD tunnel, and 'gif1' sets up a tunnel to N1URO for
 his 44.88 net.  Under OpenBSD, I assumed that the natural way to
 do this would be to add /etc/hostname.gif0 and /etc/hostname.gif1
 files and this does in fact create the tunnels at boot time.  However,
 traffic going out from my gateway doesn't seem to get sent through
 the tunnels; I did not bother to track down exactly why, but I
 believe it has to do with some kind of implicit ordering dependency
 when initializing PF.  When I set up the separate routing domain,
 it struck me that the language accepted by /etc/netstart in an
 /etc/hostname.if file was not sufficiently rich to set up tunnels
 in a routing domain, so I capitulated and just set up the static
 interfaces from /etc/rc.local; imperfect but it works.

     The second caveat is that I seem to have tickled a kernel error
 trying to set up an alias of a second IP address on my 44.44.107.1
 NIC; I get a kernel panic due to an assertion failure.  It looks a
 bug to me, but I haven't had the bandwidth to track it down.  In
 the meanwhile, simply don't add aliases to interfaces in non-default
 routing domains.

     I'll try to go through my notes and type up something for the
 wiki.

     The next step is to write a modified version of rip44d for BSD.
 I may take a stab at that this weekend if I can get some time.  As
 near as I can tell, the wire format of the protocol is strictly
 RIPv2; the difference is what the gateway does with the data in the
 RIP packet (it sets up tunnels in addition to maintaining routes).

     Anyway, I hope this can be of some small help to someone else
 who wants to run an OpenBSD-based gateway! 