> In my case, where we run a BGP session with Internet and publish our assigned segment, we use a CCR1036 having in memory the full BGP routing table, but even so, this is not a compulsory requirement for you.
Indeed for doing BGP on internet you need such a router.
To do BGP on AMPRnet (i.e. internal routes only) the RB750Gr3 is good enough.
> In example, you can configure a less expensive router with only a default gateway to your ISP and/or to the ISP that runs the BGP session for you (if that would be the case).
This is what we do at our gateway as well. The ISP advertises our network and routes the traffic to our system.
> If you want to launch a traceroute from public Internet, just to demonstrate above setup, you can use 44.133.233.8
It does not answer to us. First hop is 88.26.246.131 via IPIP tunnel.
> Being said that, and nonetheless, I don’t think a Mikrotik RB750 is a good idea as it quite skimpy to have many things running at the same time.
> A Mikrotik RB2011 (no Wi-Fi) and above will be, for sue, very good platforms to play with.
Remember the suggestion was not the RB750 but the RB750Gr3.
This is an entirely different thing. It is more powerful than the RB2011.
(but has less ports of course)
Rob
> Thanks Rob, so I can ignore all the wiki instructions then and follow the script and read me?
Yes
> I assume I don't want to use a dmz but put my vdsl router into dumb bridged modem mode?
Well that is usually better (when you don't want to be bitten by DMZ bugs), however that
complicates things a little bit. Maybe it is better to make that your second experiment :-)
When you want to put the VDSL router in bridged mode, you probably also want the MikroTik to double-up
as a NAT router for your normal internet use. In itself that is not a problem, it can do that.
But it requires a little more study of the matter, especially when you want to be able to talk from
AMPRnet to internet (no NAT) and at the same time want to talk from your RFC1918 LAN to internet using NAT.
This involves using "policy routing", i.e. two route tables, one for each usage, and "IP Route Rule"
settings to select the proper routing table based on source address. Like this:
/ip route rule
add src-address=44.x.x.x/28 table=ampr
then put all AMPRnet routes in the table "ampr". the normal table "main" has the "internet" routes.
Traffic from your normal LAN (e.g. 192.168.88.x) to normal internet addresses is routed via the "main"
table where there is a default route to your ISP, traffic from your AMPRnet network 44.x.x.x/28 is routed
to the "ampr" table where there are the 600 or so tunnel routes and a default route pointing to amprgw.
You may also want to separate the LAN side in two different networks, either by reserving physical
ports for each network or by using a tagged VLAN for one of the networks. Can be done as well.
Almost anything can be done using these routers. However, trying to do it all at once and
finding out everything with your network lying in shambles because you re-configured your VDSL
router to bridge mode at the same time is not the easiest way forward :-)
So, first experiment a bit with the modem in DMZ mode and the MikroTik behind it, find out how
things operate, lock yourself out and find out how to recover, etc. Then you can more easily
move on to a more complex configuration by adding the above mentioned things.
Rob
> Excellent thank you I now have one on order.
> Does anyone know if the setup on the wiki is still current or given recent changes is there a new one?
Well, the text about MikroTik router setup on the wiki is not something you would want to do, as it only
installs a single tunnel to UCSD, but at the bottom it refers to the YO2LOJ site and the script you can
find there (get the latest version) and the directions for installing it are the way to go to get it on
the IPIP mesh.
Of course you can use it in a lot of other ways (also at the same time), depending on the local radio
network available. We use these routers (from their availability mostly RB750Gr3, before that some
RB2011 and RB750 versions) as routers in our radio network as well, each running a couple of links to
neighbors and using BGP as an autorouting protocol. In some areas it is also possible to setup a VPN
to some nearby gateway that is on the mesh and/or is directly routed on Internet.
Rob
> I think the RB750GR3 is the best bang for your buck if you don't need
> POE or SFP.https://routerboard.com/RB750Gr3
It sure is! Make sure you get the r3 version, not the earlier ones.
It is much more powerful than the RB750 (no G) and RB2011.
(of course it has only 5 ports and no WiFi)
Rob
> Try turning off RPF (return path filtering at the kernel level) if it goes out on one interface and comes in the other, then RPF is almost always at fault as it will by default drop the connection
I expect it is something invisible and nasty like that, but rp_filter is not active on that system.
(I never enable this because it is so difficult to monitor what it does... on gw-44-137 I use packet
marking using the "rpfilter" matcher in mangle and drop of the marked packets with logging in the filter)
Rob
> The strange thing is that ping works ok when TCP doesn't connect.
> My first suspicion would be a stateful firewall, but I'm sure you
> checked that. Could it be a TTL problem? I'm just guessing here.
The TTL of the inside IP packet is 63. I first traced on the external interface
and saw the encapsulated packet, then I traced on the tunl0 interface and saw
the decapsulated packet (same without the outer IP header), and it all looks OK.
I see the SYN going out,the SYN ACK coming in, but nothing more (ACK should go out).
The firewall is stateful but I added an explicit accept for -s 44.0.0.1 at the
top of all the rules to make sure it is not that. Also, I reset the firewall and
watched the counters, did not see any packets being dropped.
And indeed, ping works OK. It is strange. Maybe something is wrong due to the
gateway external address change, although I would not know what could produce
the above scenario. The system is up for about a year, maybe I should try
rebooting it. (usually this brings nothing when I try it... it isn't Windows :-)
Of course sometime it will become clear how this can be explained.
Rob
> I'm curious as to how long you waited when trying port 25. There
> is a five-second delay after the connection is established before
> it issues the 220 greeting
I noticed that. But on 44.137.40.2 the TCP connection does not even
establish.
I tried from home (44.137.41.97) and from the gw-44-137 (44.137.0.1)
and both were able to connect:
telnet 44.0.0.1 25
Trying 44.0.0.1...
Connected to 44.0.0.1.
Escape character is '^]'.
220 gw.ampr.org ESMTP Sendmail 8.15.2/8.15.2; Thu, 13 Jul 2017 09:20:58 -0700 (PDT)
quit
221 2.0.0 gw.ampr.org closing connection
Connection closed by foreign host.
However, on 44.137.40.2:
telnet 44.0.0.1 25
Trying 44.0.0.1...
(and nothing more. probably an error when I wait a minute or so)
When running tshark on tunl0 I see the SYN+ACK packets from 44.0.0.1 to 44.137.40.2
and when I do an insert of a rule matching source address 44.0.0.1 before all
other firewall rules I see it matching, but still no ACK is going back.
I have no idea what is going on there. Ping works OK.
Rob
> Remember that the system routing table and the encap routing table are
> separate on amprgw. The system routing table is in the kernel; the
> encap table is in user space. Packets arriving at amprgw which are for
> hosts listed in the encap hosts table are diverted to the encap router;
> all other packets are left to the kernel to route.
Ok that should work fine.
> On the other hand, packets leaving (originating at) amprgw are first
> looked up in the system routing table and if a route is found (as it
> would be for your bgp-advertised subnet), they are sent out the system
> Ethernet interface and not diverted into the encap router. Packets with a
> destination not in the system routing table are tested to see if they're
> in the encap list, and if they are, they're sent to the encap router
> to be forwarded. The rest are dropped. The encap list currently has
> 19000 host entries in it.
Why do you have entries for the BGP-advertised subnets? Would those not
be adequately handled by the default route? Or is this only to handle
IPIP tunnels with a BGP-advertised net-44 tunnel endpoint address? Maybe
route entries for only those addresses would be an option.
(after all, they are known as they are listed in encap.txt)
The algorithm could then be to lookup 44-net addresses in the IPIP table
when there is no specific route for them in the routing table. When no
match is found, they could still be sent out as BGP-routed when desired,
this could be achieved with a "special" entry in the lookup array for
BGP-routed addresses that are not mentioned in the encap list.
> I have experimented with forcing 44.137.40.2 into the encap router with
> an explicit route to the loopback interface. This seems to work; the
> pings go out a tunnel to 89.18.172.156. I've left this route in place
> for now so you can repeat your tests and see if you do indeed get replies
> over the tunnel the way you expect. Please let me know.
Interesting: I can now ping 44.0.0.1 from that system, but a "telnet 44.0.0.1 25"
still yields nothing. I have not yet been able to find why that is, I see
the SYN ACK coming back on the tunnel and I have allowed all traffic from
44.0.0.1 in the firewall but it simply fails to establish. No ACK is going
back to you. I can connect to other IPIP endpoints just fine. Strange.
It is not really required to fix it, I can easily route the mail via a
different path, but I was interested in finding why this goes wrong and how
it may affect others / other applications.
> It could be argued that I should scrap FreeBSD on amprgw and start over
> with a Linux host. I don't know enough about the intricacies of Linux
> IP handling to be at all confident of being able to do that.
I can understand that! It is certainly possible to get it running, and it
would be a lot easier to get IPIP running, but it is always a big step
to do such things.
Rob
> Yes, BGP overrides encap on amprgw.
Is there a rationale behind that or is it just because of some implementation detail?
In our gateway, all routes are in the same table but at different distance (metric).
Direct/BGP routes have higher precedence than IPIP routes, when for the same subnet.
But, a more specific route (smaller subnet) always has precedence over a less specific one.
Rob
> Rob, it looks like the problem might be nearer to you...
> Interestingly, when I traceroute to 44.137.40.2 with 169.228.34.84 as
> the source address, I get:
> ...
That is normal, it will be routed over internet to our BGP subnet 44.137.0.0/16
which is via that gw-44-137 which then forwards it to 44.137.40.2 over IPIP.
However, 44.137.40.2 does not reply to pings from internet.
> When I repeat that traceroute with 44.0.0.1 as the source address,
> it gets nearly as far, but not quite:
> ...
That is strange. Don't you have a route to 44.137.40.2 in your encap table?
I would expect you to send it into an IPIP tunnel direct to that system.
Or is there some magic in amprgw that makes it ignore IPIP tunnels within BGP
routed subnets? That would cause this to fail...
> I can ping 213.222.29.194 from 169.228.34.84, but not from
> 44.0.0.1.
But you can ping 44.137.0.1 (the amprnet address of that same system) from 44.0.0.1 ?
> It looks, at first glance, like there's a 44 path problem in
> 213.222.29.194.
When you route back to 44.137.40.2 via 213.222.29.194 it is explained, because that
system has a stateful firewall that does not expect replies to traffic that has not
gone out via that same system.
Rob
