On 8/11/21 11:35 AM, Toussaint OTTAVI via 44Net wrote:
For a long time now, everything has worked perfectly for those users who announce on internet BGP and do ALSO register on the IPIP mesh. They know how to route to other IPIP mesh members via a tunnel and route the remainder to internet. They are reachable for everyone that has sane routing.
Of course, it works. But every sysadmin has to know exactly how to route every subnet. F/ex, not everybody knows 44.190 routing policy. Grouping into "Internet" and "Intranet" categories, each category in a parent subnet, would hugely simplify things.
No, it would make no difference whatsoever! The backbone system would internally use BGP to distribute the subnet routes for all client systems connected to it, much like the IPIP mesh now uses RIP to advertise that info. The difference would be that it would be dynamic, as determined by connection of endpoints, not a static portal route list. The "holes" in those routes default to the route to internet, and the 44.190 ranges would fall into that category. There is no need at all for a sysadmin to do anything special for that. Furthermore, with "every sysadmin" you suggest that this would be a task for anyone in the network, while in reality only the backbone sysadmins would be tasked with that, as the remainder of admins either have a static route for 44/9 and 44.128/10 or they have configured BGP to automatically exchange routes with the backbone routers (and their local radio network).
But managing a database on the portal about what subnet is Intranet and what subnet is Internet would probably do the job, too. Some scripts could update routing policies from the portal at POP level, which will be managed by skilled sysadmins. Not really "KISS", but it would work.
No database other than the BGP route table would be required. There is a 0.0.0.0/0 route in the route table pointing to the local internet connection, and several routes to participants in the backbone (similar to what we now have on IPIP) which are more specific and are taken when available. We have this setup here for a long time and I can assure you we do nothing special for 44.190. Our route table looks like this:
# ip route default via 213.222.29.193 dev eth0 onlink unreachable 10.0.0.0/8 44.0.0.0/9 via 213.222.29.193 dev eth0 src 44.137.0.1 44.0.0.1 via 169.228.34.84 dev tunl0 proto ampr-ripd metric 4 onlink 44.2.0.2 via 98.20.210.138 dev tunl0 proto ampr-ripd metric 4 onlink ..... 44.185.112.0/24 via 87.252.188.119 dev tunl0 proto ampr-ripd metric 4 onlink 44.188.1.1 via 70.80.196.6 dev tunl0 proto ampr-ripd metric 4 onlink 44.188.192.222 via 109.227.66.46 dev tunl0 proto ampr-ripd metric 4 onlink unreachable 169.254.0.0/16 unreachable 172.16.0.0/12 unreachable 192.168.0.0/16 213.222.29.192/28 dev eth0 proto kernel scope link src 213.222.29.194 #
The additional features of backbone routers I can think of are: - a reverse-path filter in the backbone routers that drops traffic from internet coming from addresses that are supposed to be tunnel-routed - a source-address filter in the backbone routers that drops traffic from addresses not assigned to anyone
These are protections agains address spoofing from internet
- a feature for users of the network to disallow all traffic from/to any address outside net-44, selectable in the account admin page for backbone users, for users who want (or can) have nothing to do with internet.
With such a "smart backbone" there is no need whatsoever to arrange networks in different categories based on their address. Such settings could be changed at anytime without requiring a renumber.
That is a design issue. It can be easily solved especially when the new backbone network is present that can do BGP annouces itself, so it is completely transparent to the users if each subnet is on the tunneled network, or on a 44-net subnet only announced on internet. It will always be routed correctly without the operator needing to set separate routes.
Separate routes will still be needed at POP level.
Yes, as I said above: automatically created routes for each tunneled peer, and a default route 0.0.0.0/0 for everything else (including what is now 44.190).
Endpoints will just route all to the nearest POP, so that things remain simple. At least for remote endpoints connected to a POP via a tunnel; but for endpoints connected to several neighbors via meshed radio links, things may not be as simple...
-- This makes me think our previous idea of a 3-level topology (backbone, POP, Access) may not be enough. Maybe, we need an intermediate :
- Backbone : world-wide infrastructure managed by ARDC, and providing network background
- Point of Presence (POP) : local / regional platform, connected to the backbone, and providing connectivity to end-users
- (new) Router : Intermediate mesh router, connected to several neighbors via radio links, and contributing to the global routing in relation with nearest POPs
- Access : Small low-cost router (Mikrotik/OpenWRT etc...) or software (VPN client, RPi) offering 44net IPs to endpoints by connecting to one or several POPs with PnP technologies
Of course the whole network can have several layers. In the early discussion there was already mention of separating the routing backbone from the client access PoP, and that could make sense. We have the above setup here, as our routers (MikroTik) cannot reasonably do OpenVPN which is a technology we want to offer. And we also are on IPIP mesh. So we have different routers for both and they exchange routes via BGP. And of course we have users connected via tunnel that are at the same time routers in the radio network, provide local user access via Radio, etc. The beauty of an IP network is that you can connect that all together in a semi-structured way without having to do it the same way everywhere. In our network, we could continue to service users the way we do now, and if there is an ARDC PoP in the region as well users can opt to directly connect to there, and of course we would connect there announcing our route(s). Then everyone can connect the way they want to, with or without internet routing as they wish.
Rob