On 1/29/14 2:05 PM, Steve Wright wrote:
This mesh crap really needs to be binned, or at the very least not try and do anything important over it, such as route an entire /16. If you want to connect a /24 with it to make a neat local play toy then go for it, but using it as an enterprise routing tool is absurd at the very least, and at it's WORST, it's very likely to just completely stop anyone from trying to build anything new over it because it's connectivity and throughput sucks.
This.
So this is how I'd see it work, I need to write up a proposal for it.
You have regional BGP routers that route subnets to the internet. These could then tunnel the subnets to end users via GRE. End users could route via an IGP over this tunnel to the regional speaker(s). Multiple tunnels would give redundancy.
The regional speakers would have a tunnel between them.
In the event of an outage the other BGP speakers would route subnets. Multiple links from end users to other BGP speakers (or non-speakers that are aggravation routers) would provide redundancy to the end users.
Of course nothing prevents having a direct BGP speaker with an RF link to end users, most data centers will not have roof rights however.
We could setup redistribution that would pull announcements from BGP if end nodes went down.
Each BGP speaker could announce the subnets it knows about and a /8 providing we have a mesh of the backbone bgp speakers.
Thoughts?
That is basically what I have in mind. Though I would allow the regionals to determine the VPN protocols to reach the subnets (more than one, including possibly IPIP).
Then we should have plug and play configurations for subnets. For example, I can provision a MikroTik with a VPN to a datacenter and deliver it to subnet admin for $50-100, who places on their LAN with connectivity to an ISP. Going out from there to RF or a service is the responsibility of the local subnet.
------------------------------ John D. Hays K7VE PO Box 1223, Edmonds, WA 98020-1223 http://k7ve.org/blog http://twitter.com/#!/john_hays http://www.facebook.com/john.d.hays
On Wed, Jan 29, 2014 at 12:04 PM, Bryan Fields Bryan@bryanfields.netwrote:
(Please trim inclusions from previous messages) _______________________________________________ On 1/29/14 2:05 PM, Steve Wright wrote:
This mesh crap really needs to be binned, or at the very least not try
and
do anything important over it, such as route an entire /16. If you want
to
connect a /24 with it to make a neat local play toy then go for it, but using it as an enterprise routing tool is absurd at the very least, and
at
it's WORST, it's very likely to just completely stop anyone from trying
to
build anything new over it because it's connectivity and throughput
sucks.
This.
So this is how I'd see it work, I need to write up a proposal for it.
You have regional BGP routers that route subnets to the internet. These could then tunnel the subnets to end users via GRE. End users could route via an IGP over this tunnel to the regional speaker(s). Multiple tunnels would give redundancy.
The regional speakers would have a tunnel between them.
In the event of an outage the other BGP speakers would route subnets. Multiple links from end users to other BGP speakers (or non-speakers that are aggravation routers) would provide redundancy to the end users.
Of course nothing prevents having a direct BGP speaker with an RF link to end users, most data centers will not have roof rights however.
We could setup redistribution that would pull announcements from BGP if end nodes went down.
Each BGP speaker could announce the subnets it knows about and a /8 providing we have a mesh of the backbone bgp speakers.
Thoughts?
-- Bryan Fields
727-409-1194 - Voice 727-214-2508 - Fax http://bryanfields.net _________________________________________ 44Net mailing list 44Net@hamradio.ucsd.edu http://hamradio.ucsd.edu/mailman/listinfo/44net
John would like to talk with you as I would be interested to know if mikrotik can be used instead of the rip44d.
Thanks
Sent from my iPhone
On Jan 29, 2014, at 3:20 PM, K7VE - John k7ve@k7ve.org wrote:
(Please trim inclusions from previous messages) _______________________________________________ That is basically what I have in mind. Though I would allow the regionals to determine the VPN protocols to reach the subnets (more than one, including possibly IPIP).
Then we should have plug and play configurations for subnets. For example, I can provision a MikroTik with a VPN to a datacenter and deliver it to subnet admin for $50-100, who places on their LAN with connectivity to an ISP. Going out from there to RF or a service is the responsibility of the local subnet.
John D. Hays K7VE PO Box 1223, Edmonds, WA 98020-1223 http://k7ve.org/blog http://twitter.com/#!/john_hays http://www.facebook.com/john.d.hays
On Wed, Jan 29, 2014 at 12:04 PM, Bryan Fields Bryan@bryanfields.netwrote:
(Please trim inclusions from previous messages) _______________________________________________
On 1/29/14 2:05 PM, Steve Wright wrote: This mesh crap really needs to be binned, or at the very least not try
and
do anything important over it, such as route an entire /16. If you want
to
connect a /24 with it to make a neat local play toy then go for it, but using it as an enterprise routing tool is absurd at the very least, and
at
it's WORST, it's very likely to just completely stop anyone from trying
to
build anything new over it because it's connectivity and throughput
sucks.
This.
So this is how I'd see it work, I need to write up a proposal for it.
You have regional BGP routers that route subnets to the internet. These could then tunnel the subnets to end users via GRE. End users could route via an IGP over this tunnel to the regional speaker(s). Multiple tunnels would give redundancy.
The regional speakers would have a tunnel between them.
In the event of an outage the other BGP speakers would route subnets. Multiple links from end users to other BGP speakers (or non-speakers that are aggravation routers) would provide redundancy to the end users.
Of course nothing prevents having a direct BGP speaker with an RF link to end users, most data centers will not have roof rights however.
We could setup redistribution that would pull announcements from BGP if end nodes went down.
Each BGP speaker could announce the subnets it knows about and a /8 providing we have a mesh of the backbone bgp speakers.
Thoughts?
-- Bryan Fields
727-409-1194 - Voice 727-214-2508 - Fax http://bryanfields.net _________________________________________ 44Net mailing list 44Net@hamradio.ucsd.edu http://hamradio.ucsd.edu/mailman/listinfo/44net
On Wed, Jan 29, 2014 at 3:15 PM, Chad cstar707@gmail.com wrote:
John would like to talk with you as I would be interested to know if mikrotik can be used instead of the rip44d.
Yes, I wrote a script to create the IPIP tunnels and routes on ROS, just as rip44d does for Linux. We (HamWAN) are using this script to add the AMPR IPIP mesh to our BGP-announced subnet. This is necessary because of the router upstream of amprgw that sends 44/8 back to amprgw.
https://github.com/HamWAN/hamwan_scripts/tree/master/amprupdate
Unfortunately, this doesn't run on the Mikrotik device itself* (I could not find a way to parse the encap file with ROS scripting language). It will need to be run on another device that has Python and can SSH into your Mikrotik router. Run it via cron on a regular interval to stay up-to-date with the encap file. You'll need to replace the addresses on lines 27 and 28 with your subnet and Internet address[es].
Tom KD7LXL
* It's theoretically possible to run this script, or even rip44d, in a Linux VM on your Mikrotik router, therefore not requiring another machine.
Thanks Tom will look into this thanks for sharing.
will try to set this up this weekend.
Chad Starling 4164772423 011 883 5100 0990 0046 INUM VA3CWS
On Wed, Jan 29, 2014 at 7:25 PM, Tom Hayward esarfl@gmail.com wrote:
(Please trim inclusions from previous messages) _______________________________________________ On Wed, Jan 29, 2014 at 3:15 PM, Chad cstar707@gmail.com wrote:
John would like to talk with you as I would be interested to know if
mikrotik can be used instead of the rip44d.
Yes, I wrote a script to create the IPIP tunnels and routes on ROS, just as rip44d does for Linux. We (HamWAN) are using this script to add the AMPR IPIP mesh to our BGP-announced subnet. This is necessary because of the router upstream of amprgw that sends 44/8 back to amprgw.
https://github.com/HamWAN/hamwan_scripts/tree/master/amprupdate
Unfortunately, this doesn't run on the Mikrotik device itself* (I could not find a way to parse the encap file with ROS scripting language). It will need to be run on another device that has Python and can SSH into your Mikrotik router. Run it via cron on a regular interval to stay up-to-date with the encap file. You'll need to replace the addresses on lines 27 and 28 with your subnet and Internet address[es].
Tom KD7LXL
- It's theoretically possible to run this script, or even rip44d, in a
Linux VM on your Mikrotik router, therefore not requiring another machine. _________________________________________ 44Net mailing list 44Net@hamradio.ucsd.edu http://hamradio.ucsd.edu/mailman/listinfo/44net
Hi Bryan,
You have regional BGP routers that route subnets to the internet. These
could
then tunnel the subnets to end users via GRE. End users could route via
an
IGP over this tunnel to the regional speaker(s). Multiple tunnels would
give
redundancy.
This is exactly what BGP enabled subnets are intended for and the one's already set up work as described. What you do in that subnet is every subnet's own busines. But why GRE? it has a bigger overhead compared to IPIP...
The regional speakers would have a tunnel between them.
Also, everyone can link their directly routed subnets as they wish. The current arhitecture doesn't prohibit such links.
In the event of an outage the other BGP speakers would route subnets. Multiple links from end users to other BGP speakers (or non-speakers that
are
aggravation routers) would provide redundancy to the end users.
Again this is a subnet internal isuue what you du with your end users.
Of course nothing prevents having a direct BGP speaker with an RF link to
end
users, most data centers will not have roof rights however.
The current setups allow the same thing. No one disallows this.
We could setup redistribution that would pull announcements from BGP if
end
nodes went down.
If you have redundant BGP enabled routers in the BGP announced subnet this will happen.
Each BGP speaker could announce the subnets it knows about and a /8
providing
we have a mesh of the backbone bgp speakers.
This is how BGP works... The ampr gateway is the /8 announcer
So nothing new in these idees. It is exactly how the BGP enabled subnets work right now.
As I stated, the rest of the ampr network is just another BGP announced subnet, this time the /8 covering the address space not announced by other BGP enabled subnets. So I really don't understand the issue. This is exactly what we have right now. What you do in your own subnet behind the registered gw or behind your BGP router is your personal stuff and does not affect the rest of the ampr network. If you want extra GRE tunnels, set them up, if you want redundant links via tunnels with other announcers, do it. Everything is compliant with our current setup. And if you don't want IPIP use BGP routed acces and you will not need it anymore. For acces to hosts outside your network, you will be routed via amprgw, and everything will work.
But please don't try to enforce your internal network householding on others, since not everyone can afford a BGP subnet. In my case a BGP enabled acces is considered of professional use and is about my monthly income, which of course I am not able to support.
73 de Marius, YO2LOJ
On Wed, 29 Jan 2014, Marius Petrescu wrote:
And if you don't want IPIP use BGP routed acces and you will not need it anymore. For acces to hosts outside your network, you will be routed via amprgw, and everything will work.
Sorry, did I accidentally miss an email or two, has something changed around amprgw that would make the above happen?
Last time I checked, amprgw could not route out any unencapsulated packets that have a destination address within 44/8. These would be packets from the IPIP-connected gateways going to a BGP-only site (most IPIP sites can not send unencapsulated outgoing packets with 44/8 source addresses due to spoofing filtering at ISPs). The reason was that UCSD's internal network routes all 44/8 destined packets to amprgw, so amprgw can not send packets to 44/8 BGP sites at all.
As I understand it, currently all BGP sites must have an IPIP gateway too to enable connectivity with all the rest of the non-BGP sites.
Sorry for the noise if things have changed since.
But please don't try to enforce your internal network householding on others, since not everyone can afford a BGP subnet. In my case a BGP enabled acces is considered of professional use and is about my monthly income, which of course I am not able to support.
Agreeing with all the other things you wrote!
We have a BGP setup over here now, and we are locally routing the subnets forward with either IPIP or OpenVPN (could do GRE if needed), and we could set up an IGP on top of those just as well.
Further, if there would be a local gateway, with a subnet within 44.139/16, that has a Gateway entry in the portal, it'd automatically get direct IPIP routing via our BGP gateway (instead of going via UCSD) because we have the IPIP endpoint present as well (and it gets the RIP updates).
- Hessu, OH7LZB
Tnx Hessu for the correction.
What I meant (and what seems logical to me) whas that if you try to reach an ampr host in the mesh network (let's call a host here M) from a bgp routed subnet, than you have tis scenario: - On the router of the upstram provider there shpuld be 2 routes. One to the BGP'd subnet (let's assume it called A 44.1.1.0/24) and one to 44.0.0.0/8 which is amprgw which is the gw for any 44 traffic. - If you try a connection from A to M, packets from A will go out "in the wild" and will be routed to amprgw, encapsulated, and sent to M, like any internet to ampr access. The responses will flow back the correct route because of conntrack. - Any two bgp enabled subntes will talk to each other like any subnet ion the internet. - Outgoing connections from M will not work since amprgw does not allow outgoing connects to 44 addresses. IMHO this rule should be refined to be "drop outgoing connections to hosts present in the encap file" which will solve the problem. But that's another thing. - Now, since A is not in the encap file, I would expect any gateway to masquerade the traffic to A to its own public IP, ensuring a regular connection to it, like a connection to any internet host. The only issue in this case is that the originator will not be identified as an ampr host, so here come those direct tunnels into play, and of course, BGP can be used internally to get dynamic routing. Allthough I favor OSPF for intranet routing.
Maybe now I was more explicit :-)
Marius, YO2LOJ
-----Original Message----- From: 44net-bounces+marius=yo2loj.ro@hamradio.ucsd.edu [mailto:44net-bounces+marius=yo2loj.ro@hamradio.ucsd.edu] On Behalf Of Heikki Hannikainen Sent: Thursday, January 30, 2014 07:47 To: AMPRNet working group Subject: Re: [44net] Updated BGP design (was 44Net Digest, Vol 3, Issue 20)
Last time I checked, amprgw could not route out any unencapsulated packets that have a destination address within 44/8. These would be packets from the IPIP-connected gateways going to a BGP-only site (most IPIP sites can not send unencapsulated outgoing packets with 44/8 source addresses due to spoofing filtering at ISPs). The reason was that UCSD's internal network routes all 44/8 destined packets to amprgw, so amprgw can not send packets to 44/8 BGP sites at all.
As I understand it, currently all BGP sites must have an IPIP gateway too to enable connectivity with all the rest of the non-BGP sites.
On Thu, 30 Jan 2014, Marius Petrescu wrote:
What I meant (and what seems logical to me) whas that if you try to reach an ampr host in the mesh network (let's call a host here M) from a bgp routed subnet, than you have tis scenario:
- On the router of the upstram provider there shpuld be 2 routes. One to the
BGP'd subnet (let's assume it called A 44.1.1.0/24) and one to 44.0.0.0/8 which is amprgw which is the gw for any 44 traffic.
- If you try a connection from A to M, packets from A will go out "in the
wild" and will be routed to amprgw, encapsulated, and sent to M, like any internet to ampr access. The responses will flow back the correct route because of conntrack.
conntrack? I don't think the gateways out there are running any sort of routing that would magically route back return packets "the same way". If connection tracking is used, it might be applied to firewall rules, but not routing. Besides, that "back the same way" would point to amprgw, and again, amprgw is not able to route those return packets back to A.
M will use the routing table it has, which might be one of:
1) default route to the Internet (will get dropped at upstream ISP due to the 44/8 source address)
2) some folks have a 44/8 route pointing to amprgw (won't work since amprgw currently cannot send packets to the 44/8 BGP sites)
3) if the BGP site has a Gateways database entry and an IPIP receiving endpoint, that will be used, and things work.
- Any two bgp enabled subntes will talk to each other like any subnet ion
the internet.
Correct.
- Outgoing connections from M will not work since amprgw does not allow
outgoing connects to 44 addresses. IMHO this rule should be refined to be "drop outgoing connections to hosts present in the encap file" which will solve the problem. But that's another thing.
amprgw itself does allow them, but its local upstream routers will give all packets destined to 44/8 back to amprgw.
The reality is that all outgoing *packets* from amprgw to 44/8, _unless_ _encapsulated_, will not go anywhere, since UCSD routers route 44/8 to amprgw and do not know about the more specific BGP routes.
That is not affected by the direction of a TCP connection, conntrack, or anything else.
- Hessu
Hi, I would appreciate any advice from this thread from someone that actually managed to introduce local BGP peering of a subnet and to set up whatever it takes not to disconnect this subnet from the rest of the 44 network. I am coordinating the Swedish delegation 44.140.0.0/16. Last year we made a complete restart of this network since it was totally dead after a very active pioneer period. I was unfortunately not part of that period and lack experience from the tunnelled version of the network. We now have an agreement both with ARDC and SUNET (the Swedish University Network) to provide Internet transit for this space locally via SUNET. SUNET is connected to Nordunet, which is connected to the European academic backbone GÉANT, which is connected all over the world. SUNET accepts announcements of subnets down to /24. This makes it possible for us to spread SUNET/se.amprnet gateways at SUNET points of presence all over the country. We are in the process of connecting amateur clubs to such gateways that will serve their individual members. We are also adding ipv6 support into this arrangement. Is it correctly understood from this thread that we should set up a fat tunnel in se.ampr between SUNET and 44.0.0.1? and route to 44.0.0.0/8 through it? There has been talk about both gre and ipip-tunnels. In cases where we still have to tunnel in 44.140/16 (hopefully temporarily) we are using gre-tunnels since they also support ipv6. But I guess we can use either for this particular tunnel. Anything else to think about to get it working?
Bjorn/sa0bxi
Let me get this straight: UCSD routers are not BGP enabled?
Marius, YO2LOJ
-----Original Message----- From: 44net-bounces+marius=yo2loj.ro@hamradio.ucsd.edu [mailto:44net-bounces+marius=yo2loj.ro@hamradio.ucsd.edu] On Behalf Of Heikki Hannikainen Sent: Thursday, January 30, 2014 10:48 To: AMPRNet working group Subject: Re: [44net] Updated BGP design (was 44Net Digest, Vol 3, Issue 20)
amprgw itself does allow them, but its local upstream routers will give all packets destined to 44/8 back to amprgw.
The reality is that all outgoing *packets* from amprgw to 44/8, _unless_ _encapsulated_, will not go anywhere, since UCSD routers route 44/8 to amprgw and do not know about the more specific BGP routes.
On Thu, Jan 30, 2014 at 06:09:13PM +0200, Marius Petrescu wrote:
Let me get this straight: UCSD routers are not BGP enabled?
The border routers are, of course, BGP enabled. However, there are several routers between the campus border and amprgw, and those intermediate routers don't do BGP because they're not on the border.
I hope to have that changed someday but re-engineering the entire campus network architecture to accomodate one host isn't a high priority. - Brian
On 01/30/2014 05:15 PM, Brian Kantor wrote:
The border routers are, of course, BGP enabled. However, there are several routers between the campus border and amprgw, and those intermediate routers don't do BGP because they're not on the border.
But shouldnt it be possible to have a single fat tunnel between amprgw and a border router of an other bgp island, for example se.ampr.org, which is managed by the IGP (in our case ospf) rather than bgp?
You have regional BGP routers that route subnets to the internet. These could then tunnel the subnets to end users via GRE. End users could route via an IGP over this tunnel to the regional speaker(s). Multiple tunnels would give redundancy.
Not throwing stones here but we all saw what happen when URO went down a few years ago. This topology did not work well for us, I would have seen Hub0 as a regional routers.
People lives change all the time so keeping it simple work well and has worked well for a lot of years.
I for one would prefer to route my own Gateway even if I have to deal with a little latency.
Jerry, kd4yal
-
Bjorn Pehrson -
Brian Kantor -
Bryan Fields -
Chad -
Chad Starling
-
Heikki Hannikainen -
Jerald A DeLong -
K7VE - John -
Marius Petrescu -
Tom Hayward