Good News! Our friends in the CAIDA research group at UCSD have come up with a new machine for amprgw, quite a bit newer than the existing one, and with faster CPU, more cores, and more memory. It also has RAIDed disk and dual power supplies, although unlike the current amprgw, it won't be on a UPS.
(There isn't a UPS in the new building where the new machine is located. Luckily our power is pretty reliable; besides buying utility power, UCSD also generates its own from solar and natural gas sources. And we don't have lightning storms very often here in the almost-desert.)
There will be some minor differences between the machines, of course, although small enough that I hope to move all the services over from the old machine to the new one over the next several weeks.
The primary difference is that the gateway will have a NEW ADDRESS and a slighly DIFFERENT NAME. Instead of being known as 'AMPRGW.SYSNET.UCSD.EDU' as the current one on address 169.228.66.251 is known, the new machine will be 'AMPRGW.UCSD.EDU' (no 'sysnet' in the name), and will be on address 169.228.34.84. You should probably adjust your firewalls soon, letting both machines in for a while, as they will both be operating at the same time as services are moved from old to new. Do not move your tunnel endpoints to the new machine quite yet; that won't work until the routing software is installed and reconfigured for the new address. I'll let you know what it's ready for that. - Brian
Do you want or think it will be good if you will have a UPS ?
Will you be able to place there a UPS for us ?
If answer are Yes maybe we all donate money for UPS .... you just give us what to buy or how much to donate
(There isn't a UPS in the new building where the new machine is located.
No Ronen, I don't think a UPS is worth having. If there was one there already, I'd ask to be plugged into it, but because rack space is a premium in that data center, we really can't have our own.
And we really don't have power problems here in San Diego. I can't remember the last time we had a power failure here at UCSD. I'm sure it was a few years ago. System uptimes in excess of a year are common.
Thanks for asking. - Brian
On Thu, May 25, 2017 at 10:50:02AM +0000, R P wrote:
Do you want or think it will be good if you will have a UPS ? Will you be able to place there a UPS for us ? If answer are Yes maybe we all donate money for UPS .... you just give us what to buy or how much to donate
Brian (and Craig),
At that time, can we seed a dummy /32 address with a route to the NEW_AMPRGW - from the AMPR Test Subnet or from 44.0.0.0/24?
This will allow the new address to enter our tables, especially for those running dynamic firewalls against the Encap list - since it's TBD if the router/GW will move last, or sometime before the completed migration.
So..basically on that day, something like:
- Brian changes GWs at the TBD time - We all change our default route (if in use) - the route to 44.0.0.1 changes - the dummy route is removed
- Lynwood KB3VWG
The current version of the encap file shows a route to 44.128.0.0/16 via the new amprgw address of 169.228.34.84:
route addprivate 44.128/16 encap 169.228.34.84
RIP should already be advertising it. I don't believe this will cause any problems.
Your gateways and firewalls can start learning it, and I can use it to debug the new installation. What we'll do is change the routing at the UCSD border to direct traffic for that one /16 subnet to the new amprgw instead of the old one right away, and after everything appears to be working properly, we'll change the route for the entire 44.0.0.0/8. Both systems already have a secondary address of 44.0.0.1, so that changeover will be automatic.
Of course, the people announcing their own subnets via BGP won't be affected by this change. - Brian
On Thu, May 25, 2017 at 08:45:58AM -0400, lleachii--- via 44Net wrote:
At that time, can we seed a dummy /32 address with a route to the NEW_AMPRGW
- from the AMPR Test Subnet or from 44.0.0.0/24?
This will allow the new address to enter our tables, especially for those running dynamic firewalls against the Encap list - since it's TBD if the router/GW will move last, or sometime before the completed migration.
So..basically on that day, something like:
- Brian changes GWs at the TBD time
- We all change our default route (if in use)
- the route to 44.0.0.1 changes
- the dummy route is removed
Brian,
root@router:~# ip route get 44.128.0.0/16 from 44.60.44.1 44.128.0.0 from 44.60.44.1 via 169.228.34.84 dev tunl0 table 44 cache window 840
Cool, then as long as we have uptime on that day, we should at least continue to get routes and talk to AMPRNet. At any case, OPs will need to change any scripts that seed the Old_AMPRGW IP to 169.228.34.8.
- KB3VWG
Your gateways and firewalls can start learning it, and I can use it to debug the new installation. What we'll do is change the routing at the UCSD border to direct traffic for that one /16 subnet to the new amprgw instead of the old one right away, and after everything appears to be working properly, we'll change the route for the entire 44.0.0.0/8. Both systems already have a secondary address of 44.0.0.1, so that changeover will be automatic.
I've installed the routing software on the new amprgw machine and tested it, and it's working properly. In anticipation of moving the inbound route for 44/8 from the old amprgw to the new one, it is time for people to start using the AMPRGW.UCSD.EDU (169.228.34.84) address for their outgoing tunnel endpoint.
That will require people who have a firewall to adjust their firewall rules and any default routes they're using.
However, it is still necessary to allow the old amprgw.SYSNET.ucsd.edu (169.228.66.251) in past your firewalls, as up until the moment that we switch the 44/8 route from old to new, inbound traffic from the Internet will still be coming from that old address.
Please make the change and test it as soon as practical so that you don't get caught by the switch. Be sure to let me know if you lose connectivity when you switch over to using the new machine for outbound traffic, and we'll try to figure out what went wrong.
I anticipate a week or so before we can make the switch, so please keep it in mind.
Thank you! - Brian
On Wed, May 24, 2017 at 08:03:52PM -0700, Brian Kantor wrote:
Good News! Our friends in the CAIDA research group at UCSD have come up with a new machine for amprgw, quite a bit newer than the existing one, and with faster CPU, more cores, and more memory. It also has RAIDed disk and dual power supplies, although unlike the current amprgw, it won't be on a UPS.
The primary difference is that the gateway will have a NEW ADDRESS and a slighly DIFFERENT NAME. Instead of being known as 'AMPRGW.SYSNET.UCSD.EDU' as the current one on address 169.228.66.251 is known, the new machine will be 'AMPRGW.UCSD.EDU' (no 'sysnet' in the name), and will be on address 169.228.34.84. You should probably adjust your firewalls soon, letting both machines in for a while, as they will both be operating at the same time as services are moved from old to new. Do not move your tunnel endpoints to the new machine quite yet; that won't work until the routing software is installed and reconfigured for the new address. I'll let you know what it's ready for that.
- Brian
Brian,
It's been working since you made the switch, I never changed it back from testing last night.
New_AMPRGW has been my gateway since you had me test.
Thanks, it now works!
- Lynwood KB3VWG
Trace Output: traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets 1 kb3vwg-001.ampr.org (44.60.44.1) 3.513 ms 3.558 ms 3.828 ms 2 amprgw.ucsd.edu (169.228.34.84) 71.588 ms 71.602 ms 71.600 ms 3 169.228.34.82 (169.228.34.82) 71.943 ms 72.390 ms 72.384 ms 4 nodem-core-6807-vlan3095-gw1.ucsd.edu (132.239.254.178) 73.018 ms 73.016 ms 73.838 ms 5 mx0--nodem-core-30ge.ucsd.edu (132.239.254.162) 73.937 ms 73.933 ms 74.068 ms 6 dc-sdg-agg4--ucsd-1.cenic.net (137.164.23.53) 74.189 ms 68.299 ms 69.308 ms 7 dc-tus-agg3--sdg-agg4-100ge.cenic.net (137.164.11.8) 70.865 ms 70.885 ms 71.119 ms 8 dc-lax-agg6--tus-agg3-100ge.cenic.net (137.164.11.6) 71.221 ms 72.995 ms 73.022 ms 9 74.125.49.165 (74.125.49.165) 73.012 ms 73.136 ms 73.128 ms 10 108.170.247.225 (108.170.247.225) 73.764 ms 73.707 ms 108.170.247.161 (108.170.247.161) 73.756 ms 11 216.239.51.79 (216.239.51.79) 74.742 ms 209.85.255.189 (209.85.255.189) 74.278 ms 216.239.62.81 (216.239.62.81) 74.310 ms 12 google-public-dns-a.google.com (8.8.8.8) 70.488 ms 70.494 ms 70.758 ms done ...
I have a concern that many gateway operators won't get the notification that the amprgw gateway is changing address. If you know someone who operates a gateway and DOESN'T subscribe to this mailing list, please let them know about the upcoming change.
And it would be really cool if someone could browse the wiki and update the information on amprgw. I'm afraid that its address may be hardcoded in some of the 'how-to' pages. - Brian
On Fri, May 26, 2017 at 11:28:51AM -0700, Brian Kantor wrote:
I've installed the routing software on the new amprgw machine and tested it, and it's working properly. In anticipation of moving the inbound route for 44/8 from the old amprgw to the new one, it is time for people to start using the AMPRGW.UCSD.EDU (169.228.34.84) address for their outgoing tunnel endpoint.
That will require people who have a firewall to adjust their firewall rules and any default routes they're using.
However, it is still necessary to allow the old amprgw.SYSNET.ucsd.edu (169.228.66.251) in past your firewalls, as up until the moment that we switch the 44/8 route from old to new, inbound traffic from the Internet will still be coming from that old address.
Please make the change and test it as soon as practical so that you don't get caught by the switch. Be sure to let me know if you lose connectivity when you switch over to using the new machine for outbound traffic, and we'll try to figure out what went wrong.
I anticipate a week or so before we can make the switch, so please keep it in mind.
Thank you!
- Brian
On Fri, May 26, 2017 at 11:28 AM, Brian Kantor Brian@ucsd.edu wrote:
However, it is still necessary to allow the old amprgw.SYSNET.ucsd.edu (169.228.66.251) in past your firewalls, as up until the moment that we switch the 44/8 route from old to new, inbound traffic from the Internet will still be coming from that old address.
Please make the change and test it as soon as practical so that you don't get caught by the switch. Be sure to let me know if you lose connectivity when you switch over to using the new machine for outbound traffic, and we'll try to figure out what went wrong.
Can you anycast the old address from the new server so there's no "cutoff" during the transition? Actually, this might not be a bad thing to run long-term. Keep both servers online and anycasting. When one goes down, pull the route to 169.228.66.251 and packets will continue flowing to the other server. You could even do this at various locations around the world to reduce internet latency for AMPRGW users... (permission to announce 169.228.66.251/24 or some other /24 required)
Tom KD7LXL
Sorry, no.
I don't have any way to set up anycasting without requesting some changes from the campus infrastructure folks that I'm hesitant to ask for, especially as they'd be short term only.
As for long-term, the old server will be repurposed as soon as we're through with it, so it won't be around to participate. (In a sense, we're lucky we've had it as long as we have.) - Brian
On Fri, May 26, 2017 at 01:23:29PM -0700, Tom Hayward wrote:
Can you anycast the old address from the new server so there's no "cutoff" during the transition?
What I can do is leave the old server up at its old address and have it continue forwarding packets that are still sent to it from people who didn't get the word, and contact them to get them to update before I have to shut down the old machine. We'll send their inbound traffic to them from the new machine, but it's anybody's guess if they'll get it because of firewalling.
I really can't do anything about the outbound addresses being different though, since the two machines are on distinctly different subnets and I can't change the campus backbone. I've already had to hassle the campus networking folks to get the RPF disabled, and I really don't like to bother them. - Brian
On Fri, May 26, 2017 at 01:23:29PM -0700, Tom Hayward wrote:
Can you anycast the old address from the new server so there's no "cutoff" during the transition?
Hey Brian,
What about configuring an ICMP direct on the old IP to point to the new IP?
--David KI6ZHD
On 05/26/2017 02:03 PM, Brian Kantor wrote:
(Please trim inclusions from previous messages) _______________________________________________ What I can do is leave the old server up at its old address and have it continue forwarding packets that are still sent to it from people who didn't get the word, and contact them to get them to update before I have to shut down the old machine. We'll send their inbound traffic to them from the new machine, but it's anybody's guess if they'll get it because of firewalling.
I really can't do anything about the outbound addresses being different though, since the two machines are on distinctly different subnets and I can't change the campus backbone. I've already had to hassle the campus networking folks to get the RPF disabled, and I really don't like to bother them.
- Brian
Interesting idea. It might be difficult; there's no convenient way to send one on FreeBSD.
Don't most hosts and routers ignore ICMP Redirects out of security concerns these days? - Brian
On Fri, May 26, 2017 at 04:04:24PM -0700, David Ranch wrote:
Hey Brian, What about configuring an ICMP direct on the old IP to point to the new IP? --David KI6ZHD
Brian,
Nope...that shouldn't be blocked.
I need to read up...I believe what I understood David to ask is: if someone continues to send IPENCAP packets to Old_AMPRGW, configure it to send an ICMP Redirect in the future.
This should WORK, since the device sent the packet. I'm not sure what an OS will do, though; or if the OS is handling the packet as normal (i.e. some raw modes).
This is not considering a scenario if the OP EXPLICITLY blocks ICMP (which in that case, it would fail anyway). My configuration should have received the ICMP Type 5 (I believe 5.1) packet in return for the IPENCAP I sent to Old_AMPRGW, I just don't know how LEDE would respond.
- KB3VWG
Don't most hosts and routers ignore ICMP Redirects out of security concerns these days?
What do i need to do in my MikroTik ?
Just to change the Destination of the tunnel ?
________________________________ From: 44Net 44net-bounces+ronenp=hotmail.com@hamradio.ucsd.edu on behalf of Brian Kantor Brian@UCSD.Edu Sent: Friday, May 26, 2017 11:28 AM To: 44net@hamradio.ucsd.edu Subject: Re: [44net] Good News! and some changes coming
(Please trim inclusions from previous messages) _______________________________________________ I've installed the routing software on the new amprgw machine and tested it, and it's working properly. In anticipation of moving the inbound route for 44/8 from the old amprgw to the new one, it is time for people to start using the AMPRGW.UCSD.EDU (169.228.34.84) address for their outgoing tunnel endpoint.
Yes, but only AFTER the change, since the RIP broadcasts and the incoming internet connections are still via the old gateway.
(It could be fixed to work in the current config, but the change is quite big, and will have to be undone after the new config is working)
On 27.05.2017 00:11, R P wrote:
(Please trim inclusions from previous messages) _______________________________________________ What do i need to do in my MikroTik ?
Just to change the Destination of the tunnel ?
From: 44Net 44net-bounces+ronenp=hotmail.com@hamradio.ucsd.edu on behalf of Brian Kantor Brian@UCSD.Edu Sent: Friday, May 26, 2017 11:28 AM To: 44net@hamradio.ucsd.edu Subject: Re: [44net] Good News! and some changes coming
(Please trim inclusions from previous messages) _______________________________________________ I've installed the routing software on the new amprgw machine and tested it, and it's working properly. In anticipation of moving the inbound route for 44/8 from the old amprgw to the new one, it is time for people to start using the AMPRGW.UCSD.EDU (169.228.34.84) address for their outgoing tunnel endpoint.
I havnt succeeded to work with the RIP yet
but i can work now via the new gateway
i had to open additional tunnel to the new AMPR gateway and then to change the route to the new ampr gateway
Brian what about the statistics ? where are they taken now from ?
________________________________ From: 44Net 44net-bounces+ronenp=hotmail.com@hamradio.ucsd.edu on behalf of Marius Petrescu marius@yo2loj.ro Sent: Friday, May 26, 2017 2:29 PM To: AMPRNet working group Subject: Re: [44net] Good News! and some changes coming
(Please trim inclusions from previous messages) _______________________________________________ Yes, but only AFTER the change, since the RIP broadcasts and the incoming internet connections are still via the old gateway.
I can turn on the RIP sender in the new gateway any time, but I don't know about the wisdom of having both sending at the same time, even though they'll send the same content. Marius, what do you think? - Brian
On Sat, May 27, 2017 at 12:29:46AM +0300, Marius Petrescu wrote:
Yes, but only AFTER the change, since the RIP broadcasts and the incoming internet connections are still via the old gateway.
No issues with turning them on as long as the RIP content is the same...
For the mikrotiks and possible others using individual tunnels , they can be switched only if you DO turn on the new sender, since they have defined tunnel endpoints and do not "cross-listen" for RIP.
Amprd was designed from the beginning to update its default route according to the gateway of 44.0.0.1 automatically, so it will expect a 44.0.0.1 via 169.228.34.84 and use the new one without any intervention.
I assumed this could happen some day so there is no hard coded reference for the gateway IP in any of the tools.
Sorry for the late response.
Marius, YO2LOJ
On 27.05.2017 01:24, Brian Kantor wrote:
(Please trim inclusions from previous messages) _______________________________________________ I can turn on the RIP sender in the new gateway any time, but I don't know about the wisdom of having both sending at the same time, even though they'll send the same content. Marius, what do you think?
- Brian
On Sat, May 27, 2017 at 12:29:46AM +0300, Marius Petrescu wrote:
Yes, but only AFTER the change, since the RIP broadcasts and the incoming internet connections are still via the old gateway.
44Net mailing list 44Net@hamradio.ucsd.edu http://hamradio.ucsd.edu/mailman/listinfo/44net
The content should be the same, it's built from the same data by the same program.
I've turned on the RIP sender on the new gateway. It runs 3 seconds after the 5-minute marks so as not to stomp on the old gateway which sends right on the mark.
Let me know if there are any problems. It's easy to turn it off again if needed, it just runs out of crontab. - Brian
On Sat, May 27, 2017 at 01:42:28AM +0300, Marius Petrescu wrote:
No issues with turning them on as long as the RIP content is the same...
For the mikrotiks and possible others using individual tunnels , they can be switched only if you DO turn on the new sender, since they have defined tunnel endpoints and do not "cross-listen" for RIP.
Amprd was designed from the beginning to update its default route according to the gateway of 44.0.0.1 automatically, so it will expect a 44.0.0.1 via 169.228.34.84 and use the new one without any intervention.
I assumed this could happen some day so there is no hard coded reference for the gateway IP in any of the tools.
Sorry for the late response.
Marius, YO2LOJ
Brian,
If you haven't consider this...make sure that the cron jobs aren't configured to execute on both machines at the same time. I'm not sure how you configured the intervals, but if you're able to set them 60-180 seconds apart, that should work.
My machine (and others that receive IPENCAP from ANY) may exhibit odd behavior if receiving duplicate routes...especially while the RIP44 daemon is executing the routes.
This also make me think about the ICMP Redirect thing...how bout changing the password to the routes on Old_AMPRGW, that will be a first phase-out for OPs running RIP44, as they should observe they can't send traffic to us any longer. After that, the only remaining OPs on Old_AMPRGW would be those who use the Munge Script method. At some point, you will turn off Old_AMPRGW anyway, something should have been noticed by that point in time.
- Lynwood KB3VWG
No issues with turning them on as long as the RIP content is the same...
The only things that might collide on users systems are the RIP transmissions, and I already have them separated in time by three seconds, that's three times the length of the transmission.
I'm in the process of sending an email to the registered email address of every single gateway registered with the portal which will tell them about the changes coming. I feel that this is necessary because a lot of them don't subscribe to the 44net mailing list, despite my strong recommendation that they do so.
We'll see how many complaints I get. - Brian
On Sat, May 27, 2017 at 06:31:48AM -0400, lleachii--- via 44Net wrote:
(Please trim inclusions from previous messages) _______________________________________________ Brian,
If you haven't consider this...make sure that the cron jobs aren't configured to execute on both machines at the same time. I'm not sure how you configured the intervals, but if you're able to set them 60-180 seconds apart, that should work.
My machine (and others that receive IPENCAP from ANY) may exhibit odd behavior if receiving duplicate routes...especially while the RIP44 daemon is executing the routes.
This also make me think about the ICMP Redirect thing...how bout changing the password to the routes on Old_AMPRGW, that will be a first phase-out for OPs running RIP44, as they should observe they can't send traffic to us any longer. After that, the only remaining OPs on Old_AMPRGW would be those who use the Munge Script method. At some point, you will turn off Old_AMPRGW anyway, something should have been noticed by that point in time.
- Lynwood
KB3VWG
No issues with turning them on as long as the RIP content is the same...
44Net mailing list 44Net@hamradio.ucsd.edu http://hamradio.ucsd.edu/mailman/listinfo/44net
On 27/05/2017 11:11 PM, Brian Kantor wrote:
(Please trim inclusions from previous messages) _______________________________________________ The only things that might collide on users systems are the RIP transmissions, and I already have them separated in time by three seconds, that's three times the length of the transmission.
I'm in the process of sending an email to the registered email address of every single gateway registered with the portal which will tell them about the changes coming. I feel that this is necessary because a lot of them don't subscribe to the 44net mailing list, despite my strong recommendation that they do so.
We'll see how many complaints I get.
That would be helpful. I have been (mostly) following the discussions here, but because things tend to get a bit fragmented on the mailing list as details come out over time it is difficult for me to follow. Having it all in one email will definitely help, then I can sit down and make all the necessary changes.
As with Lynwood... and since the mailman server at tapr.org is not functioning at the moment...
All references to the old IP in the pages and files on https://n1uro.ampr.org/linuxconf/ have been updated with the new IP for amprgw as have been dotun-install (a quick auto-install for an ipencap gateway on linux or a pi), along with both my sysv and systemd installers for URONode now reflect the new IP as well.
For those who I co-sysop, your gateways should be up to date and running on the new IP as well.