Hmm. We saw some unusually high traffic on the link late last week, which I thought might be the culprit, but that's died down, and if the packet loss is continuing, there may be some other issue. I've fired off an email to the UCSD netops guys to see if they know of any problems up at CENIC. Dan On 11/10/20 6:59 PM, Albert Lawson via 44Net wrote:

All, I should also note that I did not observe the issue constantly since Friday. The loss subsided over the weekend. I also haven't tested since approximately Monday - until the emails today. - KB3VWG

It certainly seems to make a big difference depending on where in the world the packets are coming from (I’m still seeing no packet loss to/from my UK servers). Hopefully the net ops guys at UCSD can shed more light on this (thanks Dan). Chris - G1FEF

On Wed, 11 Nov 2020, G1FEF via 44Net wrote: Fundemental problem here (for debugging network issues) is that neither ARDC/Amprnet/44Net (...nor CAIDA), have full direct control, or access, to their own network instructure. CAIDA/UCSD is beholden to random upstream filtering as much in 2001[1] and 2009[2] as it is now. And in-turn, AmprGW is/was beholden (2019) to an ethernet switch with customised firmware[3]... a blackbox. Graphs used to show the packet flow for Network Telescope (44net): https://www.caida.org/data/realtime/telescope/?monitor=telescope_attack&… but those broke (~2019-12-11), giving less insight for debugging. --------- ARDC now has the resources to put independently hosted redundancy in place for 44net. Perhaps it would make sense to prioritise AmprGW infrastructure over distributions of funds to other organisations? [4] -Paul [1] Shannon (2001) https://www.caida.org/research/security/code-red/coderedv2_analysis.xml#bac… "filter was put into place upstream of the monitor, we were unable to capture IP packet headers after 16:30 UTC" [2] Polterock (2012) https://blog.caida.org/best_available_data/2012/04/04/targeted-serendipity-… "increase in the amount of data stored after April 2009 due to the removal of an upstream rate limit filter on incoming packets" [3] Kantor (October 2019) "I don't know the make and model of the switch that is crashing. I believe it may be running locally-modified firmware." [4] ie. put a moratorium on the fun "grant giving" side of the business, until the primary/core mission of bringing AmprGW services under full control has been solved.

fail2ban? ________________________________________ De : 44Net <44net-bounces+petem001=hotmail.com(a)mailman.ampr.org> de la part de Marius Petrescu via 44Net <44net(a)mailman.ampr.org> Envoyé : 11 novembre 2020 12:59 À : 44net(a)mailman.ampr.org Cc : Marius Petrescu Objet : Re: [44net] Lost Packets Just FYI: I have have some 550 distinct IP addresses trying to connect from the internet via tunnel to unavailable ports via my gateway every hour. What I do is to place them in a list and blacklist them for 1 hour :-) Marius, YO2LOJ On 11.11.2020 18:00, Paul Sladen via 44Net wrote: _________________________________________ 44Net mailing list 44Net(a)mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net

You are too kind I block for three years Paul On 11/11/2020 17:59, Marius Petrescu via 44Net wrote: -- paul(a)theskywaves.net

On 11/11/20 8:00 AM, Paul Sladen via 44Net wrote: I can answer this part, at least. The make and model is a juniper ex3300-48t. It is not running any customized firmware. Rather, the link drops down into a small vlan grouping with the link from UCSD, a link to the ampr gw box, and a port replicator. For whatever reason, when the ampr gw box reboots, there is a chance it takes the switch with it. usually, the switch just reboots, but sometimes it full on crashes and needs a power cycle. It has been pretty stable recently, however. Last flapped : 2019-10-18 01:34:48 PDT (55w5d 08:34 ago) ( on all three ports ) In other news, I'm still waiting to hear back from UCSD. Keep in mind it is a holiday here, so it may be tomorrow. Dan

This may sound blunt, but why do you need to run DMR traffic from/to the internet via 44net if you don't have a BGP announced subnet? Using your public IP for it (since you have a gateway, otherwise you would not need ampr-gw) would actually circumvent all those problems related to the gateway. Also, your partners would experience a round trip time cut in half by connecting directly to you or you connecting to them via public IP instead of the client<->ampr-gw<->client option. And if they also use 44net, then again the gateway should not matter, since all tunnels are full mesh. The AMPR tunneling system is for interlinking isolated subnets and the gateway itself to provide management and occasional internet access, not for providing high speed high performance routing, like those needed for DV traffic. Talking of sustainability, you use the gateway for something it was not meant to support, not mentioning the contribution to clogging the ampr-gw router itself. Marius, YO2LOJ On 12.11.2020 07:15, Albert Lawson via 44Net wrote:

Not wanting to pile on here... but I do recommend for this use case to stand up a lightweight VPS that will do BGP announcements on your behalf and then use that as your own IPIP or GRE tunnel back to your onsite location(s). We have three points of connectivity into a single VPS that are routing our 44Net space to a multisite system that supports a strong mix of DMR, D-STAR, YSF, and Allstar/IAX2 as well as logging, telemetry, etc. Works great and is much more robust than the IPIP mesh. It also allows us to do the IPIP/GRE over IPv6 addressing so we can avoid NATing connections over standard cable modem connections. There are a few providers that will do BGP announcements for you with a cheap VPS, but we have had very good success with FreeRangeCloud.com. Jason N8EI -----Original Message----- From: 44Net &lt;44net-bounces+jason=mfamily.org(a)mailman.ampr.org&gt; On Behalf Of pete M via 44Net Sent: Thursday, November 12, 2020 2:40 PM To: AMPRNet working group &lt;44net(a)mailman.ampr.org&gt; Cc: pete M &lt;petem001(a)hotmail.com&gt; Subject: Re: [44net] Lost Packets I am with Marius on this. Using a single point to route all the 44 net traffic by tunnel is good for small communication that wont suffer from resend or such. But for a voice communication that need to have a steady stream you are looking for trouble. Of course it will work most of the time, But nope it wont work all the time. Pierre VE2PF ________________________________________ De : 44Net &lt;44net-bounces+petem001=hotmail.com(a)mailman.ampr.org&gt; de la part de Marius Petrescu via 44Net &lt;44net(a)mailman.ampr.org&gt; Envoyé : 12 novembre 2020 14:13 À : 44net(a)mailman.ampr.org Cc : Marius Petrescu Objet : Re: [44net] Lost Packets This may sound blunt, but why do you need to run DMR traffic from/to the internet via 44net if you don't have a BGP announced subnet? Using your public IP for it (since you have a gateway, otherwise you would not need ampr-gw) would actually circumvent all those problems related to the gateway. Also, your partners would experience a round trip time cut in half by connecting directly to you or you connecting to them via public IP instead of the client<->ampr-gw<->client option. And if they also use 44net, then again the gateway should not matter, since all tunnels are full mesh. The AMPR tunneling system is for interlinking isolated subnets and the gateway itself to provide management and occasional internet access, not for providing high speed high performance routing, like those needed for DV traffic. Talking of sustainability, you use the gateway for something it was not meant to support, not mentioning the contribution to clogging the ampr-gw router itself. Marius, YO2LOJ On 12.11.2020 07:15, Albert Lawson via 44Net wrote: _________________________________________ 44Net mailing list 44Net(a)mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net _________________________________________ 44Net mailing list 44Net(a)mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net

A reminder to those using icmp echo/reply (ping & traceroute) to measure network performance. Due to DOS attacks many ISPs rate limit ICMP echo/reply traffic because it is usually kicked over to router’s CPU instead of being handled in hardware. I’ve been bitten by this a couple of times trying to diagnose network issues. Neil N0SFH On Thu, Nov 12, 2020 at 6:21 PM K7VE - John via 44Net < 44net(a)mailman.ampr.org&gt; wrote: Sent from Gmail Mobile

Marius, Bluntness is fine. Coming from the old days of packet, I find that my skin is still pretty thick. Understand that I've been out of this for a while. And I'm seeing that what you say makes sense. I have a BGP provider that will advertise my route, but the standard for advertising a subnet via BGP is no less than /24. I'm waiting to hear back from Chris on expanding my allocation from /27 to /24. Thanks, 73's -Albert WB7AWL -----Original Message----- From: 44Net [mailto:44net-bounces+wb7awl=lawsonpc.com@mailman.ampr.org] On Behalf Of Marius Petrescu via 44Net Sent: Thursday, November 12, 2020 11:13 AM To: 44net(a)mailman.ampr.org Cc: Marius Petrescu &lt;marius(a)yo2loj.ro&gt; Subject: Re: [44net] Lost Packets This may sound blunt, but why do you need to run DMR traffic from/to the internet via 44net if you don't have a BGP announced subnet? Using your public IP for it (since you have a gateway, otherwise you would not need ampr-gw) would actually circumvent all those problems related to the gateway. Also, your partners would experience a round trip time cut in half by connecting directly to you or you connecting to them via public IP instead of the client<->ampr-gw<->client option. And if they also use 44net, then again the gateway should not matter, since all tunnels are full mesh. The AMPR tunneling system is for interlinking isolated subnets and the gateway itself to provide management and occasional internet access, not for providing high speed high performance routing, like those needed for DV traffic. Talking of sustainability, you use the gateway for something it was not meant to support, not mentioning the contribution to clogging the ampr-gw router itself. Marius, YO2LOJ On 12.11.2020 07:15, Albert Lawson via 44Net wrote: _________________________________________ 44Net mailing list 44Net(a)mailman.ampr.org https://mailman.ampr.org/mailman/listinfo/44net

Dan, Tried both from the office and at home....this is pretty representative of what I see from both locations.. --- ampr.org ping statistics --- 100 packets transmitted, 67 received, 33% packet loss, time 99798ms rtt min/avg/max/mdev = 32.771/37.410/62.854/5.015 ms --- 132.249.21.111 ping statistics --- 100 packets transmitted, 71 received, 29% packet loss, time 99697ms rtt min/avg/max/mdev = 33.062/36.580/50.200/3.406 ms Thanks -Albert -----Original Message----- From: Daniel E Andersen [mailto:dea@caida.org] Sent: Thursday, November 12, 2020 11:59 AM To: Albert Lawson &lt;wb7awl(a)LAWSONPC.COM&gt;OM>; AMPRNet working group &lt;44net(a)mailman.ampr.org&gt; Subject: Re: [44net] Lost Packets So, I've been told by the UCSD folks that CENIC completed some maintenance involving the LA and Pomona routes this morning. It's perhaps wishful thinking on my part, but are folks still seeing packet loss? I have not been able to replicate the issue, so for now must rely on this list for debugging information. Another possible data point that may be of use. SDSC's website www.sdsc.edu (132.249.21.111 ) should be topologically identical ( or near enough that it shouldn't matter ). If packet loss to this machine is significantly different than to 44.0.0.1, that points us to possible routing issues over a more physical issue. Dan On 11/11/20 9:15 PM, Albert Lawson wrote:

Hi This is the standard output coming on from the amprgw gateway over protocol 4 tcpdump -ni eth# proto 4 on your linux server in/out interface to gatewaay (firewalled) address entry for the 44net subnets that my system serves to mine and my downstream UK link partners over amprnet IP over UDP tunnels) Address 3 and Address 4 would (idealy) normally be axpected to be from 44net. Those that are non 44net, are mostly from attacking scanning system. I am not going to mention countries for too many. How do you determing a valid non amprnet address as being trusted you can not. (I have other security filtering devices (Firewalls) infront of my gb7cip system including it's own denigh all firewall in/out unless a valid rule is in place) So I only see BANDITs scanning attacking via the protocol 4 (tunl0) all get blocked as rule state address3 and address must be from 44net /9 and /10 (restricted valid range as of July 2019) Someone else can do the trace themselves on their server Internet facing interfaces dmz. and output interfaces to their system running their services. This issue is not new. Just getting worse de Paul g4apl

I think he is trying to say he receives: Outer (IPIP) pair: from 169.228.34.84 to his gateway (<###.###.###.###>) -> Inner pair (encapsulated): from <any non 44net address-bandits> to <gb7cip hosted 44net: gateway routes> This is how any internet originated traffic looks like, and everything comes over ampr-gw (44.0.0.1/169.228.34.84). There is not much one can do about it, except unregistering the hosts in the DNS, which will prevent forwarding by the ampr gateway. Marius, YO2LOJ On 11.11.2020 14:27, lleachii--- via 44Net wrote: