This is rather disturbing. Many of the gateways in the gateway statistics are showing ZERO encap packets coming from them. One of the busiest normally is the gateway for Germany, 141.75.245.225, which has not sent ANY encap packets to amprgw since the current stats began
Is that still the case? I receive plenty of traffic from that gateway, so when you don't there could be someone who inserted a protocol 4 filter along the way. I can still ping 44.0.0.1 via the tunnel...
traceroute to 169.228.66.251 (169.228.66.251), 30 hops max, 60 byte packets 1 xs4all-router (213.222.29.193) 0.262 ms 0.304 ms 0.276 ms 2 0.ae10.xr4.1d12.xs4all.net (194.109.5.57) 0.322 ms 0.ae11.xr4.1d12.xs4all.net (194.109.5.77) 0.341 ms 0.276 ms 3 asd2-rou-1043.NL.eurorings.net (134.222.93.144) 0.320 ms 0.355 ms asd2-rou-1044.NL.eurorings.net (134.222.97.17) 0.370 ms 4 asd2-rou-1022.NL.eurorings.net (134.222.48.222) 0.615 ms 0.544 ms 0.500 ms 5 chg-s1-rou-1041.US.eurorings.net (134.222.48.90) 95.532 ms 95.539 ms 95.543 ms 6 eq-exchange.tr01-chcgil01.transitrail.NET (206.223.119.116) 95.616 ms 95.675 ms 95.644 ms 7 ae-1.80.rtsw.chic.net.internet2.edu (64.57.20.150) 95.879 ms 95.901 ms 95.890 ms 8 et-3-1-0.4072.rtsw.kans.net.internet2.edu (64.57.20.89) 106.891 ms 106.906 ms 106.909 ms 9 ae-0.80.rtsw.salt.net.internet2.edu (64.57.20.146) 126.801 ms 126.922 ms 126.846 ms 10 ae-2.80.rtsw.losa.net.internet2.edu (64.57.20.144) 139.339 ms 139.437 ms 139.436 ms 11 64.57.20.83 (64.57.20.83) 139.681 ms 139.675 ms 140.043 ms 12 dc-tus-agg3--lax-agg6-100ge.cenic.net (137.164.11.7) 140.662 ms 140.780 ms 140.919 ms 13 dc-sdg-agg4--tus-agg3-100ge.cenic.net (137.164.11.9) 142.328 ms 142.342 ms 142.395 ms 14 dc-ucsd-1--sdg-agg4.cenic.net (137.164.23.54) 141.823 ms 141.889 ms 141.998 ms 15 nodem-core--mx0-30ge.ucsd.edu (132.239.254.163) 142.100 ms 142.099 ms 142.086 ms 16 ebu3b-6509-nodem-core-interconnect-vl910-bcast-255-131.ucsd.edu (132.239.255.131) 142.068 ms 142.094 ms 142.162 ms
Rob
I've asked our campus networking people if they've installed any filters but there's been no reply yet.
However, I doubt they'd block a specific host for a specific protocol; if they did any blocking it would be a complete block of either the host or the protocol, and I can ping the host and other sites are successfully connected using IPIP, so neither would seem to be the case. Perhaps someone in the path between us and Germany inserted a protocol 4 block, or perhaps the German router crashed and lost its routing tables.
No reply from Germany yet either. - Brian
On Wed, May 17, 2017 at 06:47:13PM +0200, Rob Janssen wrote:
Is that still the case? I receive plenty of traffic from that gateway, so when you don't there could be someone who inserted a protocol 4 filter along the way.
Hello,
On Wed, May 17, 2017 at 10:11:26AM -0700, Brian Kantor wrote: [..]
would seem to be the case. Perhaps someone in the path between us and Germany inserted a protocol 4 block, or perhaps the German router crashed and lost its routing tables.
No reply from Germany yet either.
well, I've read it, but I was not able to send an adaequate answer:
we learn the ipip routes at db0fhn. I checked that I could ping a host via that ipip tunnel. I also testet that host from a subnet behind db0fhn. And I tested to ping db0fhn.ampr.org from the internet; success. You said you cannot see inbound traffic from db0fhn. Well, in the first case (ipip mesh to another 44'er host) goes "direct", thus not via ucsd.edu. What kind of traffic do you expect? The rest of 44/8 that is not ipip and not direct-bgp? Actually, db0fhn routes non-advertised 44-ipip-routes (and non-44-hamnet-networks) to the internet (because non-ipip and non-hamnet is either direct-bgp or not-in-use). Perhaps, Jann has changed the 44/8 route; we need to wait for his answer.
vy 73, - Thomas dl9sau
All,
Can the person NJohnson who contributed major edits to:
http://wiki.ampr.org/wiki/Ubuntu_Linux_Gateway_Example
please contact me off-thread.
Thanks and 73,
- Lynwood KB3VWG
Thomas, thank you for your reply. So outbound non-44 destinations from 44.130.0.0/16 are routed out onto the internet at db0fhn instead of being sent to amprgw? But the return path has to go through amprgw. This is non-symmetric routing and I wasn't expecting it. (It's not a problem.)
I have noted in the past that the link was often busy, and I expected it would be busy in both directions. If it's not used for outbound from 44.130.0.0/16, that would explain what I'm seeing for that gateway.
I was worried that something had gone wrong somewhere because the overall inbound encap traffic dropped by 90% or more suddenly at 19:00-19:15 on Tuesday. Whatever happened did so quickly. I thought it was the db0fhn gateway that malfunctioned but I could be wrong. I'll have to look elsewhere.
You can see this sudden drop in the graph at 19:00 May 16 on https://gw.ampr.org/router/encapinpkt.svg - Brian
On Wed, May 17, 2017 at 09:54:47PM +0200, Thomas Osterried wrote:
What kind of traffic do you expect? The rest of 44/8 that is not ipip and not direct-bgp? Actually, db0fhn routes non-advertised 44-ipip-routes (and non-44-hamnet-networks) to the internet (because non-ipip and non-hamnet is either direct-bgp or not-in-use). Perhaps, Jann has changed the 44/8 route; we need to wait for his answer.
vy 73,
- Thomas dl9sau
Yes, outbound non-44 destinations from db0fhn are routed to the Internet via my other "non source route filtered" gateway. It will reduce the RTT to European destinations by almost 200ms, however traffic is non-symmetric.
As of today you shouldn't have seen any packets from db0fhn in your statistics due to the fact that ampr-ripd ignored the 44.0.0.1/32 route in the RIP broadcasts.
It seems the bidirectional communication from db0fhn to 44.0.0.1 didn't work due to non-symmetric traffic for a long time now (firewall on your end?). The route to 44.0.0.1 hasn't been announced to the HAMNET up to today.
Since you implemented some interesting services, I just set a manual route until Marius will change the ampr-ripd (maybe the upcoming weekend). I successfully tested access to gw.ampr.org from the HAMNET, however it would be nice if that would even work from hosts without valid DNS entries or do you think DNS entries should be a requirement even within the IPIP mesh?
73, Jann
Thomas, thank you for your reply. So outbound non-44 destinations from 44.130.0.0/16 are routed out onto the internet at db0fhn instead of being sent to amprgw? But the return path has to go through amprgw. This is non-symmetric routing and I wasn't expecting it. (It's not a problem.)
I have noted in the past that the link was often busy, and I expected it would be busy in both directions. If it's not used for outbound from 44.130.0.0/16, that would explain what I'm seeing for that gateway.
I was worried that something had gone wrong somewhere because the overall inbound encap traffic dropped by 90% or more suddenly at 19:00-19:15 on Tuesday. Whatever happened did so quickly. I thought it was the db0fhn gateway that malfunctioned but I could be wrong. I'll have to look elsewhere.
You can see this sudden drop in the graph at 19:00 May 16 on https://gw.ampr.org/router/encapinpkt.svg
- Brian
As a byproduct of the way that I'm routing the traffic for 44.0.0.1 (a special case) it won't work without a DNS entry. Maybe I can find a way around that, but it's not straightforward. No, I don't think DNS entries should be a requirement for intra-mesh communication.
It would be nice if everyone did register on the DNS, though. :-) - Brian
On Wed, May 17, 2017 at 11:46:00PM +0200, Jann Traschewski wrote:
Since you implemented some interesting services, I just set a manual route until Marius will change the ampr-ripd (maybe the upcoming weekend). I successfully tested access to gw.ampr.org from the HAMNET, however it would be nice if that would even work from hosts without valid DNS entries or do you think DNS entries should be a requirement even within the IPIP mesh?
Well, turns out the German gateway is ok, but that leaves the question of what happened to the incoming encap traffic unanswered. I'd very much like to know what caused the sharp decline in inbound encap packets between 19:00 and 19:15 UTC on Tuesday 05-16. What went offline?
Maybe some botnet blacklisted 44.0.0.0/8?
On 18.05.2017 01:46, Brian Kantor wrote:
(Please trim inclusions from previous messages) _______________________________________________ Well, turns out the German gateway is ok, but that leaves the question of what happened to the incoming encap traffic unanswered. I'd very much like to know what caused the sharp decline in inbound encap packets between 19:00 and 19:15 UTC on Tuesday 05-16. What went offline?
https://gw.ampr.org/router/encapinbytes.svg
- Brian
44Net mailing list 44Net@hamradio.ucsd.edu http://hamradio.ucsd.edu/mailman/listinfo/44net
To be more specific:
Most of the bogus traffic I had on the gw interface where stray fragments of TCP connections and UDP packets to random ports, alien DNS requests and bogus ICMP replies from hosts as if there where some requests originating on my ampr subnet (unused addresses) to external hosts.
At this moment, all I see are ICMP ping messages from some Argentinian and Chinese IPs and some ssh and port 19 (chargen) attempts to the gateway IP, no stray traffic.
On 18.05.2017 09:11, Marius Petrescu wrote:
(Please trim inclusions from previous messages) _______________________________________________ Maybe some botnet blacklisted 44.0.0.0/8?
Interesting. I've not analyzed the crud to that level of detail as there is simply too much of it for manual inspection and I haven't written any automation to do it.
However, the accepted/rejected ratio didn't change much around that moment, so I don't think we're suddenly exempt from inbound crud. There's no sudden large change in the graph.
It's interesting that you're not seeing as much as usual, but I think that amprgw is. - Brian
On Thu, May 18, 2017 at 09:17:58AM +0300, Marius Petrescu wrote:
To be more specific:
Most of the bogus traffic I had on the gw interface where stray fragments of TCP connections and UDP packets to random ports, alien DNS requests and bogus ICMP replies from hosts as if there where some requests originating on my ampr subnet (unused addresses) to external hosts.
At this moment, all I see are ICMP ping messages from some Argentinian and Chinese IPs and some ssh and port 19 (chargen) attempts to the gateway IP, no stray traffic.
The volume did not change (some 10-15 packets per minute), just the type.
I suspected at some point that there is a network using 44 addresses internally, had some leaks on them and that the garbage (DNS replies, ICM rejects, IP fragments and such stuff) were the replies from hosts on the internet receiving that traffic and sending replies back via the ampr-gw.
These are gone at the moment.
On 18.05.2017 09:41, Brian Kantor wrote:
(Please trim inclusions from previous messages) _______________________________________________ Interesting. I've not analyzed the crud to that level of detail as there is simply too much of it for manual inspection and I haven't written any automation to do it.
However, the accepted/rejected ratio didn't change much around that moment, so I don't think we're suddenly exempt from inbound crud. There's no sudden large change in the graph.
It's interesting that you're not seeing as much as usual, but I think that amprgw is.
- Brian
I'm currently blocking nearly 1000 high-volume inbound hosts (those that sent amprgw more than 10,000 packets a minute at the times of sampling), which may account for some of the change of traffic you're seeing get through the router.
None of the gateways are on that block list, so that doesn't account for the drop in inbound encap traffic.
I'm usually seeing around 20 million packets a minute (about 25 MB/s) at the inbound interface. Your 10-15 packets a minute is an awfully small amount to leak through. I must be doing a good job of filtering. - Brian
On Thu, May 18, 2017 at 09:46:55AM +0300, Marius Petrescu wrote:
The volume did not change (some 10-15 packets per minute), just the type.
I suspected at some point that there is a network using 44 addresses internally, had some leaks on them and that the garbage (DNS replies, ICM rejects, IP fragments and such stuff) were the replies from hosts on the internet receiving that traffic and sending replies back via the ampr-gw.
These are gone at the moment.
I think I'd be happy if that happened.
But I don't think so because the accepted/rejected ratio didn't suddenly change by much. It's just that there are a lot fewer valid encapped packets, which a botnet is not likely to be sending. We're still seeing as much junk as before. - Brian
On Thu, May 18, 2017 at 09:11:04AM +0300, Marius Petrescu wrote:
Maybe some botnet blacklisted 44.0.0.0/8?