17 May
2017
17 May
'17
12:47 p.m.
This is rather disturbing. Many of the gateways in
the gateway
statistics are showing ZERO encap packets coming from them. One of
the busiest normally is the gateway for Germany, 141.75.245.225, which
has not sent ANY encap packets to amprgw since the current stats began
Is that still the case? I receive plenty of traffic from that gateway,
so when you don't there could be someone who inserted a protocol 4 filter
along the way.
I can still ping 44.0.0.1 via the tunnel...
traceroute to 169.228.66.251 (169.228.66.251), 30 hops max, 60 byte packets
1 xs4all-router (213.222.29.193) 0.262 ms 0.304 ms 0.276 ms
2 0.ae10.xr4.1d12.xs4all.net (194.109.5.57) 0.322 ms 0.ae11.xr4.1d12.xs4all.net
(194.109.5.77) 0.341 ms 0.276 ms
3 asd2-rou-1043.NL.eurorings.net (134.222.93.144) 0.320 ms 0.355 ms
asd2-rou-1044.NL.eurorings.net (134.222.97.17) 0.370 ms
4 asd2-rou-1022.NL.eurorings.net (134.222.48.222) 0.615 ms 0.544 ms 0.500 ms
5 chg-s1-rou-1041.US.eurorings.net (134.222.48.90) 95.532 ms 95.539 ms 95.543 ms
6 eq-exchange.tr01-chcgil01.transitrail.NET (206.223.119.116) 95.616 ms 95.675 ms
95.644 ms
7 ae-1.80.rtsw.chic.net.internet2.edu (64.57.20.150) 95.879 ms 95.901 ms 95.890 ms
8 et-3-1-0.4072.rtsw.kans.net.internet2.edu (64.57.20.89) 106.891 ms 106.906 ms
106.909 ms
9 ae-0.80.rtsw.salt.net.internet2.edu (64.57.20.146) 126.801 ms 126.922 ms 126.846
ms
10 ae-2.80.rtsw.losa.net.internet2.edu (64.57.20.144) 139.339 ms 139.437 ms 139.436
ms
11 64.57.20.83 (64.57.20.83) 139.681 ms 139.675 ms 140.043 ms
12 dc-tus-agg3--lax-agg6-100ge.cenic.net (137.164.11.7) 140.662 ms 140.780 ms 140.919
ms
13 dc-sdg-agg4--tus-agg3-100ge.cenic.net (137.164.11.9) 142.328 ms 142.342 ms 142.395
ms
14 dc-ucsd-1--sdg-agg4.cenic.net (137.164.23.54) 141.823 ms 141.889 ms 141.998 ms
15 nodem-core--mx0-30ge.ucsd.edu (132.239.254.163) 142.100 ms 142.099 ms 142.086 ms
16 ebu3b-6509-nodem-core-interconnect-vl910-bcast-255-131.ucsd.edu (132.239.255.131)
142.068 ms 142.094 ms 142.162 ms
Rob
17 May
17 May
1:11 p.m.
New subject: AMPRNet outage?
I've asked our campus networking people if they've installed any
filters but there's been no reply yet.
However, I doubt they'd block a specific host for a specific
protocol; if they did any blocking it would be a complete block
of either the host or the protocol, and I can ping the host
and other sites are successfully connected using IPIP, so neither
would seem to be the case. Perhaps someone in the path between
us and Germany inserted a protocol 4 block, or perhaps the German
router crashed and lost its routing tables.
No reply from Germany yet either.
- Brian
On Wed, May 17, 2017 at 06:47:13PM +0200, Rob Janssen wrote:
Is that still the case? I receive plenty of traffic
from that gateway,
so when you don't there could be someone who inserted a protocol 4 filter
along the way.
3:54 p.m.
New subject: AMPRNet outage?
Hello,
On Wed, May 17, 2017 at 10:11:26AM -0700, Brian Kantor wrote:
[..]
would seem to be the case. Perhaps someone in the
path between
us and Germany inserted a protocol 4 block, or perhaps the German
router crashed and lost its routing tables.
No reply from Germany yet either.
well, I've read it, but I was not able to send an adaequate answer:
we learn the ipip routes at db0fhn. I checked that I could ping a host via that ipip
tunnel.
I also testet that host from a subnet behind db0fhn.
And I tested to ping db0fhn.ampr.org from the internet; success.
You said you cannot see inbound traffic from db0fhn. Well, in the first case (ipip mesh to
another 44'er host) goes "direct", thus not via ucsd.edu.
What kind of traffic do you expect? The rest of 44/8 that is not ipip and not direct-bgp?
Actually, db0fhn routes non-advertised 44-ipip-routes (and non-44-hamnet-networks) to the
internet (because non-ipip and non-hamnet is either direct-bgp or not-in-use).
Perhaps, Jann has changed the 44/8 route; we need to wait for his answer.
vy 73,
- Thomas dl9sau
4:19 p.m.
New subject: AMPRNet outage?
All,
Can the person NJohnson who contributed major edits to:
http://wiki.ampr.org/wiki/Ubuntu_Linux_Gateway_Example
please contact me off-thread.
Thanks and 73,
- Lynwood
KB3VWG
4:32 p.m.
New subject: AMPRNet outage?
Thomas, thank you for your reply. So outbound non-44 destinations from
44.130.0.0/16 are routed out onto the internet at db0fhn instead of
being sent to amprgw? But the return path has to go through amprgw.
This is non-symmetric routing and I wasn't expecting it. (It's not
a problem.)
I have noted in the past that the link was often busy, and I expected
it would be busy in both directions. If it's not used for outbound
from 44.130.0.0/16, that would explain what I'm seeing for that
gateway.
I was worried that something had gone wrong somewhere because the overall
inbound encap traffic dropped by 90% or more suddenly at 19:00-19:15 on
Tuesday. Whatever happened did so quickly. I thought it was the db0fhn
gateway that malfunctioned but I could be wrong. I'll have to look
elsewhere.
You can see this sudden drop in the graph at 19:00 May 16 on
https://gw.ampr.org/router/encapinpkt.svg
- Brian
On Wed, May 17, 2017 at 09:54:47PM +0200, Thomas Osterried wrote:
What kind of traffic do you expect? The rest of 44/8
that is not ipip and not direct-bgp? Actually, db0fhn routes non-advertised 44-ipip-routes
(and non-44-hamnet-networks) to the internet (because non-ipip and non-hamnet is either
direct-bgp or not-in-use).
Perhaps, Jann has changed the 44/8 route; we need to wait for his answer.
vy 73,
- Thomas dl9sau
5:46 p.m.
New subject: AMPRNet outage?
Yes, outbound non-44 destinations from db0fhn are routed to the Internet
via my other "non source route filtered" gateway. It will reduce the RTT
to European destinations by almost 200ms, however traffic is non-symmetric.
As of today you shouldn't have seen any packets from db0fhn in your
statistics due to the fact that ampr-ripd ignored the 44.0.0.1/32 route
in the RIP broadcasts.
It seems the bidirectional communication from db0fhn to 44.0.0.1 didn't
work due to non-symmetric traffic for a long time now (firewall on your
end?). The route to 44.0.0.1 hasn't been announced to the HAMNET up to
today.
Since you implemented some interesting services, I just set a manual
route until Marius will change the ampr-ripd (maybe the upcoming
weekend). I successfully tested access to gw.ampr.org from the HAMNET,
however it would be nice if that would even work from hosts without
valid DNS entries or do you think DNS entries should be a requirement
even within the IPIP mesh?
73,
Jann
Thomas, thank you for your reply. So outbound non-44
destinations from
44.130.0.0/16 are routed out onto the internet at db0fhn instead of
being sent to amprgw? But the return path has to go through amprgw.
This is non-symmetric routing and I wasn't expecting it. (It's not
a problem.)
I have noted in the past that the link was often busy, and I expected
it would be busy in both directions. If it's not used for outbound
from 44.130.0.0/16, that would explain what I'm seeing for that
gateway.
I was worried that something had gone wrong somewhere because the overall
inbound encap traffic dropped by 90% or more suddenly at 19:00-19:15 on
Tuesday. Whatever happened did so quickly. I thought it was the db0fhn
gateway that malfunctioned but I could be wrong. I'll have to look
elsewhere.
You can see this sudden drop in the graph at 19:00 May 16 on
https://gw.ampr.org/router/encapinpkt.svg
- Brian
6:07 p.m.
New subject: AMPRNet outage?
As a byproduct of the way that I'm routing the traffic for 44.0.0.1
(a special case) it won't work without a DNS entry. Maybe I can
find a way around that, but it's not straightforward. No, I don't
think DNS entries should be a requirement for intra-mesh communication.
It would be nice if everyone did register on the DNS, though. :-)
- Brian
On Wed, May 17, 2017 at 11:46:00PM +0200, Jann Traschewski wrote:
Since you implemented some interesting services, I
just set a manual
route until Marius will change the ampr-ripd (maybe the upcoming
weekend). I successfully tested access to gw.ampr.org from the HAMNET,
however it would be nice if that would even work from hosts without
valid DNS entries or do you think DNS entries should be a requirement
even within the IPIP mesh?
6:46 p.m.
New subject: AMPRNet outage?
Well, turns out the German gateway is ok, but that leaves the question
of what happened to the incoming encap traffic unanswered. I'd very
much like to know what caused the sharp decline in inbound encap packets
between 19:00 and 19:15 UTC on Tuesday 05-16. What went offline?
https://gw.ampr.org/router/encapinbytes.svg
- Brian
18 May
18 May
2:11 a.m.
New subject: AMPRNet outage?
Maybe some botnet blacklisted 44.0.0.0/8?
On 18.05.2017 01:46, Brian Kantor wrote:
(Please trim inclusions from previous messages)
_______________________________________________
Well, turns out the German gateway is ok, but that leaves the question
of what happened to the incoming encap traffic unanswered. I'd very
much like to know what caused the sharp decline in inbound encap packets
between 19:00 and 19:15 UTC on Tuesday 05-16. What went offline?
https://gw.ampr.org/router/encapinbytes.svg
- Brian
_________________________________________
44Net mailing list
44Net(a)hamradio.ucsd.edu
http://hamradio.ucsd.edu/mailman/listinfo/44net
2:17 a.m.
New subject: AMPRNet outage?
To be more specific:
Most of the bogus traffic I had on the gw interface where stray
fragments of TCP connections and UDP packets to random ports, alien DNS
requests and bogus ICMP replies from hosts as if there where some
requests originating on my ampr subnet (unused addresses) to external hosts.
At this moment, all I see are ICMP ping messages from some Argentinian
and Chinese IPs and some ssh and port 19 (chargen) attempts to the
gateway IP, no stray traffic.
On 18.05.2017 09:11, Marius Petrescu wrote:
(Please trim inclusions from previous messages)
_______________________________________________
Maybe some botnet blacklisted 44.0.0.0/8?
2:41 a.m.
New subject: AMPRNet outage?
Interesting. I've not analyzed the crud to that level of detail as there
is simply too much of it for manual inspection and I haven't written
any automation to do it.
However, the accepted/rejected ratio didn't change much around that
moment, so I don't think we're suddenly exempt from inbound crud.
There's no sudden large change in the graph.
It's interesting that you're not seeing as much as usual, but I think
that amprgw is.
- Brian
On Thu, May 18, 2017 at 09:17:58AM +0300, Marius Petrescu wrote:
To be more specific:
Most of the bogus traffic I had on the gw interface where stray fragments of
TCP connections and UDP packets to random ports, alien DNS requests and
bogus ICMP replies from hosts as if there where some requests originating on
my ampr subnet (unused addresses) to external hosts.
At this moment, all I see are ICMP ping messages from some Argentinian and
Chinese IPs and some ssh and port 19 (chargen) attempts to the gateway IP,
no stray traffic.
2:46 a.m.
New subject: AMPRNet outage?
The volume did not change (some 10-15 packets per minute), just the type.
I suspected at some point that there is a network using 44 addresses
internally, had some leaks on them and that the garbage (DNS replies,
ICM rejects, IP fragments and such stuff) were the replies from hosts on
the internet receiving that traffic and sending replies back via the
ampr-gw.
These are gone at the moment.
On 18.05.2017 09:41, Brian Kantor wrote:
(Please trim inclusions from previous messages)
_______________________________________________
Interesting. I've not analyzed the crud to that level of detail as there
is simply too much of it for manual inspection and I haven't written
any automation to do it.
However, the accepted/rejected ratio didn't change much around that
moment, so I don't think we're suddenly exempt from inbound crud.
There's no sudden large change in the graph.
It's interesting that you're not seeing as much as usual, but I think
that amprgw is.
- Brian
3:03 a.m.
New subject: AMPRNet outage?
I'm currently blocking nearly 1000 high-volume inbound hosts (those that
sent amprgw more than 10,000 packets a minute at the times of sampling),
which may account for some of the change of traffic you're seeing get
through the router.
None of the gateways are on that block list, so that doesn't account
for the drop in inbound encap traffic.
I'm usually seeing around 20 million packets a minute (about 25 MB/s)
at the inbound interface. Your 10-15 packets a minute is an awfully
small amount to leak through. I must be doing a good job of filtering.
- Brian
On Thu, May 18, 2017 at 09:46:55AM +0300, Marius Petrescu wrote:
The volume did not change (some 10-15 packets per
minute), just the type.
I suspected at some point that there is a network using 44 addresses
internally, had some leaks on them and that the garbage (DNS replies, ICM
rejects, IP fragments and such stuff) were the replies from hosts on the
internet receiving that traffic and sending replies back via the ampr-gw.
These are gone at the moment.
2:18 a.m.
New subject: AMPRNet outage?
I think I'd be happy if that happened.
But I don't think so because the accepted/rejected ratio didn't suddenly
change by much. It's just that there are a lot fewer valid encapped packets,
which a botnet is not likely to be sending. We're still seeing as much junk
as before.
- Brian
On Thu, May 18, 2017 at 09:11:04AM +0300, Marius Petrescu wrote:
Maybe some botnet blacklisted 44.0.0.0/8?
2745
days inactive
2746
days old
13 comments
6 participants
participants (6)
-
Brian Kantor -
Jann Traschewski -
lleachii＠aol.com -
Marius Petrescu -
Rob Janssen -
Thomas Osterried