> The strange thing is that ping works ok when TCP doesn't connect.
> My first suspicion would be a stateful firewall, but I'm sure you
> checked that. Could it be a TTL problem? I'm just guessing here.
The TTL of the inside IP packet is 63. I first traced on the external interface
and saw the encapsulated packet, then I traced on the tunl0 interface and saw
the decapsulated packet (same without the outer IP header), and it all looks OK.
I see the SYN going out,the SYN ACK coming in, but nothing more (ACK should go out).
The firewall is stateful but I added an explicit accept for -s 44.0.0.1 at the
top of all the rules to make sure it is not that. Also, I reset the firewall and
watched the counters, did not see any packets being dropped.
And indeed, ping works OK. It is strange. Maybe something is wrong due to the
gateway external address change, although I would not know what could produce
the above scenario. The system is up for about a year, maybe I should try
rebooting it. (usually this brings nothing when I try it... it isn't Windows :-)
Of course sometime it will become clear how this can be explained.
Rob
> I'm curious as to how long you waited when trying port 25. There
> is a five-second delay after the connection is established before
> it issues the 220 greeting
I noticed that. But on 44.137.40.2 the TCP connection does not even
establish.
I tried from home (44.137.41.97) and from the gw-44-137 (44.137.0.1)
and both were able to connect:
telnet 44.0.0.1 25
Trying 44.0.0.1...
Connected to 44.0.0.1.
Escape character is '^]'.
220 gw.ampr.org ESMTP Sendmail 8.15.2/8.15.2; Thu, 13 Jul 2017 09:20:58 -0700 (PDT)
quit
221 2.0.0 gw.ampr.org closing connection
Connection closed by foreign host.
However, on 44.137.40.2:
telnet 44.0.0.1 25
Trying 44.0.0.1...
(and nothing more. probably an error when I wait a minute or so)
When running tshark on tunl0 I see the SYN+ACK packets from 44.0.0.1 to 44.137.40.2
and when I do an insert of a rule matching source address 44.0.0.1 before all
other firewall rules I see it matching, but still no ACK is going back.
I have no idea what is going on there. Ping works OK.
Rob
> Remember that the system routing table and the encap routing table are
> separate on amprgw. The system routing table is in the kernel; the
> encap table is in user space. Packets arriving at amprgw which are for
> hosts listed in the encap hosts table are diverted to the encap router;
> all other packets are left to the kernel to route.
Ok that should work fine.
> On the other hand, packets leaving (originating at) amprgw are first
> looked up in the system routing table and if a route is found (as it
> would be for your bgp-advertised subnet), they are sent out the system
> Ethernet interface and not diverted into the encap router. Packets with a
> destination not in the system routing table are tested to see if they're
> in the encap list, and if they are, they're sent to the encap router
> to be forwarded. The rest are dropped. The encap list currently has
> 19000 host entries in it.
Why do you have entries for the BGP-advertised subnets? Would those not
be adequately handled by the default route? Or is this only to handle
IPIP tunnels with a BGP-advertised net-44 tunnel endpoint address? Maybe
route entries for only those addresses would be an option.
(after all, they are known as they are listed in encap.txt)
The algorithm could then be to lookup 44-net addresses in the IPIP table
when there is no specific route for them in the routing table. When no
match is found, they could still be sent out as BGP-routed when desired,
this could be achieved with a "special" entry in the lookup array for
BGP-routed addresses that are not mentioned in the encap list.
> I have experimented with forcing 44.137.40.2 into the encap router with
> an explicit route to the loopback interface. This seems to work; the
> pings go out a tunnel to 89.18.172.156. I've left this route in place
> for now so you can repeat your tests and see if you do indeed get replies
> over the tunnel the way you expect. Please let me know.
Interesting: I can now ping 44.0.0.1 from that system, but a "telnet 44.0.0.1 25"
still yields nothing. I have not yet been able to find why that is, I see
the SYN ACK coming back on the tunnel and I have allowed all traffic from
44.0.0.1 in the firewall but it simply fails to establish. No ACK is going
back to you. I can connect to other IPIP endpoints just fine. Strange.
It is not really required to fix it, I can easily route the mail via a
different path, but I was interested in finding why this goes wrong and how
it may affect others / other applications.
> It could be argued that I should scrap FreeBSD on amprgw and start over
> with a Linux host. I don't know enough about the intricacies of Linux
> IP handling to be at all confident of being able to do that.
I can understand that! It is certainly possible to get it running, and it
would be a lot easier to get IPIP running, but it is always a big step
to do such things.
Rob
> Yes, BGP overrides encap on amprgw.
Is there a rationale behind that or is it just because of some implementation detail?
In our gateway, all routes are in the same table but at different distance (metric).
Direct/BGP routes have higher precedence than IPIP routes, when for the same subnet.
But, a more specific route (smaller subnet) always has precedence over a less specific one.
Rob
> Rob, it looks like the problem might be nearer to you...
> Interestingly, when I traceroute to 44.137.40.2 with 169.228.34.84 as
> the source address, I get:
> ...
That is normal, it will be routed over internet to our BGP subnet 44.137.0.0/16
which is via that gw-44-137 which then forwards it to 44.137.40.2 over IPIP.
However, 44.137.40.2 does not reply to pings from internet.
> When I repeat that traceroute with 44.0.0.1 as the source address,
> it gets nearly as far, but not quite:
> ...
That is strange. Don't you have a route to 44.137.40.2 in your encap table?
I would expect you to send it into an IPIP tunnel direct to that system.
Or is there some magic in amprgw that makes it ignore IPIP tunnels within BGP
routed subnets? That would cause this to fail...
> I can ping 213.222.29.194 from 169.228.34.84, but not from
> 44.0.0.1.
But you can ping 44.137.0.1 (the amprnet address of that same system) from 44.0.0.1 ?
> It looks, at first glance, like there's a 44 path problem in
> 213.222.29.194.
When you route back to 44.137.40.2 via 213.222.29.194 it is explained, because that
system has a stateful firewall that does not expect replies to traffic that has not
gone out via that same system.
Rob
> I thought those problems were fixed as well, or I would not have changed
> the submission address. I am not aware of any changes that would have
> affected the connectivity of 44.0.0.1. It should be working.
> More details might be helpful, such as the actual IP addresses involved,
> or perhaps traceroutes showing where the routing stops working.
Yes, I think it worked but likely that was on the previous gateway.
The system is 44.137.40.2 (89.18.172.156)
The route is:
44.0.0.1 via 169.228.34.84 dev tunl0 proto ampr-ripd onlink
A traceroute immediately fails. (only stars)
tshark dump of a "ping" (looks OK):
Internet Protocol Version 4, Src: 89.18.172.156 (89.18.172.156), Dst: 169.228.34.84 (169.228.34.84)
Version: 4
Header Length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
Total Length: 104
Identification: 0x949e (38046)
Flags: 0x02 (Don't Fragment)
0... .... = Reserved bit: Not set
.1.. .... = Don't fragment: Set
..0. .... = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: IPIP (4)
Header checksum: 0xd40c [validation disabled]
[Good: False]
[Bad: False]
Source: 89.18.172.156 (89.18.172.156)
Destination: 169.228.34.84 (169.228.34.84)
[Source GeoIP: Unknown]
[Destination GeoIP: Unknown]
Internet Protocol Version 4, Src: 44.137.40.2 (44.137.40.2), Dst: 44.0.0.1 (44.0.0.1)
Version: 4
Header Length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
Total Length: 84
Identification: 0x8f7f (36735)
Flags: 0x02 (Don't Fragment)
0... .... = Reserved bit: Not set
.1.. .... = Don't fragment: Set
..0. .... = More fragments: Not set
Fragment offset: 0
Time to live: 64
Protocol: ICMP (1)
Header checksum: 0x2a9e [validation disabled]
[Good: False]
[Bad: False]
Source: 44.137.40.2 (44.137.40.2)
Destination: 44.0.0.1 (44.0.0.1)
[Source GeoIP: Unknown]
[Destination GeoIP: Unknown]
Internet Control Message Protocol
Type: 8 (Echo (ping) request)
Code: 0
Checksum: 0xbba6 [correct]
Identifier (BE): 23938 (0x5d82)
Identifier (LE): 33373 (0x825d)
Sequence number (BE): 1 (0x0001)
Sequence number (LE): 256 (0x0100)
Timestamp from icmp data: Jul 12, 2017 20:30:58.200360000 CEST
[Timestamp from icmp data (relative): 0.000371000 seconds]
Data (48 bytes)
0000 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 ................
0010 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 ........ !"#$%&'
0020 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37 ()*+,-./01234567
Data: 08090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f...
[Length: 48]
This system has our own gateway 213.222.29.194 as its default tunnel route
but otherwise it is a standard ampr-ripd setup with 2 route tables.
Rob
Recently I saw a header in the DNS robot reply mails that the mail address has changes and the
update mails now have to be sent @gw.ampr.org. I changed that in my script, but then I noticed
that my system on the IPIP tunnel network cannot send mail there. Of course gw.ampr.org is 44.0.0.1
and there have been issues with reaching that from the IPIP gateway systems, but I thought those were
fixed.
I have changed my mail config to send the mail via my ISP smarthost and that works, but what is the
status on this? Is it supposed to be working, does it work for others?
It only affects my gateway on the IPIP tunnel network. From the BGP routed gateway and the radio-connected
systems here in the Netherlands there is no problem, and neither from normal internet addresses.
Rob
In general, all encapsulated packets sent to the amprgw router for
decapsulation and forwarding should have inner source addresses on the
44.0.0.0/8 network, since they are being originated by hosts on that
network.
The owners of the following gateways should contact me or consult
the error listing in the /private/pkterrors.txt file regarding
misconfiguration of their gateway routing rules which is leading to the
sending of packets to amprgw which have a non-44 inner source address.
These packets are being discarded by the gateway router, so they're not
being delivered anywhere. This could be caused by an erroneous default
route pointing at amprgw. Probably you shouldn't have one of those.
This isn't causing any harm, but it's not doing any good either.
- Brian
VE3CGG 24.55.194.111
KE5MKM 98.6.215.34
EB1AJP 188.82.69.230
VE2HOM 206.80.251.222
> For several hours now, amprgw has been seeing a storm of traceroutes from
> hundreds of different source addresses. It looks like a botnet has been
> activated to probe net 44 using short-TTL packets like traceroute.
> In reaction to this, I've temporarily set the gateway to discard any
> packet with a TTL of less than 30. (The TTL is decremented by one when
> the packet is forwarded; normally, of course, only packets with a TTL
> value of zero are discarded.)
Interesting... is it real traceroute traffic (to UDP port 33434 and higher)
or is it different?
I have had this rule (with TTL limit 16 and only for UDP 33434-33499) on our
gateway for quite some time and I do not see many hits on it.
Maybe the traffic is different. I do not observe increased input traffic.
Rob
> However, there's quite a bit more insidious kind of traffic. The Nagra
> people (Kudelski Switzerland) are probing our network with false
> NTP packets from the subnet 185.35.62.0/23. The comment in the RIPE
> database is
> inetnum: 185.35.62.0 - 185.35.63.255
> descr: This IP network is used for Internet security research. Internet-scale port scanning activities are launched from this network. Don't hesitate to contactportscan at nagra.com <http://hamradio.ucsd.edu/mailman/listinfo/44net> would you have any question.
> I've added that subnet to the "security research" blocking list here.
> Seems it's a never-ending battle.
It sure is. 185.35.60.0/22 was already in the blocklist here, I have seen
"research" traffic from the bottom half of that network as well.
Rob
