Hello group,
I am wondering if anyone else is seeing the following: starting on 5
March 2020 and continuing through the present I have detected a large
spike in inbound traffic to several of my AMPR 44 IP addresses (on
44.50.1.0/24). The spike has been large enough that my logging ELK
stack is struggling to keep up.
This traffic is coming from the public internet. Most of these are
looking at standard ports 443, 80, 25, and 22.
These are being directed to IP addresses in my subnet that are not in
use, and therefore are being dropped (but logged) at the firewall.
Nothing is running on these IPs so there is no way the traffic is in
response to anything I can find coming from my network.
I realize devices periodically scan the "entire internet" but this is
more than that... in one day I saw 100,000 TCP SYN from a single public
IP address. That is a significant spike and I am not certain why they
sent so much traffic from a single IP to a single IP.
Wondering if anyone else is seeing the same?
73 DE KC0AKY
> So all traffic received on IPIP tunnels should be from net44 only in our case. Unfortunately not all of it is.
> Can you elaborate on the traffic that isn't, please?
> Is this traffic from another operator...or a non-operator?
> Can you also elaborate if this traffic forwards in any cases?
As I explained before, what I sometimes DO see is IPIP traffic from gateway A.B.C.D with an internal
packet with source A.B.C.D and destination 44.137.X.Y (inside our network). That traffic should have
been sent with source address 44.P.Q.R in the internal packet, where 44.P.Q.R is the net44 address
of that specific gateway at A.B.C.D.
As I got repeated logs in the firewall of these occurrences (one was from a Polish gateway, I remember)
I added a firewall rule to allow such traffic. The reply will of course be routed directly over
internet, not via the tunnel, so it is questionable if the connection would get established. Probably
not, when the user has the typical stateful firewall on his internet connection.
Yesterday I have removed the extra rule and I am watching the firewall log, but I have not yet observed
another instance of this error after about 12 hours. So maybe some people have woken up and fixed their
config already.
Rob
Rob,
You stated:
So all traffic received on IPIP tunnels should be from net44 only in our case. Unfortunately not all of it is.
Can you elaborate on the traffic that isn't, please?
Is this traffic from another operator...or a non-operator?
Can you also elaborate if this traffic forwards in any cases?
That's what we're tying to stop. Please note, I haven't identified this is related to any IPENCAP issue specifically (except that it appears we may have some operators that forward traffic not destined for them). While I understand your concern, I'm not sure it's related to IPENCAP 100%.
73,
- Lynwood
KB3VWG
> I'm now wondering how such a config is [incorrectly] made (i.e. the
inside Header has the incorrect SRC)....likely because of no route
policy...another discussion...
Easy: when you take a default Linux system and add something like IPIP
mesh with routes in the same table, and then you run services on the
same system, an outgoing connect to a system within net44 will just
consult the routing table, find an outgoing route and make a connection.
You then have to rely on the "source address selection" done by the
system, which may select your public IP as the source address.
This may also be configured in the service itself (when the socket is
not bound to 0.0.0.0 but to some specified address).
The outgoing connect will now be routed through the IPIP tunnel, but it
will have the public address as the source.
To prevent this, the service would have to be bound to the net44
address, or it would have to be set as a default source address in the
tunnel routes in the table.
When you run a separate system as the IPIP router and an AMPRnet
services host, you do not run into this problem because the services
host has the proper external address within net44 and the router will
not change it.
But with both combined in a single host, you can still get it working
correctly when you pay some attention. Which of course has to be done
when you want a single system that can both be a general-purpose
internet browsing system (directly via your ISP connection) and can be
an AMPRnet services host at the same time (also for services available
from public internet addresses). The routing has to be carefully set up
when doing this, and setting a preferred source address is only part of
that.
In our network the problem you mention w.r.t. AMPRGW does not occur
because internet traffic is routed directly to our gateway, not via an
IPIP tunnel.
The IPIP tunnel via AMPRGW only gets public internet traffic when our
BGP announcement is down for some reason, that is why I kept it
operational but it normally has zero traffic.
So all traffic received on IPIP tunnels should be from net44 only in our
case. Unfortunately not all of it is.
When I "just drop" the bad traffic it appears in a log and it appears
the originators of the traffic do not notice it, so it goes on and on.
As I mentioned, I sent mail to gateway owners about it, but it rarely
fixes the situation.
Rob
> One minor thing to bring up though, since I went and re-read the text. It says
> "ISPs don't configure their routers with publicly routable IP space for end users, why would you?"
> This is by and large false. Some do indeed use 1918 space for customer facing interfaces, but most do not, as this practice this can break PMTUD due to dropping PTB messages sent by 1918 numbered interfaces and is not generally not recommended.
It looks like this came from the HamWan people who apparently use a lot of RFC1918 addressing, I think at first even their whole network was on RFC1918 space and they introduced net44 capability only later.
However, in our network here we exclusively use net44 addresses, also for links and routers. Of course we do have appropriate firewalls in place.
The first line of defense is that at the internet router, all incoming traffic is blocked by default unless the destination is from an address list of only systems that provide services that are to be visible from internet.
And in the last router before the actual system there is an additional firewall that opens only certain ports for everyone, and restricts management ports to another address list that has only addresses of known operators, or at most the country subnet.
This makes the result similar to using RFC1918 addressing (which provides protection because it can only be routed locally), but without the disadvantages that you mention.
Anyway, I am not that much concerned about that part of the text, it will depend on local policies and we do not use that method of subnet allocation anyway.
My main concern at this time is that "outsiders" (not licensed radio amateurs) apparently find our network in their search for IPv4 address space, and make requests that we have to reject.
This rejection often leads to discussion (partly because there is no terminate-this-request option in the portal, a coordinator can only accept it or send it back to the requester for more detail).
Therefore I think it would be best when those outsiders immediately see that this system is not for them.
Rob
Rob,
I now understand what you meant by this. I'm not seeing [lots of] improper Header 1 traffic with a known gateway IP as a SRC IP (and if I do, I want to block as it's asymmetric, as I tax AMPRGW on the reply trip). I receive the packet from AMPRGW (i.e. they're using the Public Internet instead of AMPRNet to reach the NTP server).
* AMPR <> AMPR (OK, This allowed - as I list this service to my subnet in the Wiki)
* Registered_Public <> Registered_Public (OK; but closed for MOST services)
* Registered_Public <> AMPR (OK, not announced, taxes AMPRGW - may close soon; closed for some services)
* Tunnel (Header 0) <> Public IP (Header 1) (NO - I consider this a crafted packet or improperly configured gateway, creates asymmetry; and taxes AMPRGW on the reply) - but cannot be mitigate except by blocking registered gateways on the de-encapsulated side of tunl0
I would advise you block that traffic with an improper Header 1 from registered gateways of you flag it via Header 0.
But this discussion now causes me to recall discussions we've had before on:
1.) the security of seeing the Registered Public IP on both sides of the tunnel
2.) policy-based routing and the need to separate routing tables comes to mind....this comes from operators' failure to set that up.
I'm now wondering how such a config is [incorrectly] made (i.e. the inside Header has the incorrect SRC)....likely because of no route policy...another discussion...
Just note that bad IPENCAP is not the majority of bad traffic we're seeing on some nodes...in fact, bad IPENCAP is the least of this traffic (at least now).
- KB3VWG
Unfortunately this breaks legitimate traffic because some gateways are
incorrectly configured (as I mentioned before) and send tunneled traffic
with their own external address as source address, instead of the
AMPRnet address assigned to the gateway.
So such traffic is accepted as well here (i.e. traffic with a source
address of one of the gateways)
Are there any updated dns procedures with regards to coordinators? I have a new coordination and haven’t done it in a while and can no longer find the email with the procedure for adding dns. At one point I thought this was in the portal.
Best Regards
Elias
Kd5jfe
Louisiana AMPR coordinator
Sent from my iPhone
> The relevant page has been updated with your suggested dialogue
Thanks! It is at least an attempt, let's hope it works.
Other coordinators beware: there is a hosting company owner from the
Netherlands
that is apparently shopping for a subnet. After I rejected him, he
tried in Germany.
Rob
> Please keep in mind though, the malicious traffic I observed did not originally come from AMPRGW. I originally observed the nested IPENCAP traffic from a Polish Public IP that's still currently registered as an AMPRNet gateway.
As you know in later Linux kernels it became more difficult to see the
outer header of the IPIP packet in a firewall rule handling the tunneled
traffic.
To circumvent that, a couple of years ago I added a rule to the firewall
that sets a packet mark on traffic received from AMPRGW (matching the
source IP in the outer header).
This packet mark can then be checked in the firewall for the tunneled
traffic. Source addresses outside AMPRnet are only accepted when the
packet mark is set.
Unfortunately this breaks legitimate traffic because some gateways are
incorrectly configured (as I mentioned before) and send tunneled traffic
with their own external address as source address, instead of the
AMPRnet address assigned to the gateway.
So such traffic is accepted as well here (i.e. traffic with a source
address of one of the gateways)
We really should abandon the IPIP tunnel mesh and move on to something a
bit more secure (and easier to use on modern equipment)...
Rob