> Yes, it's subsided quite a bit. The amprgw machine is only spending less than
> 15% of its processor time filtering packets, vs over 25% earlier and on
> the weekend. Perhaps posting my filter script/program was another fine
> example of closing the barn door after the horse has bolted.
Well it may come back anytime of course...
The strange thing is that I see no peak at all in the traffic graphs made
over the past days and weeks, and there have been much higher peaks in the
past. But maybe you just were not looking at that time...
(a couple of weeks we also experienced a DDoS attack that had several
orders of magnitude more traffic)
I have done some tracing in the past to identify the most obvious problems
and I can understand that you become more and more worried when studying
the problem. As you well know, it consists of both attempts to hack the
systems and of backscatter from attempts to attack others using spoofed
source addresses.
> Just now, it took 287 seconds to gather 100 million packets, comprising
> 7100 different source addresses. This is rather more than usual. The
> blocking table now contains 18,000 entries.
I have a static blocking table that has the addresses of shodan.io and
other miscreants of this world, and the "research institutes" that consider
it research to scan other people's networks to map out vulnerabilities
etc. That includes 169.228.66.91 and 169.228.66.138 but there are lots
of others so no need to get worried.
I do not bother to block the scattered Chinese addresses that do only telnet
scanning, for that purpose I have put a rate limiter in the firewall that
limits the number of unanswered SYN requests per source address using
the "recent" matching module of iptables.
Rob
> The second one
> runs a wall display in the department, it shouldn't be probing anything.
> It's a reassigned machine, so perhaps a previous user of that machine
> was doing some research scanning too. Sorry about that.
At the time I put it on the list it had the reverse DNS ipsecscanner.sysnet.ucsd.edu
and that is what it was doing. I removed it now. A problem of my method of blocking
is that I cannot keep stats of activity of the address, so addresses may remain on the
list far too long.
But again, there are enough other entries for "research" from several other US and
German universities. It appears to be a popular way of annoying people.
Of course the interesting thing about the UCSD ones was that the traffic came through
the tunnel even though our network is BGP routed. Not sure if it is still like that, I believe
you changed something there.
Several scanners offer opt-out but I think the research is mainly to see what people do
when removal is promised but not done or only very temporarily. For example, I
contacted the shodan.io guy twice for removal from his scanner, both times he removed
our network only to re-add it within days. He probably expects that you will check after
he answered the mail, see it is gone, and then not pay attention to it so he can continue
his abuse.
Rob
> The DDoS attack on net 44 continues. I'm filtering out a goodly amount
> of it at amprgw, but the people whose subnets are directly connected (BGP
> announced) are getting hit too, and there's nothing I can do to filter it
> out here.
Our traffic is not particularly high here, of course there is a few Mbit/s of noise
but it has been higher at times.
Rob
The DDoS attack on net 44 continues. I'm filtering out a goodly amount
of it at amprgw, but the people whose subnets are directly connected (BGP
announced) are getting hit too, and there's nothing I can do to filter it
out here. Basically, if you're directly connected (ie, not on a tunnel),
you have to add a list of bad guys to your own firewall blocking.
Here is a little script that will create a list of candidate addresses
for blocking. You'll need the 'badguys' program, which I'll post below.
What this does is give you a list of addrsses that sent you more than
1000 packets in the sample period.
If you're connected via tunnel, you probably don't need this.
I run this several times a day to accumulate a list of bad guys.
I sample 100 million packets at a time; smaller subnets will probably
want to use a smaller sample size.
This was developed on FreeBSD but I'm told it works fine on Linux
Hope this helps.
- Brian
-----------------------------------------------------------
block.sh:
#!/bin/sh
#
# generate a list of possible bad guys
#
# sample incoming traffic, 100,000,000 incoming packets
# during DDoS storm on a /8, this takes about 3 minutes
time tcpdump -w /tmp/t.pcap -s 40 -c 100000000
# turn into a list of source IP addresses with counts,
# sorted by decreasing count
# throw away all those counted less than 1000,
# delete the count from the line
# elide our own addresses (put your own in the egrep)
# store in a file
./badguys /tmp/t.pcap \
| sort \
| uniq -c \
| sort -rn \
| sed -e '/^ /d' \
| sed -e 's/.* //' \
| egrep -v '^44\.|^169\.228\.' \
> /tmp/badguys
# all done with this file
rm -f /tmp/t.pcap
exit 0
-----------------------------------------------------------
badguys.c:
/*
* reads pcap capture file (use tcpdump -w to create) named
*
* spins through it, listing the IP source address on stdout
* prints summary statistics on stderr
*
* the documentation on the pcap library leaves a lot to be
* desired, use the source. a lot of what you need is in tcpdump.c
*
* compile me with
* cc -g -Wall badguys.c -lpcap -o badguys
*
* (You'll need to have libpcap installed)
*
* Brian Kantor, UCSD, 2017
*/
/* probably most of these includes are superfluous */
#include <sys/errno.h>
#include <sys/types.h>
#include <ctype.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <net/ethernet.h>
#include <netinet/in.h>
#include <netinet/in_systm.h>
#include <netinet/ip.h>
#include <arpa/inet.h>
#include <pcap/pcap.h>
#include <pcap/bpf.h>
void getpkt(u_char *, const struct pcap_pkthdr *, const u_char *);
char * iptoa(u_long);
int capcount = 0;
int skipcount = 0;
int
main(int argc, char**argv)
{
char inpcapname[FILENAME_MAX+1];
char errbuf[BUFSIZ];
pcap_t *pcap;
int rslt;
if (argc != 2) {
fprintf(stderr, "Usage: %s pcapfilename\n", argv[0]);
exit(1);
}
strncpy(inpcapname, argv[1], FILENAME_MAX);
fprintf(stderr, "opening PCAP file %s\n", inpcapname);
pcap = pcap_open_offline(inpcapname, errbuf);
if (!pcap) {
fprintf(stderr, "Error: %s\n", errbuf);
exit(1);
}
/*
* read through the capture savefile
*/
rslt = pcap_loop(pcap, 0, getpkt, 0);
if (rslt) {
fprintf(stderr, "pcap_loop returned %d\n", rslt);
pcap_perror(pcap, "pcap_loop returned");
exit(1);
}
pcap_close(pcap);
fprintf(stderr, "%d packets read from savefile\n", capcount);
fprintf(stderr, "%d non-IP packets discarded\n", skipcount);
exit(0);
}
/*
* called by pcap_loop once per packet read from savefile
*/
void
getpkt(u_char *user, const struct pcap_pkthdr *h, const u_char *p)
{
struct ip * ip;
u_short * ethertype;
capcount++;
/* get the Ethernet packet type */
ethertype = (u_short *)&p[(2*ETHER_ADDR_LEN)];
/* we only want IP packets */
if (ntohs(*ethertype) != ETHERTYPE_IP) {
skipcount++;
return;
}
/* IP packet starts after Ethernet header */
ip = (struct ip *)&p[ETHER_HDR_LEN];
puts(iptoa(ntohl(ip->ip_src.s_addr)));
return;
}
/*
* nicely format host-byte-order u_int into dotted quad
* returns pointer to a small static buffer; you can't
* call this more than once in a single syslog or printf statement
*/
char *
iptoa(u_long ipaddr)
{
static char buf[32];
snprintf(buf, sizeof buf, "%lu.%lu.%lu.%lu",
(ipaddr >> 24) & 0xff,
(ipaddr >> 16) & 0xff,
(ipaddr >> 8) & 0xff,
(ipaddr) & 0xff);
return buf;
}
> Perhaps then I didn't glean that data...but HOW does one 'decode' that
> MAC then?
> Meaning, how do I see SRC, DST protocol number, etc.?
> (this might be helpful looking through kernel-mod ipt)
That MAC is just a hexdump of the IP header so you extract the bytes
you want and convert them to decimal.
It should have been only 20 bytes but it is much longer, probably someone
misunderstood the length field somewhere.
Rob
> What [additional] information does this MAC field provide to you on the
> tunnel?
> Does this field change per packet?
> Is there some documentation on how to decode it?
> Is it a hashing of some sort, or just a hex copy of the data (IP header)?
All these questions are answered in my initial post.
It was useful, as it allowed to see what tunnel endpoint had sent a packet
when it was dropped by the firewall on the tunl0 device itself. Of course
the tunnel source can be seen on the eth0 device but there you cannot (or it
is very difficult to) examine the encapsulated packet.
Rob
Today I noticed that the log on our gateway lost some detail for IPIP traffic.
(now that everyone is discussing trouble with tunneled packets...)
In the past with Linux kernel 3.2 and 3.16, when some packet incoming over tunl0 was hitting a LOG target
in the iptables firewall, it would log a MAC address of the entire IP header as hex bytes (colon separated),
like this:
Apr 16 07:39:13 gw-44-137 kernel: [1266597.275238] Packet DROP: IN=tunl0 OUT=eth
1 MAC=45:00:00:44:10:f9:00:00:f9:04:e9:f9:54:6a:7e:b8:d5:de:1d:c2:45:00:00:30:64
:21:40:00:7e:06:29:1a:c0:a8:58:0a:2c:89:2a:51:e9:c0:14:66:1f:fa:64:c8:00:00:00:0
0:70:02:20:00:70:a9:00:00:02:04:05:b4:01:01:04:02:c8:78:03:6b:1a:74:bd:29:37:8d:
da:27:61:d7:2f:22:b0:b5:2b:b8:b4:61:3a:60:08:23:48:1b:26:15:57:80:00:00:85:ec:03
:32:df:df:85:46:bb:b3:40:e4:0f:df:4b:3d:93:e0:ed:f3:46:d4:e0:17:68:b6:dd:5d:f1:3
f:1b:1e:6f:a0:f0:69:5c:28:4a:3c:24:17:20:ff:e5:97 SRC=192.168.88.10 DST=44.137.4
2.81 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=25633 DF PROTO=TCP SPT=59840 DPT=5222
WINDOW=8192 RES=0x00 SYN URGP=0
(wrapped)
Ok, that was much too long, there was probably a bug somewhere and the intention was to show
only the outer IP header.
Recently I switched to kernel 4.9 but now the packets are logged with no info at all:
May 9 18:05:57 gw-44-137 kernel: [359091.001991] Packet DROP: IN=tunl0 OUT= MAC=
SRC=192.168.15.2 DST=44.137.75.242 LEN=243 TOS=0x00 PREC=0x00 TTL=123 ID=39243
PROTO=UDP SPT=5198 DPT=5198 LEN=223
The MAC field is now simply empty.
With the first format I had a small perl script that processed the log entries and showed the
tunnel endpoint that was sending the packet:
Apr 16 07:39:13 gw-44-137 kernel: [1266597.275238] Packet DROP: IN=tunl0 OUT=eth1
TUNL=84.106.126.184 SRC=192.168.88.10 DST=44.137.42.81 LEN=48 TOS=0x00 PREC=0x00
TTL=126 ID=25633 DF PROTO=TCP SPT=59840 DPT=5222 WINDOW=8192 RES=0x00 SYN URGP=0
However, now this is no longer possible.
I have tried finding info about it but no success yet.
Does anyone know if there is some parameter to tune this behaviour?
Rob
> In the past with Linux kernel 3.2 and 3.16
> Recently I switched to kernel 4.9
That was incorrect. The first behaviour was with kernel 3.2 and the 3.16
kernel does not log that long MAC address anymore. The 4.9 kernel is on
another system, it is not related to this.
Rob
If your gateway appears in the pkterrors.txt file, the packets which
caused that error to be logged and the packet to be dropped are now
available in a file you can retrieve with your web browser. They are
binary log files, so you'll need a program to interpret them. The URL
for a typical file is
https://gw.ampr.org/private/errors/67.164.64.8.bin
where of course the IP address part changes to whatever your gateway
address is. The files are removed and start fresh at midnight Pacific
time (GMT-7 or -8). For some error-prone sites, they get large-ish.
The format of each file is
/*
* 2 bytes error number (unsigned short)
* 2 bytes packet length (unsigned short)
* 4 bytes time (seconds since epoch)
* 4 bytes fractional seconds (microseconds)
* n bytes (packetlen) encapped IP packet in network byte order
*/
I have a 'C' program that will interpret the file, which you may have
if you're interested. It calls some library routines that you probably
don't have, so you'll have to modify it to get it to run on your system.
In particular, the error number and packet content interpreters are up
to you. I don't think the compiled code will run on Linux but you're
welcome to it if you have a FreeBSD system to run it on.
Or if you like, I'll run it on your gateway error log file and mail you
the output. That looks like this:
timestamp (GMT) len err error
-------------------- ---- ---- ----------
2017-05-08T19:04:32Z 40 [19] dropped: non-44 inner source address
ver=4 hl*4=20 tos=00 ip_len=40 id=d431 off=0000 ttl=244 proto=6 cksum=7d51 [TCP] 184.105.139.107:44864 -> 44.118.5.2:16992
2017-05-08T19:05:51Z 40 [19] dropped: non-44 inner source address
ver=4 hl*4=20 tos=00 ip_len=40 id=18f4 off=0000 ttl=43 proto=6 cksum=5b02 [TCP] 181.23.53.75:58147 -> 44.118.5.2:2222
etc.
So far, keeping these log files doesn't seem to burden the system
very much. If that changes, I'll have to discontinue them.
- Brian
