The DDoS attack on net 44 continues. I'm filtering out a goodly amount of it at amprgw, but the people whose subnets are directly connected (BGP announced) are getting hit too, and there's nothing I can do to filter it out here. Basically, if you're directly connected (ie, not on a tunnel), you have to add a list of bad guys to your own firewall blocking.
Here is a little script that will create a list of candidate addresses for blocking. You'll need the 'badguys' program, which I'll post below. What this does is give you a list of addrsses that sent you more than 1000 packets in the sample period.
If you're connected via tunnel, you probably don't need this.
I run this several times a day to accumulate a list of bad guys. I sample 100 million packets at a time; smaller subnets will probably want to use a smaller sample size.
This was developed on FreeBSD but I'm told it works fine on Linux
Hope this helps. - Brian
----------------------------------------------------------- block.sh: #!/bin/sh # # generate a list of possible bad guys #
# sample incoming traffic, 100,000,000 incoming packets # during DDoS storm on a /8, this takes about 3 minutes
time tcpdump -w /tmp/t.pcap -s 40 -c 100000000
# turn into a list of source IP addresses with counts, # sorted by decreasing count # throw away all those counted less than 1000, # delete the count from the line # elide our own addresses (put your own in the egrep) # store in a file
./badguys /tmp/t.pcap \ | sort \ | uniq -c \ | sort -rn \ | sed -e '/^ /d' \ | sed -e 's/.* //' \ | egrep -v '^44.|^169.228.' \ > /tmp/badguys
# all done with this file rm -f /tmp/t.pcap
exit 0 ----------------------------------------------------------- badguys.c: /* * reads pcap capture file (use tcpdump -w to create) named * * spins through it, listing the IP source address on stdout * prints summary statistics on stderr * * the documentation on the pcap library leaves a lot to be * desired, use the source. a lot of what you need is in tcpdump.c * * compile me with * cc -g -Wall badguys.c -lpcap -o badguys * * (You'll need to have libpcap installed) * * Brian Kantor, UCSD, 2017 */
/* probably most of these includes are superfluous */ #include <sys/errno.h> #include <sys/types.h>
#include <ctype.h> #include <errno.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h>
#include <net/ethernet.h>
#include <netinet/in.h> #include <netinet/in_systm.h> #include <netinet/ip.h>
#include <arpa/inet.h>
#include <pcap/pcap.h> #include <pcap/bpf.h>
void getpkt(u_char *, const struct pcap_pkthdr *, const u_char *);
char * iptoa(u_long);
int capcount = 0; int skipcount = 0;
int main(int argc, char**argv) { char inpcapname[FILENAME_MAX+1]; char errbuf[BUFSIZ]; pcap_t *pcap; int rslt;
if (argc != 2) { fprintf(stderr, "Usage: %s pcapfilename\n", argv[0]); exit(1); }
strncpy(inpcapname, argv[1], FILENAME_MAX); fprintf(stderr, "opening PCAP file %s\n", inpcapname);
pcap = pcap_open_offline(inpcapname, errbuf); if (!pcap) { fprintf(stderr, "Error: %s\n", errbuf); exit(1); } /* * read through the capture savefile */ rslt = pcap_loop(pcap, 0, getpkt, 0); if (rslt) { fprintf(stderr, "pcap_loop returned %d\n", rslt); pcap_perror(pcap, "pcap_loop returned"); exit(1); } pcap_close(pcap);
fprintf(stderr, "%d packets read from savefile\n", capcount); fprintf(stderr, "%d non-IP packets discarded\n", skipcount);
exit(0); }
/* * called by pcap_loop once per packet read from savefile */ void getpkt(u_char *user, const struct pcap_pkthdr *h, const u_char *p) { struct ip * ip; u_short * ethertype;
capcount++;
/* get the Ethernet packet type */ ethertype = (u_short *)&p[(2*ETHER_ADDR_LEN)];
/* we only want IP packets */ if (ntohs(*ethertype) != ETHERTYPE_IP) { skipcount++; return; }
/* IP packet starts after Ethernet header */ ip = (struct ip *)&p[ETHER_HDR_LEN];
puts(iptoa(ntohl(ip->ip_src.s_addr)));
return; }
/* * nicely format host-byte-order u_int into dotted quad * returns pointer to a small static buffer; you can't * call this more than once in a single syslog or printf statement */ char * iptoa(u_long ipaddr) { static char buf[32];
snprintf(buf, sizeof buf, "%lu.%lu.%lu.%lu", (ipaddr >> 24) & 0xff, (ipaddr >> 16) & 0xff, (ipaddr >> 8) & 0xff, (ipaddr) & 0xff);
return buf; }
Does this addresses stay forever in the block list ? or it is released after some time ?
________________________________ From: 44Net 44net-bounces+ronenp=hotmail.com@hamradio.ucsd.edu on behalf of Brian Kantor Brian@UCSD.Edu Sent: Tuesday, May 9, 2017 9:40 PM To: 44net@hamradio.ucsd.edu Subject: [44net] storm blocking
(Please trim inclusions from previous messages) _______________________________________________ I'm filtering out a goodly amount of it at amprgw,
On amprgw, I clear the list and regenerate it after a couple of days. It's a manual operation, not automated. It would be up to you to clear them out yourself if you were doing your own blocking. - Brian
On Wed, May 10, 2017 at 05:28:36AM +0000, R P wrote:
Does this addresses stay forever in the block list ? or it is released after some time ?
On 5/10/17 12:40 AM, Brian Kantor wrote:
The DDoS attack on net 44 continues. I'm filtering out a goodly amount of it at amprgw, but the people whose subnets are directly connected (BGP announced) are getting hit too, and there's nothing I can do to filter it out here. Basically, if you're directly connected (ie, not on a tunnel), you have to add a list of bad guys to your own firewall blocking.
I'm not seeing anything out of the ordinary on any of our blocks here in the bay area.
Seeing about ~1kpps here on our upstream and we have several "high traffic" voip hubs on here.
We did have a user leave a radio with default passwords on it the other day. Luckily someone on the internet was able to change this for him.
I'm seeing around 30 million packets a minute here, but of course a /8 is going to see more than narrower subnets. What's startling is the number of source addresses. I've seen over 25,000 different sources in the space of just minutes at the storm's peak. Just a moment ago, there were 6,000 different sources in 100 million incoming packets.
It's running about 30 MB/s, most of which is very short TCP connect requests to seemingly random 44-net addresses. Our normal (non-storm) inbound is around 18 MB/s. This past weekend, I saw peaks of 60 MB/s or higher. - Brian
On Wed, May 10, 2017 at 04:56:50AM -0400, Bryan Fields wrote:
I'm not seeing anything out of the ordinary on any of our blocks here in the bay area.
Seeing about ~1kpps here on our upstream and we have several "high traffic" voip hubs on here.
On Tue, May 9, 2017 at 9:40 PM, Brian Kantor Brian@ucsd.edu wrote:
What this does is give you a list of addrsses that sent you more than 1000 packets in the sample period.
...
# sample incoming traffic, 100,000,000 incoming packets # during DDoS storm on a /8, this takes about 3 minutes
time tcpdump -w /tmp/t.pcap -s 40 -c 100000000
If I'm reading this right, the "unacceptable" level of usage of amprgw is 5pps (1000 packets / 180 seconds), or a maximum of about 8 KB/s assuming 1480 bytes/packet. This seems extremely low. I bet two simultaneous Allstar link (VoIP) conversations could get your address blocked.
Just now, it took 287 seconds to gather 100 million packets, comprising
Worse, in periods of lower traffic, it will take longer to collect 100000000 packets, so the allowed pps goes down. If 1000 packets per 287 seconds is the new threshold, hosts sending 3.5pps (max 5 KB/s) through amprgw will be blocked. This is a fatal flaw in the system because it can cause runaway--the more hosts blocked, the longer it will take to collect 100 million packets, continually decreasing the allowed pps.
Surely there will be some false positives with this threshold.
Tom KD7LXL
Tcpdump collection takes place off the raw interface, BEFORE any packets are blocked, so the amount of blocking doesn't affect the sample.
Unfortunately, tcpdump doesn't have a 'run for N seconds then exit' option. I can do it manually but that ruins the automation.
Still, you're probably right about the threshold being too low.
Right now the received packet rate is about 25 MB/s, which is a little high for normal, but not unusual. So I'm going to turn off the blocking (which won't affect the received rate as again it's sampled before the blocking) and we'll see how the router program statistics fare.
Thanks! - Brian
On Wed, May 10, 2017 at 10:03:10AM -0700, Tom Hayward wrote:
Worse, in periods of lower traffic, it will take longer to collect 100000000 packets, so the allowed pps goes down. If 1000 packets per 287 seconds is the new threshold, hosts sending 3.5pps (max 5 KB/s) through amprgw will be blocked. This is a fatal flaw in the system because it can cause runaway--the more hosts blocked, the longer it will take to collect 100 million packets, continually decreasing the allowed pps.
Surely there will be some false positives with this threshold.
On Wed, May 10, 2017 at 10:18 AM, Brian Kantor Brian@ucsd.edu wrote:
Unfortunately, tcpdump doesn't have a 'run for N seconds then exit' option. I can do it manually but that ruins the automation.
How about `tcpdump -G 15 -W 1`?
Tom
A 10-second sample just now shows 5400 packets inbound to known AMPRNet addresses, and the remaining 3622300 packets discarded for no known destination. - Brian
On Wed, May 10, 2017 at 10:18:16AM -0700, Brian Kantor wrote:
Right now the received packet rate is about 25 MB/s, which is a little high for normal, but not unusual. So I'm going to turn off the blocking (which won't affect the received rate as again it's sampled before the blocking) and we'll see how the router program statistics fare.
A one-minute sample just now yields 22 million packets from 533,000 different source addresses. This seems high to me. - Brian
On Wed, May 10, 2017 at 10:35:28AM -0700, Brian Kantor wrote:
A 10-second sample just now shows 5400 packets inbound to known AMPRNet addresses, and the remaining 3622300 packets discarded for no known destination.
- Brian
On Wed, May 10, 2017 at 11:01 AM, Brian Kantor Brian@ucsd.edu wrote:
A one-minute sample just now yields 22 million packets from 533,000 different source addresses. This seems high to me.
How many of those packets are to known destinations?
Have you inspected the contents at all to see if there are similarities?
Tom KD7LXL
On Wed, May 10, 2017 at 11:15:45AM -0700, Tom Hayward wrote:
How many of those packets are to known destinations?
From past samples, not many. For example, a previous
sample showed 5400 to known hosts out of 3.6 million captured.
Have you inspected the contents at all to see if there are similarities?
Yes. The majority of the packets are TCP open requests with no data, mostly to ports 23 and 80.
A significant number are UDP to port 53, which could be probes or it could be innocent hosts trying to look up the names of hosts they are receiving IBR packets from.
Part of the difficulty in analyzing these is that tcpdump is SLOW; it can take a very long time to display all the packets in order to do statistical analysis. Also, the capture size is only 40 bytes; we'd have to re-do the capture with a larger capture size in order to get the full packet decode. - Brian
On Wed, May 10, 2017 at 11:26 AM, Brian Kantor Brian@ucsd.edu wrote:
Have you inspected the contents at all to see if there are similarities?
Yes. The majority of the packets are TCP open requests with no data, mostly to ports 23 and 80.
I'm just trying to brainstorm ways you could reduce the number of false positives...
Maybe instead of counting packets, you could count TCP open requests. While >1000 packets per minute might be totally normal for an FTP transfer session, 1000 TCP open requests to unroutable addresses is much more abnormal.
Tom KD7LXL
What is the mechanism that decide who to add to the black list ?
Can it be that by accident adresses that are part of legal communication to legal amprnet host will be blocked ?
example say i look on a video camera that sit on amprnet at my home from work ... trafic can be high due to the fact that its video so bitrate can be high ....
________________________________
Right now the received packet rate is about 25 MB/s, which is a little high for normal, but not unusual.