Oops, left out the host name: kk7kx.ampr.org TCP -> port 7000 UDP -> port 7001 Thanks, Assi kk7kx/4x1kx On Mon, Jul 20, 2015 at 8:32 PM, Assi Friedman <assi(a)kiloxray.com> wrote:

we need a white-hat bot-net to shut down that black-hat bot-net!! ;-) On 15-07-21 02:32 AM, Brian Kantor wrote:

On Tue, Jul 21, 2015 at 11:00:54AM -0700, Tom Hayward wrote: They have already agreed to do so. - Brian

On Tue, Jul 21, 2015 at 11:13:53AM -0700, Bill Vodall wrote: I believe Tom is speaking of dividing up the address space among several routers, which would require some tricky route advertisement to provide any kind of redundancy. - Brian

On 7/21/15 1:32 AM, Brian Kantor wrote: What is the configuration of the UCSD gateway? I've asked this before and never received an answer, other than it uses 10g ports on some "ancient" version of FreeBSD, but the switch/router it's unlinked to is a 1g only switch. From what I know the gateway software does not do anything in kernel mode, but again the software is not public so we can't work to improve it, or study it. Why is it so hard to get details on how this stuff works? I'm not saying we need to have this level of detail, https://wikitech.wikimedia.org/wiki/Main_Page but it would be awesome to have some details on it. Get me a listing of the physical interconnects and a source tarball and I'll happily draw/write/format it up and add it on the wiki. -- Bryan Fields 727-409-1194 - Voice 727-214-2508 - Fax http://bryanfields.net

Would breaking the rule out into separate smaller rules help improve the bottleneck? It looks like ipfw uses a 'first rule wins', so perhaps re-ordering could help. Filter all the bogon and RFC1918 IPs out first, then filter out the netbios traffic and anything else that globally shouldn't be allowed, with the divert rule being simplified and left to the end. -- Will On 7/21/15 2:51 PM, Brian Kantor wrote:

Well, I found and fixed the cause of the packet loss problem. It turns out that the single-threaded nature of IP processing in the FreeBSD kernel means that when ipfw was told to forward the packet to the network telescope, the process blocked for a significant period of time while the outgoing packet was rewritten and enqueued. This caused the inbound work queue to lengthen to the point where incoming packets were ignored and dropped, which played Hob with the throughput for udp and tcp connections. With the cooperation of the CAIDA people, we stopped forwarding packets to the telescope and will instead feed it off a network switch mirror port. They will filter out our legitimate subnets leaving the IBR that they want to study. That this was the cause is shown by now lossless pinging of various end destinations on AMPRNet. For example, --- kk7kx.ampr.org ping statistics --- 100 packets transmitted, 100 received, 0% packet loss, time 99120ms rtt min/avg/max/mdev = 26.457/30.128/189.503/17.459 ms And is also shown by the reduced task queue on the input interface /0% /10 /20 /30 /40 /50 /60 /70 /80 /90 /100 root idle: cpu3 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX root idle: cpu2 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX root idle: cpu1 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX root idle: cpu0 XXXXXXXXXXXXXXXXXXXXXXXXX root em0 taskq XXXXXXXXXXXXXXXXXXXXX root ipipd X At the moment, input packet rates are running around 12 MB/s, which the amprgw seems to be handling easily. It's possible that the remaining ipfw rules could be optimized somewhat to reduce the input queue even further, but I think I'll call it a day for now. - Brian

Wow, what a difference. Here's iperf results from home to kk7kx.ampr.org: [root@raptor phantom]# iperf -c kk7kx.ampr.org -p 7000 -i 1 ------------------------------------------------------------ Client connecting to kk7kx.ampr.org, TCP port 7000 TCP window size: 19.1 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.12 port 51468 connected with 44.8.0.160 port 7000 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 1.12 MBytes 9.44 Mbits/sec [ 3] 1.0- 2.0 sec 1.00 MBytes 8.39 Mbits/sec [ 3] 2.0- 3.0 sec 1.12 MBytes 9.44 Mbits/sec [ 3] 3.0- 4.0 sec 896 KBytes 7.34 Mbits/sec [ 3] 4.0- 5.0 sec 896 KBytes 7.34 Mbits/sec [ 3] 5.0- 6.0 sec 768 KBytes 6.29 Mbits/sec [ 3] 6.0- 7.0 sec 512 KBytes 4.19 Mbits/sec [ 3] 7.0- 8.0 sec 640 KBytes 5.24 Mbits/sec [ 3] 8.0- 9.0 sec 640 KBytes 5.24 Mbits/sec [ 3] 9.0-10.0 sec 512 KBytes 4.19 Mbits/sec [ 3] 0.0-10.2 sec 8.12 MBytes 6.70 Mbits/sec [root@raptor phantom]# iperf -c kk7kx.ampr.org -p 7001 -i 1 -u ------------------------------------------------------------ Client connecting to kk7kx.ampr.org, UDP port 7001 Sending 1470 byte datagrams UDP buffer size: 122 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.12 port 53954 connected with 44.8.0.160 port 7001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 129 KBytes 1.06 Mbits/sec [ 3] 1.0- 2.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 2.0- 3.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 3.0- 4.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 4.0- 5.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 5.0- 6.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 6.0- 7.0 sec 129 KBytes 1.06 Mbits/sec [ 3] 7.0- 8.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 8.0- 9.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 9.0-10.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec [ 3] Sent 893 datagrams [ 3] Server Report: [ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 1.073 ms 0/ 893 (0%) Note the 0% packet loss in UDP mode. MTR results are also consistent with this. Thanks Brian! Assi On Tue, Jul 21, 2015 at 8:15 PM, Brian Kantor <Brian(a)ucsd.edu> wrote:

Hello Brian, Thanks for working on this issue. Few questions: 1. Was this "telescope" experiment a recent change to the system or has this been there for a long time? 2. Is there a specific reason why you're using FreeBSD vs. Linux? I would assume that linux's iptables is threaded and could perform better but I don't know for sure. 3. I liked Tom Hayward's idea to automatically filter netblocks that aren't activated in the portal / DNS. That seems like a very cheap way to knock out known bogus traffic. Ideally this would be done at the farthest edge of the network to prevent the traffic from ever even reaching the Dell server. --David KI6ZHD On 07/21/2015 08:15 PM, Brian Kantor wrote:

On 7/22/15 11:17 AM, Brian Kantor wrote: pf and ipfw on FreeBSD are true stateful firewalls, where no Linux firewall that I'm aware of is truly stateful. iptables treats each packet individually where pf/ipfw will add it as a flow and track bi-directional traffic for the duration of the connection. This is why pf / ipfw are not threaded, however they do automatically optimize rule sets when you load them to be as efficient as possible. Filtering at a router is a sure fire way to bring throughput to a crawl. Proper campus routers are designed with ASICs optimized for routing in hardware, and fire-walling is done in software. I have seen enterprise small office routers handle 450~500mbps of straight routing but max out around 40mbps when fire-walling because it's CPU bound. The results are similar when stepping up to large chassis routers. A better option in my opinion is splitting up tunnelling and fire-walling onto separate machines. This would allow which ever system can handle fire-walling or tunnelling best to be configured for each task and would increase throughput capacity. Of course this does require more rack-space, power, cooling, another system to configure, and someone with the time and energy to set it up. -- Will

On 22 Jul 2015, at 9:05 PM, Will Gwin <N5KH(a)n5kh.org> wrote: iptables (Netfilter) had stateful connection tracking from day one (AFAIK since ~14 years ago). Both Linux and BSD network stacks are very mature in both features and performance. Differences exist (I believe FreeBSD is more efficient per packet) but won't matter for AMPRNet (ever) since the kernels are now being tuned to keep up with the fastest NICs on the market (that is, 100Gb/s, which is probably faster than all AMPRNet tunnels combined for the foreseeable future). At work, my colleague runs the 10Gb/s-capable firewall for the whole division with IPFW (no issues, so no need to re-write the policy for PF): http://stats.meraka.csir.co.za/cacti/graph_view.php?action=tree&tree_id… Does the gateway need stateful filtering? If not, this can be done at line-rate at the router with ACLs. I would be curious to know if the current gateway is configured to track connection states, and if so, how many concurrent connections it peaks at?

Brian et al; You could monitor it with a simple stats tool such as MRTG or if you want deeper stats look into Ganglia. -- A computer once beat me at chess, but it was no match against me in kick-boxing - Emo Phillips 73 de Brian Rogers - N1URO email: (see above) Web: http://www.n1uro.net/ Ampr1: http://n1uro.ampr.org/ Ampr2: http://nos.n1uro.ampr.org Linux Amateur Radio Services axMail-Fax & URONode http://uronode.sourceforge.net http://axmail.sourceforge.net AmprNet coordinator for: Connecticut, Delaware, Maine, Maryland, Massachusetts, New Hampshire, Pennsylvania, Rhode Island, and Vermont.

On 7/22/15 10:27 PM, David Ranch wrote: My mistake. It's been years since I was actively comparing the different fire-walling methods in use by Linux. I went hardware for years and only within the last few years went to software, when I went OpenBSD due to native IPsec support as well as pf. Note I said 'router', not 'firewall'. Routers are designed from the silicon up to forward packets, reduce broadcast domains and connect networks. Firewalls are designed from the silicon up to restrict the flow of packets. Yes firewalls will forward packets from one network to another, but their primary purpose is inspection and restriction. Again, please note that I said 'router', not 'firewall'. As to the type of router I was referring to in that specific example was a Cisco enterprise branch router. Campus and data center grade routers do minimal traffic filtering if any due to the CPU hit they incur, hence why large hardware firewalls exist. Proper tool for the job. The current configuration was choking, hence the discussion. Brian has worked with CAIDA and resolved the congestion for now. Please note that router and firewall are not the same thing. They can do the same job, but not as effectively as the device purpose built for the job. Also Brian already stated: and Fire-walling is done at the AMPR edge, but traffic was overwhelming the current configuration. Moving filtering to the provider router is technologically improper and operationally restricted, hence my suggestion to split filtering and tunneling onto separate machines to increase capacity. The suggestion Tom made of running an IGP to selectively advertise only subnets which have valid destinations via the tunnels would also restrict the amount of traffic that will ultimately be blocked from reaching the firewall. This type of routing combined with a large null route is a common practice in large enterprise networks. Reducing the amount of traffic that is going to get blocked from reaching the AMPR edge will help system load but won't help with the timeouts due to slow [or down] tunnel peers. As this thread has demonstrated, there are a few different ways to increase capacity of the AMPR gateway. While it may not be necessary at this time, it's still useful information to have for whoever is going to be responding next time there is an issue. -- Will