Wow, what a difference. Here's iperf results from home to kk7kx.ampr.org:
[root@raptor phantom]# iperf -c kk7kx.ampr.org -p 7000 -i 1 ------------------------------------------------------------ Client connecting to kk7kx.ampr.org, TCP port 7000 TCP window size: 19.1 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.12 port 51468 connected with 44.8.0.160 port 7000 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 1.12 MBytes 9.44 Mbits/sec [ 3] 1.0- 2.0 sec 1.00 MBytes 8.39 Mbits/sec [ 3] 2.0- 3.0 sec 1.12 MBytes 9.44 Mbits/sec [ 3] 3.0- 4.0 sec 896 KBytes 7.34 Mbits/sec [ 3] 4.0- 5.0 sec 896 KBytes 7.34 Mbits/sec [ 3] 5.0- 6.0 sec 768 KBytes 6.29 Mbits/sec [ 3] 6.0- 7.0 sec 512 KBytes 4.19 Mbits/sec [ 3] 7.0- 8.0 sec 640 KBytes 5.24 Mbits/sec [ 3] 8.0- 9.0 sec 640 KBytes 5.24 Mbits/sec [ 3] 9.0-10.0 sec 512 KBytes 4.19 Mbits/sec [ 3] 0.0-10.2 sec 8.12 MBytes 6.70 Mbits/sec
[root@raptor phantom]# iperf -c kk7kx.ampr.org -p 7001 -i 1 -u ------------------------------------------------------------ Client connecting to kk7kx.ampr.org, UDP port 7001 Sending 1470 byte datagrams UDP buffer size: 122 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.12 port 53954 connected with 44.8.0.160 port 7001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 129 KBytes 1.06 Mbits/sec [ 3] 1.0- 2.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 2.0- 3.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 3.0- 4.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 4.0- 5.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 5.0- 6.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 6.0- 7.0 sec 129 KBytes 1.06 Mbits/sec [ 3] 7.0- 8.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 8.0- 9.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 9.0-10.0 sec 128 KBytes 1.05 Mbits/sec [ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec [ 3] Sent 893 datagrams [ 3] Server Report: [ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 1.073 ms 0/ 893 (0%)
Note the 0% packet loss in UDP mode. MTR results are also consistent with this.
Thanks Brian! Assi
On Tue, Jul 21, 2015 at 8:15 PM, Brian Kantor Brian@ucsd.edu wrote:
(Please trim inclusions from previous messages) _______________________________________________ Well, I found and fixed the cause of the packet loss problem.
It turns out that the single-threaded nature of IP processing in the FreeBSD kernel means that when ipfw was told to forward the packet to the network telescope, the process blocked for a significant period of time while the outgoing packet was rewritten and enqueued. This caused the inbound work queue to lengthen to the point where incoming packets were ignored and dropped, which played Hob with the throughput for udp and tcp connections.
With the cooperation of the CAIDA people, we stopped forwarding packets to the telescope and will instead feed it off a network switch mirror port. They will filter out our legitimate subnets leaving the IBR that they want to study.
That this was the cause is shown by now lossless pinging of various end destinations on AMPRNet. For example,
--- kk7kx.ampr.org ping statistics --- 100 packets transmitted, 100 received, 0% packet loss, time 99120ms rtt min/avg/max/mdev = 26.457/30.128/189.503/17.459 ms
And is also shown by the reduced task queue on the input interface
/0% /10 /20 /30 /40 /50 /60 /70 /80 /90 /100root idle: cpu3 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX root idle: cpu2 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX root idle: cpu1 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX root idle: cpu0 XXXXXXXXXXXXXXXXXXXXXXXXX root em0 taskq XXXXXXXXXXXXXXXXXXXXX root ipipd X
At the moment, input packet rates are running around 12 MB/s, which the amprgw seems to be handling easily. It's possible that the remaining ipfw rules could be optimized somewhat to reduce the input queue even further, but I think I'll call it a day for now. - Brian
44Net mailing list 44Net@hamradio.ucsd.edu http://hamradio.ucsd.edu/mailman/listinfo/44net