> I think the most efficient technique is to make the per-address lookup
> table an mmap'd array of pointers to entries in the existing route table.
> That makes it effectively addrtable[2][2**24], right?
Yes. Of course a single pointer would be 8 bytes when it is compiled for 64 bits,
but it would not need to be. When your existing route table is an array
rather than a collection of malloc'ed objects linked by pointers, the "pointer"
from the address lookup table into the route table could be the smaller index
into the route table (that would easily fit in an unsigned short integer, allowing
for 65535 gateways).
Or, in 32 bit mode a simple pointer can be used (4 bytes per entry).
Rob
> You're absolutely right. I dropped some decimal places. Thanks! However,
> gateway addresses don't fit in a byte, they're stored in an unsigned long,
> which is 4 bytes each. 4 * 16 million, right? And I think you need
> counters and flags. So it's an array of structs of some modest size each.
My estimate of 128MB was based on having 4 bytes per entry and 2 tables for
convenient updating (update one table then toggle a single indicator or pointer
to make the updated table active).
Of course when you require more bytes per entry the table will expand, but
8-16 bytes per entry should still fit comfortably in a modern machine.
> But I see a complication: You will have to have n entries per route,
> which will make loading the table a little less straightforward.
Well, searching a datastructure for the correct route isn't straightforward
either... remember when there is a tunnel to e.g. 44.137.0.0/16 and another
one to 44.137.1.80/28 (an example from the current table) then any traffic to
44.137.1.81 should go to the tunnel for 44.137.1.80/28, not something that
you can easily do with a binary search. However, a lookup in the table
populated with entries for 44.137.0.0/16 and then overwritten with entries
for 44.137.1.80/28 is easy. It will find the correct gateway with a single
array index operation. So you only need to build the table starting
from the subnets sorted by number of subnet bits to get it right.
Indeed it is best to make the program multithreaded, or you could put the
lookup table in shared memory and have a process doing the routing and another
process doing the updating. I would still opt for having a "current" and
a "next" table where the routing code always has a working table and the
switch to the next version is instaneous.
Of course you can also do a quick check before starting the laborious update
process, to see if the new encap table is different from the previous one.
Rob
> My estimate of 128MB was based on having 4 bytes per entry and 2 tables for
> convenient updating (update one table then toggle a single indicator or pointer
> to make the updated table active).
> Of course when you require more bytes per entry the table will expand, but
> 8-16 bytes per entry should still fit comfortably in a modern machine.
Another possibility would be to have a 16-million entry array of short integers
holding a "gateway number" (starting at 1) for each IP address, and a separate
table of gateways holding all the other info you want to keep per gateway.
(e.g. counters)
Then the processing of a packet would first index the destination IP in the
first array, retrieving the gateway number (0 means drop the packet), then
use that number as an index in the gateway table to access the per-gateway
data including the endpoint address and the counters. This would require only
slightly more than 32MB of memory for the tables, which should be no problem without
any tricks.
Rob
> 128MB is only 3% of 4GB. What would be the problem reserving that for a lookup table?
BTW, a nice trick for such tables is to do a mmap of /dev/zero with the proper size as a
starting point, then write only the entries that are nonzero.
That way the vast spaces that are only zeroes will not occupy physical memory, and when
they are read by the forwarding code they read back as zeroes.
Linux also has a "|MAP_ANONYMOUS" flag to obtain the same result, I don't know if BSD has that. |Rob
> Interesting concept. Someday if we have enough memory I may try it, but
> right now amprgw (an old machine) has only 4 GB of memory. It'd die swapping.
128MB is only 3% of 4GB. What would be the problem reserving that for a lookup table?
Rob
On Sun, Apr 30, 2017 at 5:31 AM, Marc <monsieurmarc(a)btinternet.com> wrote:
> Maybe we should start sharing block lists.
Marc,
HamWAN has a public blacklist system. Feel free to subscribe to it. It
does not publish a full list, but rather sends addresses one by one,
instantly*, as they are blocked.
*It takes about 1.5 seconds for report of a hack attempt to propagate
to our logging system, pass analysis, and be published to our edge
routers' firewall.
Here is the code behind the system (including a Mikrotik script you
can use to subscribe):
https://github.com/kd7lxl/blacklist-service
Anything blocked by the HamWAN network will be published here:
http://monitoring.hamwan.net/blacklist
If it seems like it's not responding, that's normal. It is an HTTP
longpoll service, so it will hang until there is data to be published,
then that data is sent immediately. This mechanism allows pushing data
(in this case, a blacklisted address) to a Mikrotik router without
having to store admin credentials of that router on the blacklist
system. Since it uses a standard protocol, it can be adapted for other
platforms.
Tom KD7LXL
As I've mentioned, I'm now logging statistics for the amprgw
router. I could make these available on a web site, but I'm
hesitant to do so without consensus from the community because
the per-gateway stats have the side effect of revealing what
subnet goes to what gateway, thus giving the bad guys more
insight into how the AMPRNet-Internet gateway works and what
hosts to impersonate to inject bad packets into the network.
A sample of the statistics file is below.
What do you folks think about making this available?
- Brian
started at Sat Apr 29 10:15:37 2017
snapshot at Sat Apr 29 10:45:00 2017
uptime: 0+00:29:23 (1763 seconds)
packets/bytes
---------/---------
121781/76089093 ipip encapped input
117631/73371842 forwarded out
0/0 no source gateway
467445/48802372 raw input
467434/48801678 encapped out
0/0 no destination gateway
2646/175516 encap to encap dropped
7692 discarded
0 TTL exceeded
0 ICMP sent
615 routes loaded (gateway counters cleared) 1486 seconds ago at Sat Apr 29 10:20:14 2017
entry subnet encap gateway outpkts outbytes inpkts inbytes
----- ------------------- ----- ---------------- --------- --------- --------- ---------
0 44.0.0.1/32 ipip 169.228.66.251 2 175 0 0
[etc]
The portal has apparently crashed. I have no way to remotely
restart it, so it'll probably be down all night until Chris
wakes up and can take a look at it.
- Brian
Recently I noticed people sending IGMP protocol packets to amprgw, which
discards them (as it does for all multicast destinations sent to it).
I don't think I'm going to write the necessary code to handle group
membership, so dropping the packet is probably the best thing to do.
Amprgw currently does NOT send 'destination unreachable' ICMP packets,
including 'protocol unreachable', because those should be rate limited
which makes coding the necessary response mechanism somewhat complicated.
(True, I could steal the necessary code from the FreeBSD kernel and
adapt it, but that's a lot of work. Maybe someday.) And I also don't
want to use up encap bandwidth sending ICMP back to the AMPR hosts.
The only ICMP that amprgw currently will originate is TIMXCEED when an
encapped packet's TTL reaches zero, so that 'traceroute' will work.
- Brian
> That is the Mirkotik discovery protocol indeed. I struggled with this too at first until I found that you can enable/disable it on select interfaces. By default it sends and listens on all interfaces. I'll post a small tutorial on where to find and disable it per interface when I arrive at work (unless someone beats me to it)
> I also firewalled in&outbound that on all but my internal interfaces just to be extra certain. I would recomend everyone doing so too unless you need it for some reason on an external interface.
> Like with Cisco's CDP or Juniper's LLDP, you normally don't need it on external interfaces.
Well, actually it would not be so bad to run this, and implement a receiver program on amprgw and other gateways.
It provides useful info about what is at the other end of the tunnel, including the IP address of the router,
software version, identity, etc.
When everyone would be running this, you would have an immediate overview about which tunnels are alive, etc.
A bit problematic could be that the router likely sends all packets at the same time and thus some 400 packets
are queued for output at once, clogging up the internet interface.
Rob
