Hello Will,
Modern ASIC based firewalls can handle 100,000s of stateless filters on a per interface basis.
Note I said 'router', not 'firewall'. Routers are designed from the silicon up to forward packets, reduce broadcast domains and connect networks. Firewalls are designed from the silicon up to restrict the flow of packets. Yes firewalls will forward packets from one network to another, but their primary purpose is inspection and restriction.
Oopps.. and I misspoke too.. modern ASIC "ROUTERS" can handle 100k filter rules w/o much impact to the performance to the interface. That does depend on the grade of the ASIC router though.
Again, please note that I said 'router', not 'firewall'. As to the type of router I was referring to in that specific example was a Cisco enterprise branch router. Campus and data center grade routers do minimal traffic filtering if any due to the CPU hit they incur, hence why large hardware firewalls exist. Proper tool for the job.
Branch routers aren't always ASIC based and many can be CPU based still. Also consider that many Cisco enterprise products are really switches with some level of L3/L4 routing included. When I talk about ASIC-based routers, I'm specifically talking about service provider class like Juniper MX, Cisco ASR, etc.
Just statelessly filtering at the border edge with a modern router would solve much of these issues.
Please note that router and firewall are not the same thing. They can do the same job, but not as effectively as the device purpose built for the job. Also Brian already stated:
Absolutely agreed and I personally think for this role at UCSD, a stateless packet filter only allowing the specific ACTIVATED AMPR routes would work well. Tom also recommended maybe running an IGP routing protocol on the FreeBSD box as a method to update the FIB routes on the upstream UCSD routers. That would also work well and effectively drops traffic via the lack of a route vs. a stateless packet filter.
Fire-walling is done at the AMPR edge, but traffic was overwhelming the current configuration. Moving filtering to the provider router is technologically improper and operationally restricted, hence my suggestion to split filtering and tunneling onto separate machines to increase capacity.
This is a problem that has been solved for a LONG time now. There are ways to programmatically update remote routers with dynamic filters, etc. One method to to this is the BGP enabled flowspec. There are other mechanisms to do this as well.
The suggestion Tom made of running an IGP to selectively advertise only subnets which have valid destinations via the tunnels would also restrict the amount of traffic that will ultimately be blocked from reaching the firewall. This type of routing combined with a large null route is a common practice in large enterprise networks. Reducing the amount of traffic that is going to get blocked from reaching the AMPR edge will help system load but won't help with the timeouts due to slow [or down] tunnel peers.
Agreed but this specific issue wasn't about tunnel peers and like any other Internet based network, the end to end performance depends on every link in the chain.
As this thread has demonstrated, there are a few different ways to increase capacity of the AMPR gateway. While it may not be necessary at this time, it's still useful information to have for whoever is going to be responding next time there is an issue.
While I agree the current problem has been mitigated, I think that a few additional improvements could go a LONG way to prevent this from happening again in the face of say DDOS attacks, etc.
--David KI6ZHD