On Sun, May 5, 2024 at 6:00 PM lleachii@aol.com <lleachii@aol.com> wrote:
> Dan,
>
> Yes - the archives show we experienced packet loss in November 2020 and began to discuss it on the reflector. The losses were mostly UDP.

Interesting.

What's curious about this (to me, anyway) is that it's not _just_
packet loss, but (seemingly?) only loss of very specific packets. If
the problem were just random UDP packet loss, I'd expect some of the
queries that I observe to always succeed to occasionally fail, while
some of those that always seem to fail would occasionally succeed.
Instead, I see repeatable patterns of success and failure. Indeed, I
can alternate queries between succeeding and failure and the results
are very consistent.

Curiously, in between my original message and now, some queries
started working (consistently!) while others still consistently fail.
Also, just as a data point, the note about UDP reminded me that I
ought to test over TCP, and that works reliably for every case that I
tried.

> If I understand the issue you describe, any of us can help you test by running tcpdump on our tunl0 interface to determine if we receive 'PTR?' packets thru AMPRGW from your public IP, correct?
>
> tcpdump -vvvn -i tunl0 udp and port 53 and host 44.60.44.3 and host xxx.xxx.xxx.xxx
>
> where xxx.xxx.xxx.xxx == a Public IP

I think that's right; I've been running tests from 166.84.136.80. I
just tried to send a couple of queries to `dns-mdc.ampr.org` but got
no response.

> I have seen various failures in the past.
>
> I have one theory. Some administrators blacklist the 44 space.

If that were the case, I'd expect all success or all failure, or some
random combination of success and failure.

But instead, I see very specific success and failure cases, with no
discernable pattern to what works and what doesn't. It doesn't appear
to be totally random, nor is it an all-or-nothing sort of thing.

I can think of a few possibilities/questions.

1. Are these packets even making it to the upstream side of AMPRGW? I
have no way of knowing, really, and it's possible AMPRGW never sees
them. But why just these packets containing these specific PTR
queries? I suspect most of them arrive at UCSD upstream, but don't get
passed.
2. Does AMPRGW block incoming DNS requests to non-authoritative
servers? Or maybe there's an allowlist somewhere I'm not on? This
seems unlikely; first, no one seems to know anything about that and
I'd figure that they would if that sort of thing existed. Second, if
that were the case, I'd suspect all DNS queries to fail, or at least
all of a specific type (e.g., all `PTR?` queries or something like
that). But that doesn't fit the observed data.
3. Could it be that AMPRGW is doing some kind of deep packet
inspection and filtering queries that match certain, specific,
filters? I find this hard to believe; from what I know of AMPRGW
(which admittedly isn't THAT much, mostly reading between the lines of
old posts by Brian Kantor), I just don't think it's that kind of
system.

What seems more likely to me is that there's something specific, but
incidental, about the packets containing these queries that's tickling
AMPRGW in just the right way that it's dropping them. I looked at the
various stat files on gw.ampr.org/private, but nothing sticks out to
me as obvious.

Thanks again for the response, Lynwood.

        - Dan C.