Having the robots.txt file at the site AMPR.ORG apply to pages served by that host is completely reasonable.
Having the robots.txt file at the site AMPR.ORG apply to pages served from an entirely separate site called, e.g., VA7LBB.AMPR.ORG is one of the most colossal pieces of internet engineering stupidity I have ever encountered.
Yet it would appear that is what the person at Google is telling you is happening to your site.
From Wikipedia:
"A robots.txt file covers one origin. For websites with multiple subdomains, each subdomain must have its own robots.txt file. If example.com had a robots.txt file but a.example.com did not, the rules that would apply for example.com WOULD NOT APPLY to a.example.com. In addition, each protocol and port needs its own robots.txt file; http://example.com/robots.txt does not apply to pages under http://example.com:8080/ or https://example.com/"
If that's not what their crawlers are doing, their crawlers are broken. They broke it, they get to fix it. It's not our problem. - Brian
On Thu, May 02, 2019 at 06:13:40PM -0700, Roger Andrews wrote:
Hi, I recently talked with Brian briefly about this and wanted to throw it out to the group. It’s incredibly rare to see any of the tunnels that have been created, represented in a Google search. While I understand and agree that any site that will become a high volume site has no place on Amprnet (we have to share resources) it also seems pointless to create a website that is undiscoverable. After all, isn’t the primary purpose of a website to share it’s content with others. I recently created a website on a 44net gateway and after several weeks, (and even convincing Brian to add a meta TXT entry allowing me to ask google to crawl), I am not seeing any content on Google. I put in a service request to google (not the easiest task) and I was advised that robots.txt or some other prevention device is blocking indexing all the subdirectories on amp.org. I was told that the few gateways that I see in the results were likely crawled before the restriction on ampr.org was applied. I created the website for our ARES group and placed it on an ampr gateway because we don’t have funds, and in reality, see very little traffic. We had a .net site last year and averaged about 50 visitors a month. My question is - is it really necessary to prevent the whole of ampr.org from being crawled (except of course the top domain which does show up). So many ip addresses, but almost none visible seems a real pity. Thanks for listening. My only hope is that this creates a little bit of debate around the issue.
73 Roger VA7LBB