Hi all,
I’m a bit sorry to have brought the issue of google’s inability to crawl ampr.org subdomains. I thought others might also feel that it would be nice to have their ARES group or other Ham group represented on the Ham network (44Net) for the work to see. Instead it feels like there is no interest at all in figuring out why this happens, instead blaming it on Google, or me for brining the issue forward and trying to contact google for help (you do realize that the only way to actually talk to google is to know someone there, so I’m sorry that I have “a friend” at google that tried to help with some inside information that the google search console does not provide.)
Roger
VA7LBB
> On May 4, 2019, at 12:00, 44net-request(a)mailman.ampr.org wrote:
>
> Send 44Net mailing list submissions to
> 44net(a)mailman.ampr.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://mailman.ampr.org/mailman/listinfo/44net
> or, via email, send a message with subject or body 'help' to
> 44net-request(a)mailman.ampr.org
>
> You can reach the person managing the list at
> 44net-owner(a)mailman.ampr.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of 44Net digest..."
> Today's Topics:
>
> 1. Re: Google indexing (Roger)
> 2. Re: Google indexing (Roger)
> 3. Re: Google indexing (Steve L)
> 4. Re: Google indexing (Rob Janssen)
> <mime-attachment>
> <mime-attachment>
> <mime-attachment>
> <mime-attachment>
> _______________________________________________
> 44Net mailing list
> 44Net(a)mailman.ampr.org
> https://mailman.ampr.org/mailman/listinfo/44net
Rob,
When I put ampr.org in the google search almost all come back as first the TLD and wiki. The others are sites talking about ampr.org and the group that are subdomains are old dead links.
As I said in the first email, I did consult the google search console and submitted the site. I’ve submitted the site and Brian was kind enough to add. a TXT line (as per google search console) with the code proving I have the right to submit the domain. That is several weeks ago and the search console shows it attempted to crawl. Googles server now (unlike years ago) honor request to not crawl. It attempted the crawl and declined to proceed.
You probably know that it’s nearly impossible to contact google for support. Therein lies my luck to have a friend that could look behind the scene at the google server. His answer is that this is not a google problem but an issue with some sort of blocking at the TLD. He just can’t tell what that is except, the google crawler is being told to not crawl subdomains in some way.
All I was asking was that if someone more knowledgeable than me with access to the DNS could see what possibly would cause this denial.
Have a careful look through the google search response to ampr.org. There are none.
Roger.
VA7LBB
On May 4, 2019, at 02:19, Rob Janssen <pe1chl(a)amsat.org> wrote:
>> My friend at google isn’t 100% sure what’s happening
>
> That is very obvious!
> Maybe you should consult the google search engine itself, instead of your friend.
>
> Enter this in the Google search bar: site:ampr.org
>
> This returns 17700 results of all kinds of different sites below .ampr.org.
> You see the domain names xxxx.ampr.org listed in the results.
>
> So there really is no problem indexing ampr.org sites.
>
> I think what you encounter is the difficulty to add a NEW site to the index, not any generic
> problem with .ampr.org.
> They probably aren't that eager to add a new site, of which you specify only a start URL, to
> their index. And rightly so. That capability was likely abused widely.
>
> Instead, you should (ask to) put links to your site from other sites relevant to the subject
> of your own site. E.g. the page of your local club, other people's private sites, etc.
> Then, after some time, indexing will occur.
>
> Rob
>
>
>
>
> My friend at google isn’t 100% sure what’s happening
That is very obvious!
Maybe you should consult the google search engine itself, instead of your friend.
Enter this in the Google search bar: site:ampr.org
This returns 17700 results of all kinds of different sites below .ampr.org.
You see the domain names xxxx.ampr.org listed in the results.
So there really is no problem indexing ampr.org sites.
I think what you encounter is the difficulty to add a NEW site to the index, not any generic
problem with .ampr.org.
They probably aren't that eager to add a new site, of which you specify only a start URL, to
their index. And rightly so. That capability was likely abused widely.
Instead, you should (ask to) put links to your site from other sites relevant to the subject
of your own site. E.g. the page of your local club, other people's private sites, etc.
Then, after some time, indexing will occur.
Rob
Brian,
My friend at google isn’t 100% sure what’s happening, just that when the crawler attempts the crawl, it comes back with the same code it gets for a robots.txt or something similar. So to clarify, google claims, something at the “top domain” (their words) has prevented almost all subdomains from being crawled. And whatever that is, it clearly has an affect because I really don’t find any results for ampr.org, except the main site, portal and wiki.
Jann:
I realize there are many other reasons and uses for amprnet. I was just talking about webserving some sort of content out to the internet at large. The issue I’m having affects anyone using 44Net, including those in your examples, that are wanting it discoverable by the world at large.
73
Roger
VA7LBB
> On May 2, 2019, at 19:31, Brian Kantor <Brian(a)bkantor.net> wrote:
>
> Having the robots.txt file at the site AMPR.ORG apply to pages
> served by that host is completely reasonable.
>
> Having the robots.txt file at the site AMPR.ORG apply to pages
> served from an entirely separate site called, e.g., VA7LBB.AMPR.ORG
> is one of the most colossal pieces of internet engineering stupidity
> I have ever encountered.
>
> Yet it would appear that is what the person at Google is telling
> you is happening to your site.
>
>> From Wikipedia:
> "A robots.txt file covers one origin. For websites with multiple
> subdomains, each subdomain must have its own robots.txt file.
> If example.com had a robots.txt file but a.example.com did not,
> the rules that would apply for example.com WOULD NOT APPLY to
> a.example.com. In addition, each protocol and port needs its own
> robots.txt file; http://example.com/robots.txt does not apply
> to pages under http://example.com:8080/ or https://example.com/"
>
> If that's not what their crawlers are doing, their crawlers are broken.
> They broke it, they get to fix it. It's not our problem.
> - Brian
>
>
>
>
>> On Thu, May 02, 2019 at 06:13:40PM -0700, Roger Andrews wrote:
>> Hi,
>> I recently talked with Brian briefly about this and wanted to
>> throw it out to the group. It’s incredibly rare to see any of the
>> tunnels that have been created, represented in a Google search.
>> While I understand and agree that any site that will become a high
>> volume site has no place on Amprnet (we have to share resources)
>> it also seems pointless to create a website that is undiscoverable.
>> After all, isn’t the primary purpose of a website to share it’s
>> content with others. I recently created a website on a 44net gateway
>> and after several weeks, (and even convincing Brian to add a meta
>> TXT entry allowing me to ask google to crawl), I am not seeing any
>> content on Google. I put in a service request to google (not the
>> easiest task) and I was advised that robots.txt or some other
>> prevention device is blocking indexing all the subdirectories on
>> amp.org. I was told that the few gateways that I see in the results
>> were likely crawled before the restriction on ampr.org was applied.
>> I created the website for our ARES group and placed it on an ampr
>> gateway because we don’t have funds, and in reality, see very little
>> traffic. We had a .net site last year and averaged about 50 visitors
>> a month. My question is - is it really necessary to prevent the
>> whole of ampr.org from being crawled (except of course the top
>> domain which does show up). So many ip addresses, but almost none
>> visible seems a real pity.
>> Thanks for listening. My only hope is that this creates a little
>> bit of debate around the issue.
>>
>> 73
>> Roger
>> VA7LBB
>
I don't think there is an issue with Google indexing, they index my/our servers just fine.
As Brian already wrote, a http://ampr.org/robots.txt file should have no effect on that.
Besides (at least in the current version) there is nothing in that file that would prevent such indexing
aside from a /wp-admin/ directory.
What you need for Google indexing is links from other sites, that are already being indexed, that point to you.
And some patience. It is not like a newly crawled site will appear in the index the next day. Or even the next week.
(maybe they do some re-crawling to avoid indexing scams and other malicious sites)
Rob
Hi,
I recently talked with Brian briefly about this and wanted to throw it out to the group. It’s incredibly rare to see any of the tunnels that have been created, represented in a Google search. While I understand and agree that any site that will become a high volume site has no place on Amprnet (we have to share resources) it also seems pointless to create a website that is undiscoverable. After all, isn’t the primary purpose of a website to share it’s content with others. I recently created a website on a 44net gateway and after several weeks, (and even convincing Brian to add a meta TXT entry allowing me to ask google to crawl), I am not seeing any content on Google. I put in a service request to google (not the easiest task) and I was advised that robots.txt or some other prevention device is blocking indexing all the subdirectories on amp.org. I was told that the few gateways that I see in the results were likely crawled before the restriction on ampr.org <http://ampr.org/> was applied. I created the website for our ARES group and placed it on an ampr gateway because we don’t have funds, and in reality, see very little traffic. We had a .net site last year and averaged about 50 visitors a month. My question is - is it really necessary to prevent the whole of ampr.org <http://ampr.org/> from being crawled (except of course the top domain which does show up). So many ip addresses, but almost none visible seems a real pity.
Thanks for listening. My only hope is that this creates a little bit of debate around the issue.
73
Roger
VA7LBB
Hello,
amprd 3.0 is available in the usual locations.
http://yo2tm.ampr.org/hamprojectshttp://www.yo2loj.ro/hamprojects
What's new:
- Added support for AMPR subnets in ignore list
- Added disable/enable option for incoming multicast and broadcast traffic
- Added support for selectable PID file
In regard of the issues with those ICMP unreachable messages:
For those keen ones, I added an optional kernel module, which needs to
be compiled and installed (howto is in the module subfolder).
Also, there is a new global option in the config file to enable its use.
This is a netfilter module which intercepts the IPIP 4 protocol and
converts it to protocol 94 and the other way around, circumventing the
series 4 kernel behavior that triggered the sending of those messages.
So those that want to try it: compile it (you will need to have the
kernel toolset and kernel headers installed), install it and enable the
use_redirector option.
For the rest, the filtering option will still work as expected.
Setails are in the readme files.
Have fun!
Marius, YO2LOJ
I have a question to the sysops running gateways in Europe.
Lately, my subnets (44.182.21.0/24, 44.182.20.0/24 and 44.182.21.253
using gw 89.122.215.236 and yo2loj.go.ro) seem to have been lockout from
different regions like Germany, Poland, Portugal and Bulgaria, while
access from 44.182.21.1/32 still works.
Is this intentional or just a configuration issue?
Marius, YO2LOJ
Subject: VPN
Hello,
Due to some annoying ISP firewalling that I cant control, I'm going to have
to VPN into the network, or set up a vps somewhere and vpn into that.
Ideally the first option since its cheaper :)
The link to www.arrl.org/instructions under this article
http://wiki.ampr.org/wiki/AMPRNet_VPN is invalid however. I'm not sure who
exactly to contact about this, since I cant find an author's name in the
wiki. Any help would be appreciated.
Thanks!
-Nate
