9 May
2019
9 May
'19
6:35 a.m.
However, I'm not convinced that this is the
problem as the site works OK
for me over internet. Why wouldn't it work for Google then?
We have to speculate somewhat..I don't know what the crawler uses for
TCP stack. I know each OS is different in the ways it deals with MTU
and blackholes.
I did a wireshark capture on my Windows desktop and show a lot of
black/red. I lowered my MTU by 40 and re-tried and there was lots of
green.
The TCP MSS coming from the web server was already lowered by 20. I
don't know what it looks like on the far end. I'm also speculating
that it's not getting ICMP or seeing a lower MSS for me.
Another reason I saw red -- a few links point to HTTPS and it is not
enabled. Fix that, and set a lower MTU on the host, and we'll at
least fix *some* problem. Maybe not the Googling indexing one. :)
Regards,
Scott