Now that I know where to look.. PMTU has caused me a lot of headache lately. I believe it could be the problem. Sending large packets to 44.135.179.28 yields no reply. tracepath does send back need to frag, but when TTL expires at amprgw.ucsd.edu. I believe amprgw.ucsd.edu should send back need-to-frag for higher TTLs as well.
That is always a bit tricky, often those packets *are* sent back but they are blocked somewhere closer to the client, and/or the TCP stack of the system does not process them in a reasonable way.
It is possible to work around that by adjusting the MSS of a TCP SYN passing the point where outgoing MTU is smaller than incoming MTU (incidentally something that I invented and implemented in NET in 1995, but later almost any router and routing software started to support it) so as a result the TCP segments sent by the endpoints will be smaller and won't need to be fragmented.
Roger can do that on his own server, e.g. like this:
iptables -t mangle -A INPUT -p tcp --syn -j TCPMSS --set-mss 1400 iptables -t mangle -A OUTPUT -p tcp --syn -j TCPMSS --set-mss 1400
Or on a router/gateway along the path (using FORWARD instead of INPUT/OUTPUT).
However, I'm not convinced that this is the problem as the site works OK for me over internet. Why wouldn't it work for Google then?
Rob
However, I'm not convinced that this is the problem as the site works OK for me over internet. Why wouldn't it work for Google then?
We have to speculate somewhat..I don't know what the crawler uses for TCP stack. I know each OS is different in the ways it deals with MTU and blackholes. I did a wireshark capture on my Windows desktop and show a lot of black/red. I lowered my MTU by 40 and re-tried and there was lots of green. The TCP MSS coming from the web server was already lowered by 20. I don't know what it looks like on the far end. I'm also speculating that it's not getting ICMP or seeing a lower MSS for me.
Another reason I saw red -- a few links point to HTTPS and it is not enabled. Fix that, and set a lower MTU on the host, and we'll at least fix *some* problem. Maybe not the Googling indexing one. :)
Regards, Scott