Page 1 of 1

GoogleBot - (Automatically?) running VRS Reports?

Posted: Thu Feb 12, 2015 3:11 pm
by SkierInAvon
General Awareness....

Just caught this - this morning (12FEB2015)...
As we all know...Google has been indexing the Web and Web sites for many years...

Today - I saw the GoogleBot - no only indexing my VRS web site - BUT also (RUNNING REPORTS??!!!)

The reports I saw the GoogleBot running....

/DateReport.htm
/IcaoReport.htm
/ReportRows.json

The VRS Reports being run were being from from Goggle at (66.249.69.12 66.249.69.233 66.249.69.248) respectively.
Just run a simple (Trace Route) back to those IP address and you'll see it's Google...

Not sure (how?) Google is actually running reports (filling out a web form?) in addition to Google's normal indexing of web pages... Curious

Your comments welcome.
Yes, I know I can lock down my VRS web site using passwords required for access.

I didn't know just how cleaver Google was getting.

Your comments welcome.

-pete :shock:

Re: GoogleBot - (Automatically?) running VRS Reports?

Posted: Sat Feb 14, 2015 5:57 pm
by agw
Those reports are links in the site's HTML, it's just following those links. I think this may be a side-effect for sites in the directory on the VRS web site, Google spiders the VRS web site fairly often and it will follow the links from the directory page to your site, and from there start following links to the reports.

You can tell Google what it can and cannot spider, I'll configure the web site to tell it not to follow the links on the directory page. I'll add a robots.txt to VRS as well to stop search engine following links to the reports.

Re: GoogleBot - (Automatically?) running VRS Reports?

Posted: Fri Jun 26, 2015 8:12 am
by Meghna
Blocking Googleboat is the best solution. Here is the code for Robots.txt....

user-agent:Googleboat
Disallow: /DateReport.htm
Disallow: /IcaoReport.htm
Disallow: /ReportRows.json