Sunday, June 12, 2005

Google not crawling urls

In the beginning of May, I wrote about how I bought a new smoke alarm, and the problems that I had with it.  Shortly after that, Google change the ads in my AdSense section to be related to smoke alarms, which I think is appropriate.  But then after I posted a number of other articles, that post was no longer visible on the home page, but the ads for the smoke alarms still are visible.

It appears that Google is not crawling all of the possible urls for determining which ads should be shown.  For example, still shows the smoke alarm ads, while (which is a link to the same page) does not.

Periodically, after I see an access to one of these pages, I see an access from a Google server with the http client string "Mediapartners-Google/2.1", but only from  and not  It seems that they changed something in the past month.