Handy Robots.txt Analysis Tool
This is going to be one of those “quick plug for a utility I like” posts, nothing too substantial.
If you’re like me and forget the syntax used for generating Robots.txt exclusions this is a handy website to use: http://phpweby.com/services/robots , simply plugin a URL and it will tell you if it’s excluded or not per your Robots.txt file. So now you won’t wonder “Does Google think this page should be indexed?”
That’s all! …. oh, and I have to throw in the usual disclaimer of “there is no obligation for search engines to obey your robots.txt file”