WebDiggin.com: An Adventure to Make Money Online

Apparently everyone needs to think about Search Engine Optimization and PageRank. For the longest while, I was quite happy just blogging my thoughts away without thinking about SEO, PR, and all those other acronyms. Apparently, I’m wrong.

Josh Spaulding’s post on robots.txt showed me the light. Here’s what I learned about robots.txt from Josh’s post and by looking at his robots.txt file:

• Use it to remove pages from the search engines so that they can concentrate on your content (i.e. remove i.e. your wp-admin page, your contact-us page, etc)
• The robots.txt file can be read by anyone so the pages that you are trying to hide from the search engines can be seen by humans. (i.e. don’t hide your ebook link in a robots.txt file)
• Be careful. If you make a mistake, it might tell search engines to avoid pages you want crawled. (But you can see what Google sees in your Google Webmaster Tools account)

Josh’s post pointed us to some information from Matt Cutts, software engineer at Google. Matt was interviewed by Eric Enge at StoneTemple Consulting and said the following:

Now, robots.txt says you are not allowed to crawl a page, and Google therefore does not crawl pages that are forbidden in robots.txt. However, they can accrue PageRank, and they can be returned in our search results… we wouldn’t crawl you from robots.txt, but we could return that URL reference that we saw. [Based on the links from other sites to those pages]… we would return the un-crawled reference.

Here’s what I learned from Matt’s interview:

• You can use robots.txt to hide certain pages from Google, but they can still turn up in the search results. (Google can use other links to that page to get information, even if they don’t go there themselves.)
• You can use a meta tag (NoIndex) on any page to keep Google from showing it in the search results. (Google won’t return a NoIndex page in a search result, but Yahoo and the others might do it differently.)
• A NoIndex page can accumulate PageRank, can still build up and pass on PageRank (Google will still follow links in and out of a noindex page. It just won’t show that page in the results. Matt’s example was to noindex a login page. A login page has very little content, but links out to lots of good places and receives lots of good links.)
• A meta tag (NoFollow) on a page will keep Google from following outgoing links and so that page will keep the PageRank juice.
• An anchor tag (<a rel=”NoFollow” href=”site.com” mce_href=”site.com”>) will keep Google from sending PageRank juice to that link.

Is any of this important for beginning websites? I don’t know.

Matt said these were things to “sculpt where you want your PageRank to flow, or where you want Googlebot to spend more time and attention.” We have a PageRank of 0. Does it really matter? (As long as we turn off the pagerank leakage.)

Bottom Line: We looked at Josh’s robots.txt post, as well as at ShoeMoney’s robot.txt post to figure out what we want our robots.txt file to look like.

Question: Have you changed your robots.txt file to build pagerank by persuading the search engines to hang out in specific areas of your site?

(No Ratings Yet)

 Loading ...