|
|
Optimizing Your Website URLs For Crawling And Indexing
By Courtney Mills
Expert Author
Article Date: 2009-08-19
Of all the millions of questions Google is asked each year by search marketers and webmaster, they all seem to relate back to the same question: How easy is it for search engine to crawl your site? To help you out with this, Google have shared some of their key takeaways on the topic of crawling and indexing your website URLs.
Google's Webmaster Trends Analyst, Susan Moskwa, describes the relationship between website URL and the search engine crawler as like a bridge. Here is more on the analogy:
URLs are like the bridges between your website and a search engine's crawler: crawlers need to be able to find and cross those bridges in order to get to your site's content. If your URLs are complicated or redundant, crawlers are going to spend time tracing and retracing their steps; if your URLs are organized and lead directly to distinct content, crawlers can spend their time accessing your content rather than crawling through empty pages, or crawling the same content over and over via different URLs.
In the video here, Susan also shares some best practice tips for ensuring your URLs are fully optimized to be crawled and indexed by the Google crawler.
There are a few things that Susan recommends you follow when ensuring your URLs are set-up correctly, which will help the crawlers find and crawl your content faster. These include:
- Remove user-specific details from URLs.
URL parameters that don't change the content of the page-like session IDs or sort order-can be removed from the URL and put into a cookie.
- Disallow actions Googlebot can't perform.
Using your robots.txt file, you can disallow crawling of login pages, contact forms, shopping carts, and other pages whose sole functionality is something that a crawler can't perform.
- One URL, one set of content
Click here for additional recommendations from the Google Webmaster Central Blog.
Have you got a tip to make is easier for the Google crawler to crawl your website URLs? If so, we would love to share them with our readers, simply leave your suggestion in the comments section below.
Comments
About the Author:
Courtney is an online marketing specialist at ineedhits - a leading search engine marketing firm with over 11 years experience. Courtney has been living and breathing marketing for over 2 years and specializes in web and permission email marketing and providing news and opinion to online marketing communities.
|
|
