After the Google Penguin, Panda and other sub sequent updates, it seems SEO folks around the world are trying their level best to ensure that their websites are technically perfect. In short, the realization has finally dawn on the SEO world that the days of ruling the web with imperfect and uninteresting website has come to end. Therefore, we need to put more focus on On-Page SEO factors that matter the most. Of all these On Page SEO factors, Website crawlability is certainly the most crucial one.
Just think for a second, if search engine crawlers are unable to craw your website, how on earth your website will be able to rank high on competitive terms. The very concept of ranking high with website that has some serious crawlability issue seems flawed at its best.
Imagine for a second that you have created a web page on a popular product and you are aspiring to rank high in some competitive terms. Now, what if search engines bots fail to crawl that page? If it never gets crawled by search engines, it will never get indexed and therefore, there is no way your web page will appear on the SERP.
Now, it is common for people to make their website uncrawlable unintentionally if they are not accustomed the way search engine works. Sometimes, people use JavaScript for the navigation and if the navigational links is the only way to locate some internal links, your website’s overall visibility could be in for trouble. Most Search engines, including Google, are not that much good at parsing JavaScript or Ajax and that means, if you are using any of these scripts in the main navigation, you are basically making it difficult for Search Engines to crawl and index your website.
So, if search engines can only crawl and index a small section of your website, it might consider the website low quality. Moreover, by making a significant percentage of all the pages of your website uncrawlable, you are just sabotaging the growth of your website on the web. If search engines are unable to crawl website, how can they determine whether your website deserve to rank for a term or not?
Now, sometimes people block search engine bots without realize it –
· Robots.txt – You need to check whether you have blocked any page via robots.txt or not. If you find any tag that begins with “Disallow:/ “, it means the robots.txt is blocking that page or folder from search engines.
· Meta NoIndex – Check the <head></head> section of all the web pages of your website and if you locate a tag like this - <meta name=”robots” content=”noindex, nofollow” />, it means the page is blocked from search engines.
· Make sure that the page is returning 200 OK server side response code. If it is returning something else, there is something definitely wrong with that web page.
So, there is not an iota of doubt that site crawlability has a direct impact on SEO and therefore, we need to go extra mile to ensure that our website is crawler-friendly.