In recent weeks, crawl budget and indexing questions and issues have come up more and more with prospects and clients. Crawl budget and indexing issues were once more a big site issue, but now something smaller businesses need to be conscious of. As we continue to grow good content for all of our sites, we find that these more technical issues come into play a bit more. Today I just wanted to give a quick overview of what this means and why it matters.
The search engine index is basically their file of all the URL’s or web pages they are aware of. Search engines like Google “crawl” your site with bots and basically take a snapshot to store in their index. There was a time over ten years ago when I started, that we would often have to wait a month or more for Google to come and find changes to a website. These days, Google’s “bots”, as they are often called, come around and look at sites much more frequently, but it is important that you make the right pages as indexable as possible. If Google has trouble finding or reading a page on your site, it reduces the chance that it will save it and use it in the index, as well as find more pages that are connected to it.
Google and other search engines basically will only spend so much time on your site taking snapshots and looking around. Anything that makes this difficult on the bot, will make less of your site indexable, and therefore important pages you want to rank, may not get the chance to. The important thing to remember is that if Google had not indexed your page, it has no chance of ranking on Google. It is like it doesn’t exist in an SEO sense. In the past, many small businesses didn’t have to worry about this because they had sites with few actual pages and they were easily indexed. As we build on blog posts and categories and portfolios, the URL count goes up, and the indexing starts to get more complicated.
Which leads us to crawl budget. This is basically the amount of time or pages Google will allow your particular site. As mentioned above, if Google comes and visits your site and wastes a lot of the time you have on pages that are difficult to index or aren’t meant to rank, it reduces the time left to hit the important pages. Because of this, it is often a good idea to “NoIndex” pages that aren’t necessarily meant to rank and mostly serve just as filler or helpful tools for the user or for organizing things. For example, I had a company a year or so ago that had accidentally created author pages and image pages for every picture on their site. Not just the image, but a separate page for each image. They dropped in the search rankings. Google was finding these pages and using their crawl budget on them and then not indexing important pages that they had worked hard on to rank. We needed to go in and sculpt their indexable pages and make sure big sections of their site that were eating up their crawl budget were “NoIndexed” and therefore, the budget could be used on the important content pages.
Most SEO plugins like All in One or Yoast have options on each page to NOINDEX them and at a higher settings level. You may want to NOINDEX your portfolio or author pages to avoid wasting crawl budget. You also should be careful with your blog post categories and tags. Often these are very similar and may be duplicative. In most cases, both the tags and categories don’t need to be indexed. In many cases, we make too many different categories and tags. More thought should be made into creating helpful tags and categories that help users break down your blog content. If you find you have a lot of one post categories or tags, that can be wasteful of crawl budget and is also a bad user experience.
Indexing and Crawl Budget Get Complicated
Sometimes these issues get a bit more complicated than the average SEO DIYer can handle. We can step in and help sites with crawl issues. Often they don’t even know they have them but just wonder why rankings dropped or they can’t seem to rank. Sometimes it is as simple as blocking off a wasteful section of your site or organizing a bit better. Give us a call if you need help with your indexing issues or even if you aren’t sure. It may be worth digging and knowing for sure.