When you hear the words “search engine optimization,” what do you think of? My mind leaps straight to a list of SEO ranking factors, such as proper tags, relevant keywords, a clean sitemap, great design elements, and a steady stream of high-quality content.
However, a recent article by my colleague, Yauhen Khutarniuk, made me realize that I should be adding “crawl budget” to my list. While many SEO experts overlook crawl budget because it’s not very well understood, Khutarniuk brings some compelling evidence to the table – which I’ll come back to later in this post – that crawl budget can, and should, be optimized.
This made me wonder: how does crawl budget optimization overlap with SEO, and what can websites do to improve their crawl rate?
First Things First – What Is a Crawl Budget?
Web services and search engines use web crawler bots, aka “spiders,” to crawl web pages, collect information about them, and add them to their index. These spiders also detect links on the pages they visit and attempt to crawl these new pages too.
Examples of bots that you’re probably familiar with include Googlebot, which discovers new pages and adds them to the Google Index, or Bingbot, Microsoft’s equivalent. Most SEO tools and other web services also rely on spiders to gather information. For example, my company’s backlink index, WebMeUp, is built using a spider called BLEXBot, which crawls up to 15 billion pages daily gathering backlink data.
The number of times a search engine spider crawls your website in a given time allotment is what we call your “crawl budget.” So if Googlebot hits your site 32 times per day, we can say that your typical Google crawl budget is approximately 960 per month.
You can use tools such as Google Search Console and Bing Webmaster Tools to figure out your website’s approximate crawl budget. Just log in to Crawl > Crawl Stats to see the average number of pages crawled per day.
Web services and search engines use web crawler bots, aka “spiders,” to crawl web pages, collect information about them, and add them to their index. These spiders also detect links on the pages they visit and attempt to crawl these new pages too.
Examples of bots that you’re probably familiar with include Googlebot, which discovers new pages and adds them to the Google Index, or Bingbot, Microsoft’s equivalent. Most SEO tools and other web services also rely on spiders to gather information. For example, my company’s backlink index, WebMeUp, is built using a spider called BLEXBot, which crawls up to 15 billion pages daily gathering backlink data.
The number of times a search engine spider crawls your website in a given time allotment is what we call your “crawl budget.” So if Googlebot hits your site 32 times per day, we can say that your typical Google crawl budget is approximately 960 per month.
You can use tools such as Google Search Console and Bing Webmaster Tools to figure out your website’s approximate crawl budget. Just log in to Crawl > Crawl Stats to see the average number of pages crawled per day.
View source to continue reading https://www.searchenginejournal.com/9-tips-optimize-crawl-budget-seo/176009/
There is so much in this article that I would never have thought of on my own. Your content gives readers things to think about in an interesting way. Thank you for your clear information. buy backlinks
ReplyDeletethanks for this usefull article, waiting for this article like this again. buy backlinks to your site
ReplyDeleteI have read your blog it is very helpful for me. I want to say thanks to you. I have bookmark your site for future updates 토토사이트
ReplyDeleteHope you will give more information on this topics in your next articles 메이저놀이터
ReplyDeleteYou have a real ability for writing unique content. I like how you think and the way you represent your views in this article. I agree with your way of thinking. Thank you for sharing. www.marketing1on1.com/link-building-services
ReplyDeleteFantastic blog you have here. You’ll discover me looking at your stuff often. Saved! link building
ReplyDelete