{"componentChunkName":"component---src-templates-articles-index-jsx","path":"/articles/why-google-bots-refuse-to-crawl-your-site/","result":{"data":{"props":{"adsense":true,"adsenseMobile":null,"byLine":"","canonical":"https://staging.business.com/articles/why-google-bots-refuse-to-crawl-your-site/","content":"\n

Over 5 billion Google searches are made each day! Everyone's on Google to find answers to their queries and discover new things. In fact, Google is rated as the most popular website in both the global and U.S. markets. If your business isn't featured on Google's search engine result page (SERP), you are doomed!

\r\n

Why are Google bots important?

\r\n

Crawling. Indexing. Ranking. These are the three basic steps used by Google's search engine automated robots (also called crawlers or spiders) to generate results on the SERP. If your website is unfriendly to these crawlers, you stand no chance of attracting organic traffic to your site.

\r\n

So, how can you make Google bots find and crawl your site? First things first, know where you stand. Conduct a thorough SEO audit of your site to gauge its onsite, offsite and technical SEO performance. Second, determine how many pages are indexed. Simply type \"site:yoursite.com\" into the Google search bar. If the number of results is drastically lower than the actual number of pages on your site, Google is not crawling all the pages on your site and you need to do something about it. 

\r\n

Six reasons why Google bots aren't crawling your site

\r\n

Without further ado, let's understand what makes a website crawler-unfriendly and what webmasters can do about it.

\r\n

1. You have blocked Google bots.

\r\n

Is Google not indexing your entire website? In this case, the first thing you need to check is your robots.txt file. Look for code snippets that disallow the bots from crawling any page on your site and simply remove such code.

\r\n

Further, check for a crawl block in the robots.txt file using the URL inspection tool in Google Search Console. If you see an error saying that the crawl is blocked by robots.txt, get rid of it to help Google bots crawl and index your page.

\r\n

At times, it takes more than a week for Google to crawl a new website. In such cases, it is wise to open a Google Search Console account and point Google to your sitemap URL. If your site doesn't have a sitemap, create one now.

\r\n

Another way of barring search indexing from your website is by having the \"noindex\" meta tag. If you see the following code in the meta tag, get rid of it to allow Google to index your site.

\r\n\n

2. You haven't created a Google Console/Analytics account yet.

\r\n

Google Analytics is a free web analytics tool that collects and organizes traffic data into customizable reports and Google Search Console offers webmasters in-depth information on how Google sees a website.

\r\n

Manually activating these Google services will send a signal to the Google bots that you are seriously working towards building your web presence. In fact, Search Console can help you gauge the health of your website and fix issues that are stopping your pages from getting indexed.

\r\n

For instance, if you have a new page on your site, it's quite possible that Google hasn't got a chance to crawl it yet. The URL inspection tool in GSC can help you find out whether or not the page is indexed and offer you a complete report. So, say hello to Google by setting up a Search Console account and visit it regularly to see how your site performs in the SERP.

\r\n

Another point to bear in mind is that the old Google Search Console allowed webmasters to have Google test, render, crawl and index any URL using the \"Fetch as Google\"' tool. Though this feature doesn't exist in the new version, you can still ask Google to index your webpages.

\r\n

3. Your website has a poor internal linking profile.

\r\n

Internal links are key to helping Google find, understand and index your webpages. They enable users to easily navigate a site, establish information hierarchy and spread link equity through the site. For instance, according to Moz, the optimal link structure for a website should look like a pyramid, with your homepage at the top of the structure.

\r\n

Most e-commerce sites, including Amazon, use this structure and add internal links from their most authoritative pages. Google will recrawl such powerful pages, enabling it to find the internal link and index the respective page. You can find the most authoritative pages on your website using tools like Google Analytics and Ahrefs Site Explorer.

\r\n

Finally, Google bots do not crawl links with the rel=\"nofollow\" tag. Nofollow internal links cause Google bots to ignore the link. Hence, it's important to remove the nofollow tag from the internal links unless they point to an unimportant page that you want to exclude from the search engine's index.

\r\n

4. Google doesn't like your URL structure.

\r\n

Google advises webmasters to keep URL structures simple and readable. Hence, you should avoid using long, complex IDs that can cause problems for crawlers. According to Google, such complex URLs contain multiple parameters and create unnecessarily high numbers of URLs that point to identical content on your site. This will cause Google bots to consume more bandwidth to crawl the webpage or to not crawl the page at all.

\r\n

Wherever possible, have a clean URL taxonomy that bots can understand. Further, use the robots.txt file to block the bot's access to problematic URLs if there are any.

\r\n

Permalinks are URLs that help link your content on your web, enabling Google to find the page with ease. Google likes short URLs that clearly state the title or important keywords.

\r\n

WordPress, by default, creates weird permalinks or URL structures that may contain day, date, month or post IDs. These aren't preferred by Google. If your site is hosted by WordPress, use the \"Post name\" structure in the Permalink Settings on the WordPress dashboard.

\r\n

5. Google has temporarily removed your site from its index.

\r\n

If your website failed to meet Google's quality guidelines or has a shady history, the search engine may deindex, penalize or remove your site from the search results.

\r\n\n

Generally, Google alerts webmasters when their websites violate quality guidelines. In such cases, it is advisable to modify the site and ask Google to review the site after the issues are fixed.

\r\n

6. You haven't optimized for Google bots.

\r\n

Optimizing your website for Google bots isn't the same as search engine optimization. Once you submit your website to the search engine, Google bots crawl the pages for content. These spiders scan your site for meta content, keyword saturation, relevant content and certain other factors. Therefore, it is important to optimize your sire for such scans.

\r\n

Build a site that's indexable and offers relevant information to Google bots. Pay attention to the technical ranking factors to improve your site's crawler experience. Here are a few parameters that you shouldn't ignore.

\r\n\n

Regardless of how many backlinks they have or what high-quality content they share, crawler-unfriendly sites do not exist in the eyes of Google. If your website or webpages have crawlability issues, Google bots will not be able to discover or index them, causing you to lose your online ranking. The information shared in this post will help you identify why Google bots aren't crawling your site, allowing you to take the necessary corrective measures.

\n","docType":"article","guid":"articles_d89e815864a3b80ecd79bec1720cc1bb","lastModified":"2019-11-20T16:00:00.000Z","published":"2019-11-20T16:00:00.000Z","structuredData":null,"subtitle":"If your site isn't generating traffic, Google bots may not be crawling your webpages.","template":"v1","title":"Why Google Bots Refuse to Crawl Your Site","type":"article","author":{"avatar":null,"bio":"His most recent accomplishment was initiating and scaling Shopify’s very first international SEO effort. He has been successful in helping Shopify achieve organic growth in over 25 international markets. Having graduated from the John Molson School of Business at Concordia University, he developed a keen interest in digital marketing.\r\n\r\nKevin gained most of his experience in some of the most competitive fields in digital marketing, such as ecommerce and video streaming. He enjoys writing on a variety of topics that range from SEO, technical & International SEO to content marketing.","name":"Kevin Reis","image":null,"slug":null,"social":{"facebook":"","linkedin":"https://www.linkedin.com/in/kevin-reis-09642ba5/","twitter":null,"website":"https://shopify.com"},"status":"enabled","title":null,"type":"New BDC Member","user":{"imagePath":"https://staging.business.com/images/uploads/user/profile/4c46b48cbfeb521038b2c1a27757dc0d17f1f.jpg","profile":{"company":"Shopify","jobTitle":"SEO Specialist","website":"https://shopify.com"}}},"breadcrumbs":null,"channels":{"primary":{"name":"Marketing","slug":"marketing"},"sub":{"name":"Marketing Strategy","slug":"marketing-strategy"}},"meta":{"robots":"index, follow","description":"Having trouble driving traffic to your site? Google bots may not be crawling your webpages. Check out six reasons why your site might not be crawlable.","title":"Why Google Bots Aren't Crawling Your Site - business.com"},"parsedContent":{"toc":[{"id":"why-are-google-bots-important","title":"Why are Google bots important?"},{"id":"six-reasons-why-google-bots-arent-crawling-your-site","title":"Six reasons why Google bots aren't crawling your site"}],"json":[{"component":"p","children":["Over ",{"component":"a","props":{"href":"https://hostingfacts.com/internet-facts-stats/","target":"_blank","rel":"nofollow"},"children":["5 billion Google searches"]}," are made each day! Everyone's on Google to find answers to their queries and discover new things. In fact, Google is rated as ",{"component":"a","props":{"href":"https://www.lifewire.com/most-popular-sites-3483140","target":"_blank","rel":"nofollow"},"children":["the most popular website"]}," in both the global and U.S. markets. If your business isn't featured on Google's search engine result page (SERP), you are doomed!"]},{"component":"h2","props":{"id":"why-are-google-bots-important"},"children":["Why are Google bots important?"]},{"component":"p","children":["Crawling. Indexing. Ranking. These are the three basic steps used by Google's search engine automated robots (also called crawlers or spiders) to generate results on the SERP. If your website is unfriendly to these crawlers, you stand no chance of attracting organic traffic to your site."]},{"component":"p","children":["So, how can you make Google bots find and crawl your site? First things first, know where you stand. Conduct a thorough SEO audit of your site to gauge its onsite, offsite and technical SEO performance. Second, determine how many pages are indexed. Simply type \"site:yoursite.com\" into the Google search bar. If the number of results is drastically lower than the actual number of pages on your site, Google is not crawling all the pages on your site and you need to do something about it. "]},{"component":"h2","props":{"id":"six-reasons-why-google-bots-arent-crawling-your-site"},"children":["Six reasons why Google bots aren't crawling your site"]},{"component":"p","children":["Without further ado, let's understand what makes a website crawler-unfriendly and what webmasters can do about it.",{"component":"strong"}]},{"component":"h3","children":["1. You have blocked Google bots."]},{"component":"p","children":["Is Google not indexing your entire website? In this case, the first thing you need to check is your robots.txt file. Look for ",{"component":"a","props":{"href":"https://www.quora.com/Is-it-better-to-use-robot-meta-tags-or-the-robots-txt-file-for-informing-crawlers-to-include-or-exclude-a-page","target":"_blank","rel":"nofollow"},"children":["code snippets that disallow the bots"]}," from crawling any page on your site and simply remove such code."]},{"component":"p","children":["Further, check for a crawl block in the robots.txt file using the URL inspection tool in Google Search Console. If you see an error saying that the crawl is blocked by robots.txt, get rid of it to help Google bots crawl and index your page."]},{"component":"p","children":["At times, it takes more than a week for Google to crawl a new website. In such cases, it is wise to open a Google Search Console account and ",{"component":"a","props":{"href":"https://support.google.com/webmasters/answer/156184","target":"_blank","rel":"nofollow"},"children":["point Google to your sitemap"]}," URL. If your site doesn't have a sitemap, ",{"component":"a","props":{"href":"https://www.xml-sitemaps.com/","target":"_blank","rel":"nofollow"},"children":["create one now"]},"."]},{"component":"p","children":["Another way of barring search indexing from your website is by having the \"",{"component":"a","props":{"href":"https://blog.hubspot.com/marketing/how-to-unindex-pages-from-search-engines","target":"_blank","rel":"nofollow"},"children":["noindex"]},"\" meta tag. If you see the following code in the meta tag, get rid of it to allow Google to index your site."]},{"component":"ul","children":[{"component":"li","children":[""]}]},{"component":"h3","children":["2. You haven't created a Google Console/Analytics account yet."]},{"component":"p","children":["Google Analytics is a free web analytics tool that collects and organizes traffic data into customizable reports and Google Search Console offers webmasters in-depth information on how Google sees a website."]},{"component":"p","children":["Manually activating these Google services will send a signal to the Google bots that you are seriously working towards building your web presence. In fact, Search Console can help you gauge the health of your website and fix issues that are stopping your pages from getting indexed."]},{"component":"p","children":["For instance, if you have a new page on your site, it's quite possible that Google hasn't got a chance to crawl it yet. The ",{"component":"a","props":{"href":"https://ww.9to5google.com/2018/06/25/google-search-console-url-inspection/","target":"_blank","rel":"nofollow"},"children":["URL inspection tool"]}," in GSC can help you find out whether or not the page is indexed and offer you a complete report. So, say hello to Google by ",{"component":"a","props":{"href":"https://search.google.com/search-console/about","target":"_blank","rel":"nofollow"},"children":["setting up a Search Console account"]}," and visit it regularly to see how your site performs in the SERP."]},{"component":"p","children":["Another point to bear in mind is that the old Google Search Console allowed webmasters to have Google test, render, crawl and index any URL using the \"Fetch as Google\"' tool. Though this feature doesn't exist in the new version, you can still ",{"component":"a","props":{"href":"https://www.youtube.com/watch?v=MlPh_I71hds","target":"_blank","rel":"nofollow"},"children":["ask Google to index your webpages"]},"."]},{"component":"h3","children":["3. Your website has a poor internal linking profile."]},{"component":"p","children":["Internal links are key to helping Google find, understand and index your webpages. They enable users to easily navigate a site, establish information hierarchy and spread link equity through the site. For instance, according to ",{"component":"a","props":{"href":"https://moz.com/learn/seo/internal-link","target":"_blank","rel":"nofollow"},"children":["Moz"]},", the optimal link structure for a website should look like a pyramid, with your homepage at the top of the structure."]},{"component":"p","children":["Most e-commerce sites, including Amazon, use this structure and add internal links from their most authoritative pages. Google will recrawl such powerful pages, enabling it to find the internal link and index the respective page. You can find the most authoritative pages on your website using tools like Google Analytics and Ahrefs Site Explorer."]},{"component":"p","children":["Finally, Google bots ",{"component":"a","props":{"href":"https://support.google.com/webmasters/answer/96569?hl=en","target":"_blank","rel":"nofollow"},"children":["do not crawl links with the rel=\"nofollow\" tag"]},". Nofollow internal links cause Google bots to ignore the link. Hence, it's important to remove the nofollow tag from the internal links unless they point to an unimportant page that you want to exclude from the search engine's index."]},{"component":"h3","children":["4. Google doesn't like your URL structure."]},{"component":"p","children":["Google advises webmasters to ",{"component":"a","props":{"href":"https://support.google.com/webmasters/answer/76329?hl=en","target":"_blank","rel":"nofollow"},"children":["keep URL structures simple and readable"]},". Hence, you should avoid using long, complex IDs that can cause problems for crawlers. According to Google, such complex URLs contain multiple parameters and create unnecessarily high numbers of URLs that point to identical content on your site. This will cause Google bots to consume more bandwidth to crawl the webpage or to not crawl the page at all."]},{"component":"p","children":["Wherever possible, have a ",{"component":"a","props":{"href":"https://codeable.io/get-your-custom-taxonomy-urls-in-order/","target":"_blank","rel":"nofollow"},"children":["clean URL taxonomy"]}," that bots can understand. Further, use the robots.txt file to block the bot's access to problematic URLs if there are any."]},{"component":"p","children":["Permalinks are URLs that help link your content on your web, enabling Google to find the page with ease. Google likes short URLs that clearly state the title or important keywords."]},{"component":"p","children":["WordPress, by default, creates weird permalinks or URL structures that may contain day, date, month or post IDs. These aren't preferred by Google. If your site is hosted by WordPress, ",{"component":"a","props":{"href":"https://www.acmethemes.com/blog/settings-permalink-wordpress-sites/","target":"_blank","rel":"nofollow"},"children":["use the \"Post name\" structure in the Permalink Settings"]}," on the WordPress dashboard."]},{"component":"h3","children":["5. Google has temporarily removed your site from its index."]},{"component":"p","children":["If your website failed to meet Google's quality guidelines or has a shady history, the search engine may deindex, penalize or remove your site from the search results."]},{"component":"ul","children":[{"component":"li","children":[{"component":"strong","children":["Deindexed or banned: "]},"If a website is completely removed from the Google search page, it is deindexed."]},{"component":"li","children":[{"component":"strong","children":["Penalized:"]}," At times, a lurking manual penalty may prevent your site from getting indexed. If your website or a page still exists but cannot be found in the search results, Google has penalized your site. This penalty can be enforced by Google's algorithm or manually applied by Google's quality engineer."]},{"component":"li","children":[{"component":"strong","children":["Sandboxed:"]}," ",{"component":"a","props":{"href":"https://ahrefs.com/blog/google-sandbox/","target":"_blank","rel":"nofollow"},"children":["Google Sandbox"]}," is an alleged filter that prevents new websites from ranking high. If the traffic for your new site or page dropped suddenly and it wasn't deindexed or penalized, Google has sandboxed your site."]}]},{"component":"p","children":["Generally, Google alerts webmasters when their websites violate quality guidelines. In such cases, it is advisable to modify the site and ",{"component":"a","props":{"href":"https://support.google.com/webmasters/answer/35843","target":"_blank","rel":"nofollow"},"children":["ask Google to review the site"]}," after the issues are fixed."]},{"component":"h3","children":["6. You haven't optimized for Google bots."]},{"component":"p","children":["Optimizing your website for Google bots isn't the same as search engine optimization. Once you submit your website to the search engine, Google bots crawl the pages for content. These spiders scan your site for meta content, keyword saturation, relevant content and certain other factors. Therefore, it is important to optimize your sire for such scans."]},{"component":"p","children":["Build a site that's indexable and offers relevant information to Google bots. Pay attention to the ",{"component":"a","props":{"href":"https://www.searchenginejournal.com/google-200-ranking-factors-facts/265085/","target":"_blank","rel":"nofollow"},"children":["technical ranking factors"]}," to improve your site's crawler experience. Here are a few parameters that you shouldn't ignore."]},{"component":"ul","children":[{"component":"li","children":[{"component":"strong","children":["Good-quality content: "]},"Create relevant and high-quality content for your audience. Google's algorithm awards sites that offer original and relevant content with a higher ranking than those who use fillers or share duplicate content. Though canonicalizing pages makes sense, do so wisely. Canonicalization, when not done carefully, can confuse Google's spiders, making it tough for them to crawl and index your site. "]},{"component":"li","children":[{"component":"strong","children":["Easy navigation: "]},"Make sure your website has a navigation bar that links to all major pages on your website."]},{"component":"li","children":[{"component":"strong","children":["Text-to-HTML ratio: "]},"Since Google's bots read text, make sure your website has a high ",{"component":"a","props":{"href":"https://www.woorank.com/en/blog/are-text-to-html-ratios-important","target":"_blank","rel":"nofollow"},"children":["text-to-HTML ratio"]}," (ideally, between 25% and 70%) in favor of text. Further, minimize JavaScript or ",{"component":"a","props":{"href":"https://www.kirupa.com/html5/running_your_code_at_the_right_time.htm","target":"_blank","rel":"nofollow"},"children":["make sure it loads after HTML"]},", as bots get signals from the text in the HTML code. "]},{"component":"li","children":[{"component":"strong","children":["Site speed: "]},"Your site's loading time is an important ranking factor that Google bots consider when indexing your site. Make sure you test your site's speed and take the necessary measures to improve its loading time. "]},{"component":"li","children":[{"component":"strong","children":["Structured data: "]},{"component":"a","props":{"href":"https://neilpatel.com/blog/structured-data/","target":"_blank","rel":"nofollow"},"children":["Schema markup"]}," or structured data gives context to your website, allowing Google's spiders to make the meaning of the content and index the pages with ease. Boost your site's SEO by using schema markup.  "]}]},{"component":"p","children":["Regardless of how many backlinks they have or what high-quality content they share, crawler-unfriendly sites do not exist in the eyes of Google. If your website or webpages have crawlability issues, Google bots will not be able to discover or index them, causing you to lose your online ranking. The information shared in this post will help you identify why Google bots aren't crawling your site, allowing you to take the necessary corrective measures."]}]},"resource":null,"sponsorship":null,"thumbnail":{"path":"https://staging.business.com/images/content/5dd/46aec7b437465148b456f/1500-0-","caption":"fizkes / Getty Images","isBlurred":false},"bestPicks":null},"related":{"relatedArticles":[{"title":"The Best SEO Strategies for Your WordPress Site","canonical":"https://www.business.com/articles/wordpress-seo-strategies/","slug":"wordpress-seo-strategies","thumbnail":"https://www.business.com/images/content/5dc/f007b5a215e7f2f8b4573/1500-0-","dek":"WordPress is one of the most popular and easy-to-use website platforms out there. But do you know how to optimize your WordPress site? Explore different strategies you can use to improve WordPress SEO and help your site grow.","lastModified":"2019-11-18T16:00:00Z","published":"2019-11-18T16:00:00Z","sponsored":false},{"title":"How to Leverage SEO to Expand Your Small Business's Audience","canonical":"https://www.business.com/articles/seo-for-small-business/","slug":"seo-for-small-business","thumbnail":"https://www.business.com/images/content/5e7/27a337b4374f8638b4570/1500-0-","dek":"Search engine optimization helps get your content in front of more people. Here's how to use SEO to your small business's benefit.","lastModified":"2020-03-18T22:45:00Z","published":"2020-03-18T22:44:00Z","sponsored":false},{"title":"Does Your Site Drive Success? 10 Signs You've Got a Winning Website","canonical":"https://www.business.com/articles/10-visible-signs-youve-got-a-winning-website/","slug":"10-visible-signs-youve-got-a-winning-website","thumbnail":"https://www.business.com/images/channel/marketing.jpg","dek":"These days online marketing has a lot of moving parts but your website is still central to your online success. Is yours up to scratch?","lastModified":"2016-09-27T22:00:00Z","published":"2016-09-27T22:00:00Z","sponsored":false},{"title":"How to Create an Effective Survey","canonical":"https://www.business.com/articles/build-a-better-survey/","slug":"build-a-better-survey","thumbnail":"https://www.business.com/images/content/5d5/9a36d9c658ab24a8b4569/1500-0-","dek":"Don't spend hours building customer surveys that no one cares about. Here’s what you’re doing wrong.","lastModified":"2019-08-19T20:00:00Z","published":"2019-08-19T20:00:00Z","sponsored":false},{"title":"6 Technical SEO Tips to Increase Traffic and Conversions on Your Website","canonical":"https://www.business.com/articles/6-technical-seo-tips-to-increase-traffic-and-conversions-on-your-website/","slug":"6-technical-seo-tips-to-increase-traffic-and-conversions-on-your-website","thumbnail":"https://www.business.com/images/channel/marketing.jpg","dek":"A site with a strong technical SEO foundation makes it easier for search engines to crawl and index its content.","lastModified":"2016-08-02T22:00:00Z","published":"2016-08-02T22:00:00Z","sponsored":false},{"title":"How to Improve Your Business's SEO Results on Google","canonical":"https://www.business.com/articles/improve-business-seo-on-google/","slug":"improve-business-seo-on-google","thumbnail":"https://www.business.com/images/content/5a3/95db692a3a4f71b8b457e/1500-0-","dek":"Improving your rank on Google means creating SEO-friendly web pages.","lastModified":"2018-01-02T17:36:00Z","published":"2018-01-02T17:36:00Z","sponsored":false},{"title":"7 Things Entrepreneurs Should Know About SEO","canonical":"https://www.business.com/articles/seo-things-entrepreneurs-should-know/","slug":"seo-things-entrepreneurs-should-know","thumbnail":"https://www.business.com/images/content/5d9/135f37b437458048b4568/1500-0-","dek":"Boosting the visibility of your business and generating more sales isn't just about creating a website.","lastModified":"2019-10-01T18:00:00Z","published":"2019-10-01T18:00:00Z","sponsored":false},{"title":"How to Use Content Marketing to Boost Organic Search","canonical":"https://www.business.com/articles/content-marketing-boost-organic-search/","slug":"content-marketing-boost-organic-search","thumbnail":"https://www.business.com/images/content/5ed/025019c658a58598b4577/1500-0-","dek":"Crafting the right content can help you reach the right audience and boost your SEO.","lastModified":"2020-06-02T16:55:00Z","published":"2018-08-21T12:13:00Z","sponsored":false},{"title":"How to Manage Digital Marketing for Small Business","canonical":"https://www.business.com/articles/the-digital-marketing-revolution-and-how-small-business-owners-can-keep-up/","slug":"the-digital-marketing-revolution-and-how-small-business-owners-can-keep-up","thumbnail":"https://www.business.com/images/content/5db/99fc09c658a1b268b4579/1500-0-","dek":"As the digital marketing landscape changes and evolves, what small business owners need to know in order to stay ahead of the curve.","lastModified":"2019-11-01T18:00:00Z","published":"2019-11-01T18:00:00Z","sponsored":false},{"title":"12 Signs You’re Losing Customers on Facebook","canonical":"https://www.business.com/articles/signs-youre-losing-facebook-followers/","slug":"signs-youre-losing-facebook-followers","thumbnail":"https://www.business.com/images/content/5e7/b99885a215ecd228b4579/1500-0-","dek":"It’s normal to occasionally lose some customers or followers on Facebook. It happens all the time. However, if you’re losing customers in large numbers and consistently, that doesn’t mean the end for your Facebook marketing campaign.","lastModified":"2020-03-25T20:49:00Z","published":"2020-03-25T20:49:00Z","sponsored":false},{"title":"The Small Business Guide to Local Marketing #DreamSmallBiz","canonical":"https://www.business.com/articles/the-small-business-guide-to-local-marketing/","slug":"the-small-business-guide-to-local-marketing","thumbnail":"https://www.business.com/images/channel/marketing.jpg","dek":"Online or off, as a small business, you have a myriad options to market locally, with unique advantages over big businesses.","lastModified":"2016-05-05T16:00:00Z","published":"2016-05-05T16:00:00Z","sponsored":false},{"title":"8 Ways to Build a Spectacular Online Presence","canonical":"https://www.business.com/articles/building-online-presence/","slug":"building-online-presence","thumbnail":"https://www.business.com/images/content/5dc/dfa625a215e777f8b4569/1500-0-","dek":"In order to succeed in today's climate, it is critical your business has a solid online footprint.","lastModified":"2019-11-15T16:00:00Z","published":"2019-11-15T16:00:00Z","sponsored":false}],"relatedReviews":[{"doc_type":"category","thumbnail":"https://staging.business.com/images/rz/5b99633151ee47a8528b4582/Call_center_bbernard.jpg","canonical":"https://staging.business.com/categories/call-center-services/","title":"The Best Call Center Services for 2021","dek":"Our team has compared the best call center services for 2021. See up-to-date reviews for the top-rated answering services.","slug":"call-center-services","lastModified":"2021-02-26T14:22:00Z"},{"doc_type":"category","thumbnail":"https://staging.business.com/images/rz/5df7b1b69c658a61728b4567/Answering_Service_scyther5_Getty.jpg","canonical":"https://staging.business.com/categories/best-answering-services/","title":"The Best Answering Services of 2021","dek":"Our teams have compared the best answering services for 2020. See up-to-date reviews and prices for these top-rated services.","slug":"best-answering-services","lastModified":"2021-02-26T14:18:00Z"},{"doc_type":"category","thumbnail":"https://staging.business.com/images/rz/5d4c65367b43741a258b4572/Laptop_Sfio_Cracho.png","canonical":"https://staging.business.com/categories/crm-software/","title":"The Best CRM Software of 2021","dek":"Looking for the best CRM software? We have easy-to-read, expert unbiased reviews and feature comparisons of the best and cheapest programs.","slug":"crm-software","lastModified":"2021-02-24T14:09:00Z"}],"relatedSponsoredArticles":null}},"pageContext":{"displayModified":"2019-11-20T16:00:00.000Z","template":"v1","title":"Why Google Bots Refuse to Crawl Your Site","meta":{"robots":"index, follow"},"docType":"articles","slug":"why-google-bots-refuse-to-crawl-your-site","canonical":"https://staging.business.com/articles/why-google-bots-refuse-to-crawl-your-site/","skipTOC":false}},"staticQueryHashes":[]}