.Google's John Mueller responded to an inquiry concerning why Google.com marks web pages that are actually refused from crawling by robots.txt and why the it's secure to ignore the associated Look Console reports regarding those crawls.Crawler Website Traffic To Query Criterion URLs.The individual asking the question chronicled that bots were creating web links to non-existent inquiry specification URLs (? q= xyz) to web pages with noindex meta tags that are likewise shut out in robots.txt. What motivated the inquiry is that Google is actually creeping the hyperlinks to those webpages, acquiring shut out through robots.txt (without watching a noindex robotics meta tag) at that point receiving shown up in Google Explore Console as "Indexed, though blocked by robots.txt.".The individual inquired the following inquiry:." However right here is actually the significant concern: why would certainly Google index web pages when they can not even find the information? What is actually the benefit in that?".Google's John Mueller confirmed that if they can't crawl the web page they can not view the noindex meta tag. He also makes an exciting acknowledgment of the web site: hunt driver, encouraging to neglect the outcomes since the "typical" individuals won't find those results.He created:." Yes, you're correct: if our experts can't crawl the webpage, our experts can't view the noindex. That claimed, if we can not crawl the web pages, at that point there is actually certainly not a lot for us to index. So while you could view some of those webpages along with a targeted site:- inquiry, the average consumer will not find all of them, so I definitely would not bother it. Noindex is likewise alright (without robots.txt disallow), it only suggests the URLs will wind up being actually crawled (as well as end up in the Look Console report for crawled/not recorded-- neither of these statuses induce problems to the rest of the internet site). The important part is that you don't make them crawlable + indexable.".Takeaways:.1. Mueller's response confirms the limitations in using the Website: search accelerated search operator for analysis causes. Some of those causes is because it is actually not connected to the normal hunt index, it's a distinct point altogether.Google's John Mueller commented on the web site search driver in 2021:." The short answer is that a website: query is actually certainly not suggested to be complete, neither utilized for diagnostics purposes.A site query is a particular kind of search that restricts the outcomes to a particular site. It's primarily only words website, a digestive tract, and afterwards the web site's domain name.This concern limits the results to a particular website. It's certainly not meant to become a comprehensive assortment of all the pages coming from that site.".2. Noindex tag without making use of a robots.txt is actually fine for these sort of circumstances where a crawler is linking to non-existent webpages that are acquiring found out by Googlebot.3. Links along with the noindex tag will create a "crawled/not indexed" entry in Explore Console which those will not possess an adverse result on the remainder of the website.Go through the inquiry as well as answer on LinkedIn:.Why will Google.com index webpages when they can not even observe the material?Included Graphic by Shutterstock/Krakenimages. com.