.Google's John Mueller addressed a question regarding why Google.com marks webpages that are prohibited from crawling through robots.txt as well as why the it is actually risk-free to disregard the relevant Look Console documents regarding those creeps.Crawler Website Traffic To Concern Criterion URLs.The individual talking to the inquiry recorded that bots were actually developing hyperlinks to non-existent inquiry parameter URLs (? q= xyz) to web pages along with noindex meta tags that are also blocked out in robots.txt. What cued the inquiry is actually that Google.com is actually creeping the links to those pages, receiving shut out through robots.txt (without envisioning a noindex robots meta tag) after that receiving shown up in Google Look Console as "Indexed, though shut out by robots.txt.".The individual inquired the observing question:." However below is actually the big question: why would certainly Google.com mark web pages when they can't also find the information? What is actually the conveniences because?".Google's John Mueller confirmed that if they can't crawl the page they can not see the noindex meta tag. He additionally helps make an interesting acknowledgment of the site: search operator, suggesting to ignore the outcomes because the "common" customers will not find those end results.He wrote:." Yes, you are actually appropriate: if our company can not crawl the webpage, our experts can't view the noindex. That pointed out, if our company can't creep the pages, at that point there is actually not a great deal for our company to mark. Therefore while you might view a number of those pages with a targeted site:- inquiry, the ordinary consumer will not see them, so I definitely would not fuss over it. Noindex is likewise fine (without robots.txt disallow), it just means the URLs will certainly wind up being crawled (and also find yourself in the Search Console report for crawled/not listed-- neither of these conditions induce problems to the remainder of the website). The important part is actually that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the constraints in using the Web site: hunt evolved hunt driver for diagnostic reasons. Some of those factors is actually since it is actually certainly not attached to the normal search index, it is actually a separate point altogether.Google.com's John Mueller commented on the website search driver in 2021:." The short response is actually that a web site: question is certainly not meant to be total, neither used for diagnostics purposes.A website inquiry is a specific type of search that restricts the outcomes to a particular internet site. It is actually generally only the word web site, a digestive tract, and afterwards the internet site's domain.This query restricts the end results to a particular internet site. It is actually certainly not suggested to become an extensive compilation of all the webpages from that internet site.".2. Noindex tag without utilizing a robots.txt is alright for these type of circumstances where a robot is actually connecting to non-existent pages that are actually acquiring found by Googlebot.3. URLs with the noindex tag will certainly generate a "crawled/not indexed" entry in Search Console and also those will not have a bad effect on the remainder of the web site.Go through the concern and also respond to on LinkedIn:.Why would certainly Google.com mark pages when they can not also find the content?Featured Image by Shutterstock/Krakenimages. com.