Microsoft Clarity Now Shows Which Bots Ignore Robots.txt

What is Microsoft Clarity's robots.txt violations feature?

The robots.txt violations feature is an addition to Microsoft Clarity's Bot Analytics that reports when crawlers fetch URLs you have disallowed. Clarity already showed you how much bot traffic your site received and which bots were behind it. This update goes one step further and checks that activity against your stated crawl rules, then flags every request that broke them.

It matters because robots.txt has always been an honor system. The file states your preferences, but nothing forces a crawler to obey. Until now, most site owners had no easy way to know whether bots were actually respecting it. Clarity closes that gap with a Violations card that expresses non-compliant requests as a percentage of total bot requests, giving you a quick read on how big the problem is on your specific site. And because Clarity is free, this visibility is no longer reserved for sites running enterprise log-analysis tools.

What does the violations report actually show?

The report breaks one number into the details you need to act on. At the top, the Violations card gives you non-compliant requests as a share of all bot traffic, your headline metric. Below that, a violation trendline plots those requests over time, so you can catch a sudden spike or a crawler that violates your rules day after day.

The strength of the feature is in its filters. You can narrow the data by operator, by bot name, and by activity type to isolate a single offending crawler, and see exactly which URLs and paths are generating violations. A comparative view then separates the bots respecting your crawl preferences from the ones ignoring them. According to Cloudflare's 2025 analysis, training now drives close to 80% of AI bot activity, up from roughly 72% a year earlier, so knowing which crawlers touch which pages is no longer a niche concern.

How does this fit into Clarity's AI bot tracking?

The violations card is one panel inside a larger AI Bot Activity dashboard that Clarity has built out over 2026. Clarity launched AI Bot Activity on January 22, 2026, then added a deeper layer on May 26 covering three things: scale, quality, and patterns.

Scale metrics show AI bot request volume, AI bot traffic share as a percentage of all requests, and how much of your site bots have actually crawled. Quality metrics break requests down by outcome, successful, redirected, or unsuccessful, so you can spot bots hammering dead URLs and creating needless server load. Pattern metrics let you filter by operator and path to see where crawling concentrates. Crucially, all of this is powered by server-side logs from your connected CDN, not client-side JavaScript, which is why it can see bots that never execute page scripts. Robots.txt violations sit on top of that foundation as the compliance layer.

Why do bots ignore robots.txt in the first place?

Robots.txt is a voluntary standard, not a lock. The file lives at your domain root and lists which user agents may crawl which paths, but compliance depends entirely on the crawler operator choosing to honor it. A well-behaved bot reads the file and obeys. A bad actor reads it and does whatever it wants, and there is no technical barrier stopping it.

The reasons split into a few buckets. Some bots are openly non-compliant: security vendors regularly document bad bots that spoof user agents and ignore directives outright. Some are non-compliant by design: Google's user-triggered fetchers skip robots.txt because Google treats them as proxies acting on behalf of a real person, not as crawlers. And some are deceptive. On August 4, 2025, Cloudflare published a report showing Perplexity using undeclared crawlers that rotated user agents, IP addresses, and networks to evade no-crawl rules. That last category is exactly why a violations report is useful: it catches behavior a simple robots.txt file never could.

Why does this matter for AI search visibility?

This is where a bot-analytics update becomes an AI search story. The crawlers showing up in your violations report are, increasingly, AI crawlers: training bots feeding large language models and retrieval bots pulling answers for tools like ChatGPT, Perplexity, and Google AI Overviews. Microsoft frames this bot activity as the earliest observable signal of how your content may be summarized, cited, or surfaced in assistant-driven experiences, a signal that has stayed invisible to most site owners until now. You cannot optimize for AI answers if you cannot see which AI systems are reading you.

It also forces a real decision rather than a default. Blocking a crawler in robots.txt only works if the crawler obeys, and the data shows many do not. A 2025 Cloudflare report put Anthropic's crawl-to-referral ratio at roughly 73,000 to 1, meaning enormous crawling for almost no traffic back. Seeing that imbalance in your own Clarity dashboard lets you choose deliberately: open the door to bots that send value, and escalate to a firewall or CDN-level block for bots that take without giving.

As Ana D'Oliveira, founder of Anaverse, puts it: robots.txt tells bots what you want, analytics tells you what they actually did, and only one of those is the truth. You cannot manage AI crawlers you cannot measure, and this is the first free tool that lets most sites measure them at all.

How do you set up robots.txt violation tracking in Clarity?

Setup takes a few minutes but has one hard requirement: Clarity needs to see your server-edge traffic, which means connecting a supported CDN. The feature reads request data at the CDN layer, so without that connection there is nothing to analyze.

• Open your Clarity project and go to Project Settings.
• Connect a supported CDN. AI Bot Activity supports Fastly, Amazon CloudFront, Cloudflare, Azure Front Door, and Akamai; the violations card launched on Fastly, CloudFront, and Cloudflare.
• If you run WordPress, update to the latest Microsoft Clarity plugin. Older versions do not support the feature.
• Go to Dashboards, then AI Visibility, then AI Bot Activity, and find the Violations card.
• Review the percentage of non-compliant requests, then apply filters by operator or bot name to investigate specific crawlers.
• Open the affected URLs to see which paths are being hit, and compare compliant versus non-compliant trends over time.

What should you do once you see violations?

Seeing the data is step one; the value is in what you do next. Start by separating the noise from the signal. A user-triggered fetcher hitting a disallowed page once is rarely worth chasing. A crawler systematically pulling your premium content while ignoring an explicit Disallow is a different problem and deserves a response.

• Identify your repeat offenders: use the trendline and filters to find bots that violate consistently, not just once.
• Decide value per bot: if a crawler sends meaningful referral traffic, you may want it crawling; if it takes and never returns, treat it as a cost.
• Escalate beyond robots.txt: for bots that ignore the file, a polite Disallow does nothing, so block at the WAF, firewall, or CDN level.
• Protect high-value paths first: prioritize the URLs your violations report shows are being targeted, not your whole site.
• Re-check the trend: after you act, watch the trendline to confirm the violations actually dropped.