Gemini Notebook Ignores robots.txt. Here's What Controls It

What actually changed in July 2026?

Two things landed a day apart, and only one of them is operationally new. On July 16, 2026, Google rebranded NotebookLM to Gemini Notebook and folded it deeper into the Gemini ecosystem, with planned integration into Search's AI Mode. On July 17, Google's crawling documentation changelog renamed the fetcher's user agent from Google-NotebookLM to Google-GeminiNotebook, with the legacy string supported only through a transition window reported as lasting until around August 2026. No specific day has been published.

The fetcher itself is not new. Google first documented Google-NotebookLM as a user-triggered fetcher in October 2025, roughly nine months before the rename. Its behavior (fetching only URLs a user's action requests, ignoring robots.txt) is identical under the new name. As Search Engine Journal put it, 'Gemini Notebook is the exact same thing. There is no change to how it works or what it does.' What changes is visibility: a product now wired into the Gemini app and AI Mode will route more research sessions, and therefore more fetches, through your content.

The failure mode nobody sees coming: if you blocked the fetcher after October 2025, your rule matches the literal string Google-NotebookLM. Once traffic fully moves to the new name, that rule stops matching anything. Not because Google removed the fetcher, but because the fetcher renamed itself past your pattern.

Why doesn't robots.txt work on this fetcher?

Google sorts its bots into classes, and the class decides whether robots.txt applies. Per the official documentation, 'user-triggered fetchers are initiated by users to perform a fetching function within a Google product,' and because each fetch answers a specific user request, these fetchers 'generally ignore robots.txt rules.' The logic: a person pasted the URL or asked for sources, so the robots exclusion protocol, written for autonomous crawlers, does not govern it.

The practical consequence trips up experienced operators. A robots.txt directive targeting this user agent is not rejected, logged, or errored. It is simply never read. A rule like 'User-agent: Google-GeminiNotebook / Disallow: /' does nothing at all.

The narrow claim matters: robots.txt is not broken. It still fully governs autonomous crawlers like Googlebot, and Google-Extended remains the documented opt-out for AI-training use of your content. What this post covers is the class of Google bots that the robots.txt layer cannot reach, and Gemini Notebook's fetcher is the highest-traffic member of it right now.

The mental-model fix: stop thinking of robots.txt as the bot-control lever. It is one lever that binds one class of bots. Autonomous crawlers read it. User-triggered fetchers, by Google's explicit design, do not. Controls for the second class live one layer down the stack: firewall, server config, IP ranges.

What is Discover Sources, and why is it the real risk?

Both of Gemini Notebook's fetch paths use the same Google-GeminiNotebook user agent, but they carry very different risk profiles, and conflating them leads to bad blocking decisions. The classic path is a manual source add: a user pastes your URL into a notebook, the fetcher retrieves that one page, and the user works with content they deliberately chose. That is close kin to a person reading your page with an AI notepad.

Discover Sources is different. The user types a topic, and the feature scrapes up to 10 articles for that query, then produces an AI summary with no referral traffic back to any of the sources. The user never curates the URL list and does not necessarily see which pages were fetched before the summary lands. An entire research session can run on your reporting and your analysis without a single visit registering in your analytics.

Search Engine Journal flags a second-order risk worth naming, and attributing, precisely: content ingested this way can be repurposed by users into Audio Overviews and Video Overviews, formats Gemini Notebook has long supported, without attribution back to the original site, and if those derivative assets are republished, they could compete with the source in search. That scrape-summarize-remix-republish chain is SEJ's risk framing, not a Google statement. But the underlying mechanics it rests on (10 sources per query, zero referrals, built-in derivative formats) are documented product behavior.

The honest counterweight: this is also how modern research gets done, and being one of the 10 sources a strategist's briefing draws on has real influence value even when no click follows. Google states the product reaches 30 million-plus users and 600,000-plus organizations.

Which Google agents actually obey robots.txt?

The coverage of this rename treats Gemini Notebook in isolation. The more useful view is the category: which Google agents read robots.txt, and which are exempt by class. Google now operates roughly ten agents that your robots.txt file cannot govern, and the AI-era additions (Gemini Notebook's fetcher, Google-Agent) are the ones fetching editorial content specifically.

The first group reads and obeys robots.txt. These are the crawlers and control tokens you already know: Googlebot, which runs autonomous crawls for Search and is controlled with robots.txt Disallow rules, and Google-Extended, the AI-training opt-out token controlled with a robots.txt User-agent rule.

The second group ignores robots.txt by class. None of these fetchers ever read the file, so the only working controls sit at the firewall, server-config, or IP-range layer. Read that second table as a category and the strategic picture sharpens. Each new user-facing AI surface Google ships arrives as a user-triggered fetcher, robots.txt-exempt by classification, and the control burden shifts one layer down the stack. It moves from a text file any marketer can edit to firewall and server configuration that many marketing teams cannot touch without engineering support.

How do you actually block or control it?

One scoping note first: this section is about controlling one robots.txt-exempt fetcher class. The broader policy question (which AI crawlers to allow for training versus search citations, and how robots.txt and llms.txt fit together) is a different decision layer. Everything below exists because this fetcher class reads none of those directives.

Option 1: .htaccess or server-config UA block. .htaccess is a settings file on Apache-family web hosting that lets you set rules for how the server responds to requests. The most accessible control here. 'Match the UA substring' just means: look at the name tag on the incoming request, and if it contains your target text (not an exact full-line match), block it. 'Return 403' means the server refuses with a 'Forbidden' response instead of serving the page. During the transition window, match both strings: RewriteEngine On / RewriteCond %{HTTP_USER_AGENT} (Google-GeminiNotebook|Google-NotebookLM) [NC] / RewriteRule .* - [F,L].

In plain terms, that reads: 'If the request's name tag contains Google-GeminiNotebook or Google-NotebookLM (case-insensitive), stop and return Forbidden.' On nginx or at a CDN/WAF layer (a WAF is a Web Application Firewall, the security filter sitting in front of your site at providers like Cloudflare), the equivalent is a 'user-agent-contains' condition returning 403. Same logic, different syntax. If you are on managed hosting (Shopify, Webflow, Squarespace), you usually cannot touch this layer yourself; you would set it at your CDN or ask your host.

Option 2: IP-range blocking. An IP range is the set of network addresses a company's servers send from. User-agent strings are self-reported, and anyone can spoof Google-GeminiNotebook in a request header. For verification or harder enforcement, Google publishes the IP ranges its fetchers operate from as JSON files. A firewall rule keyed to those published ranges blocks the traffic regardless of what name tag it claims, at the cost of maintaining the range list as Google updates it.

Option 3: Web Bot Auth. Google's documentation points to the experimental Web Bot Auth protocol, which uses cryptographic signatures to prove a bot is who it says it is, rather than trusting a name tag anyone can fake. It is early and not a practical dependency for most sites in 2026, but it signals where this is heading: verifiable bot identity at the request layer, because self-declared name tags were never a security boundary.

What not to bother with: robots.txt directives aimed at this UA (never read), meta robots tags (those are indexing instructions in a page's HTML; the fetcher is not indexing your page, so they are beside the point), and blanket Googlebot blocks (which take your search visibility down with them).

A UA-string block is a courtesy barrier, not a security control. It stops the documented fetcher, which honestly identifies itself, and nothing that lies about its name tag. That is usually the right trade here: the goal is controlling a known Google product's access, not defending against adversarial scrapers. For the latter, you need the IP-range and WAF-behavioral layer.

Should you block it at all?

Now the strategic question, because the technical ability to block does not make blocking right. The calculus differs sharply by business model, and blocking a research tool that 30 million-plus people use to build briefings and study guides is not a free action. Being among the sources an AI summary draws on is a form of reach, even at zero referrals. Being absent from it is invisibility, not protection.

A quick gloss on 'renders server-side' for the gated row: a fetcher sees what any logged-out request sees. If your paywall locks content in the browser (JavaScript hides it after load), the full text may still sit in the initial HTML the fetcher receives, meaning it can read 'paid' content for free. Server-side rendering means the locked content is never sent unless the request is authenticated. Check which one your paywall does before assuming you are protected.

This is the AEO tradeoff in its purest form, and it is the same call I walk through in how to get cited by AI search engines. If your business model depends on being cited, a hard block on a research-class fetcher is working against yourself. If your business model depends on pageviews or a paywall, an allow-by-default posture is a leak. There is no universal right answer, only the answer that matches how you make money.

The projection to plan around: the user-triggered fetcher class grows. Gemini Notebook is being wired into the Gemini app and Search's AI Mode, Google-Agent already sits in the same exempt category, and every agentic product across the industry has the same architectural excuse (a human asked, so robots.txt does not apply). By 2027, access control for content sites will look less like a robots.txt file and more like a firewall policy: identity-verified bots, published IP ranges, per-class allow and deny decisions maintained at the CDN.

What should you do this week?

Three moves, in order. Do them before the August transition window closes.

The larger signal is the one worth sitting with. As AI products reframe fetching as something a user asked for, the robots exclusion protocol quietly shrinks from the web's access-control standard to one lever among several. The sites that navigate the next two years well will be the ones that treated that shift as an infrastructure project: measured first, decided per bot class, and kept their rules current through renames like this one.

Automate the repeat. Review the risk. And know which lever actually reaches the bot you're trying to control.

• Update every UA-keyed rule and log query to match both strings: Google-GeminiNotebook and Google-NotebookLM. If your log dashboards only track the old string, your bot-traffic reporting develops a quiet gap exactly when the trend line gets interesting.
• Measure before you decide. Your access logs are the record your web server keeps of every request it receives, including the name tag each one wore. Search those logs for both strings and quantify actual fetch volume. Most hosting dashboards or a tool like GoAccess can surface this without you touching a command line. Many sites will find the volume negligible today; the blocking decision can wait for data.
• Decide your posture on Discover Sources by business model, not reflex. Zero-referral, 10-articles-per-query summarization is the feature that matters. Block it if pageviews or a paywall fund you. Allow it if influence and citations fund you.