For twenty years, "can Google reach my content" meant crawling your site. You wrote a robots.txt file, you watched your server logs for Googlebot, and that was the entire governance conversation. Two statements from Google in October 2026 make it clear that the conversation now has a second half, and many haven’t started on it.
In remarks covered on October 22, 2026, Sundar Pichai said AI agents will reach a publisher's site two ways going forward. They'll browse pages the way a person or a search crawler always has, and they'll also call a publisher's APIs directly. It's a plain statement from Google's own CEO, naming a channel most sites have never thought to put a policy around, because until now there wasn’t a reason to.
In a separate clarification that same week, Google's Gary Illyes drew a different but related line. Robots.txt, he confirmed, only governs whether a crawler is allowed to fetch a page. It has no bearing on what happens to content an AI system has already collected, nor training, summarization, generation, or whatever comes next. A separate Google explainer lays out which robots.txt syntax the company actually supports, making the same point from the mechanics side. The robots.txt file is a gate and not a contract.
These statements didn't land in a vacuum. We'd just finished tracking how little say publishers get over Google's own interface changes, after Google quietly reshuffled how AI Overviews surface links with no publisher input at all. The pattern holds here, too: Google sets the terms, and publishers find out after the fact unless they've built something ahead of time.
The Crawl Gate Was Never the Whole Door
This is the part that should unsettle anyone who treats robots.txt as their AI policy. Blocking a crawler stops that crawler from fetching a page. But it says nothing about a model trained on content fetched before the block went up, nothing about a different system that accessed the same content through a licensing deal, and nothing about an agent that reaches the content by some route other than a page fetch. The crawl gate and the usage question are two separate door locks, and robots.txt only ever held one of them.
Google quietly reinforced this from another angle, too. An updated help document lays out the full toolkit for controlling what reaches Google's systems, including the robots meta tag, data-nosnippet, X-Robots-Tag, and a setting Google calls search generative AI control, which manages what's shared specifically with Google's generative features. Four separate controls, four separate jobs. If your entire governance stack is still one robots.txt file, you're missing most of the available levers, let alone the new ones.
We made this case about training versus crawling back in September 2026, when we wrote about AI agents starting to pay publishers for content and the sites that still couldn't collect on it. The structural problem was the same then. One robots.txt file was trying to answer two different questions. This latest round of Google statements doesn't repeat that argument; it confirms it, straight from Google's own engineering team, and adds a layer our September piece didn't need to cover yet, because APIs weren't a part of the conversation at the time.
Pichai Just Named the Second Channel
Page access and API access aren't the same governance problem, and treating them as one is how a publisher ends up unintentionally wide open on the side nobody's watching. A page fetch is visible. It shows up in server logs, it's throttled by rate limits that already exist, and robots.txt at least attempts to speak to it. An API call is a different animal. It can be fast, structured, and invisible to the monitoring most publishers built for human and crawler traffic, because nobody designed that monitoring with agent traffic in mind.
Most publishers have no governance model for the second channel. Not a restrictive one, not a permissive one, none at all. There's no robots.txt-for-APIs convention, no equivalent of a crawl-delay directive, no established norm. That gap is why Pichai naming it matters. It turns a question publishers could previously ignore into one Google's own leadership says they now have to answer.
We wrote about the adjacent problem in September 2026, too, in a piece on why a site now needs to be legible to AI agents, not just search engines. That piece was about whether an agent could understand a site's structure well enough to act on it. This is the next layer down. It's about whether an agent can reach a site's capabilities at all, through a channel that bypasses the page entirely. Legibility and access are related problems, but they're not the same fight, and Pichai's comments are squarely about the second one.
A Second Readiness Layer, Built on Purpose
Here's the plain version of the problem. If you haven't decided what your APIs expose to agents, something has already decided for you, by default, and probably badly. Either the door's locked tight and even your own agentic workforce can't use it, or the door's effectively unlocked because nobody ever built a gate. Neither is a strategy. Both are accidents waiting to be discovered, by a competitor's agent, by a security researcher, or by Google in a future announcement that treats the gap as newsworthy.
If you've already thought seriously about robots.txt governance, you have half the mental model in place. Explicit, documented, and intentional rules beat silence every time. The job now is applying that same discipline to a channel robots.txt can't touch.
That's the reason RebelMouse's Agentic CMS is built around MCP connections, rather than one-off integrations bolted on as an afterthought. The Agentic Workflows and Integrations Manager is the piece of our stack that chains agents together across a publisher's existing tools, and it works by connecting through MCP on purpose: A governed, visible layer that sits between your systems and anything, human or agent, that wants to act on them. The API-access question Pichai recently put a name to isn't new to us. Our own architecture has been answering it since before this was a headline, by giving your agentic workforce a defined way to expose capabilities instead of leaving that door to be found informally by whoever gets there first.
More Doors Means More Decisions, Not Fewer
The AI-replaces-judgment framing falls apart right here. That's where the story lands. Adding a second access channel doesn't shrink the number of decisions you have to make. It multiplies them. Every new thing an API can expose is a new question.
- Who gets to call it?
- What are they allowed to do with the response?
- Has a given agent even earned the access it's asking for?
None of that is something a default setting answers well, and none of it is something you'd want a default setting to answer at all.
This is the judgment call that belongs with a person, which is why the inbox and its approval gates exist inside our Agentic CMS in the first place. An agent can draft the connection, propose the exposure, even flag which of a publisher's existing tools would benefit from it. A person still approves what gets opened and to whom. Execution is abundant now. Pichai's and Illyes's own comments confirm there are more ways than ever for that execution to reach a publisher's content. Judgment about what to expose, and to whom, is the part that doesn't get automated. It gets routed to someone who can weigh it.
Where to Start
None of this requires you to solve every access question overnight. It requires picking a first move.
- Audit what APIs already exist and have never had a policy attached.
- Decide what needs exposing to an agent versus what's been sitting exposed by accident.
- Put a real approval step in front of any new exposure instead of defaulting to open or closed.
Our optimization tools already handle the discovery side of this equation for a lot of publishers. The access side is the half that's still mostly unbuilt. If you want that built to spec for your business, rather than assembled piecemeal, that's something we can help with.
Google split the question into two parts. Most publishers have only ever answered one of them. The other one is sitting out in the open right now, and it won't stay unnoticed for long.
