Google, Cloudflare and Shopify Are Rebuilding the Web for AI Agents

Soon, a much larger share of website traffic will not come from humans at all. It will come from AI bots and agents, and I think this shift will happen much faster than most marketing teams expect.

The simplest way to understand it is this: when an AI assistant searches the web, checks sources, or researches an answer, machines are being sent to retrieve and extract information from websites. Agents take this one step further by executing tasks on behalf of users, including research, comparisons, purchases, and other actions where they are allowed to do so.

The web is becoming agentic, but most websites were never built for this.

Most websites are still designed only for humans

Websites were built around humans opening a browser, navigating menus, scrolling pages, and interpreting visual information. AI bots and agents behave differently. They need to discover information quickly, extract it, understand relationships between entities, and increasingly understand which actions they can perform.

This is why LLM crawlability, structured data, machine-readable endpoints, agent manifests, and callable skills are becoming infrastructure rather than experimental SEO features.

Across thousands of websites tested through LightSite’s tools, we keep seeing the same problems: weak structured data, important information buried inside HTML and JavaScript, and AI bots blocked without the marketing team even knowing it. In one scan of 2,870 websites, 27% were blocking at least one major LLM crawler. You can see the wider dataset in our State of AI Search Readiness 2026 report.

Why this matters

There are two reasons I think marketers should care.

The first is control. Your website is still the source you control more than anything else on the internet. You cannot fully control what Reddit, review sites, journalists, or competitors say about you, but you can control how clearly your own site explains who you are, what you sell, who you serve, and why somebody should trust you.

If an AI bot needs to reconstruct all of that from raw HTML, navigation, JavaScript, and marketing copy every time it visits, you are creating unnecessary ambiguity. A proper machine-readable website layer gives AI systems a cleaner way to understand the same information without changing the human experience.

The second reason is measurement. Bot activity is one of the few deterministic signals we have in AI search. A bot either visited a page or it did not. It either used a particular endpoint or it did not. A campaign either produced more machine activity or it did not.

That is different from share of voice, mentions, sentiment, and simulated prompt tests. Those metrics are useful, but they are still simulations. Real bot behavior tells you what actually happened.

This is also the topic I will be discussing with Search Engine Journal in our webinar on measuring AI search with real behavioral signals rather than relying only on simulated visibility metrics.

The industry is sending a very clear signal

When we started building LightSite as infrastructure for AI bots, one of the most common reactions was: why would anyone need this? Bots already understand normal websites.

Technically, they can. But that does not mean the current web is optimized for them.

Google recently introduced Agentic Resource Discovery, a specification designed to help agents discover and verify tools, skills, and other capabilities available across the web. Instead of forcing agents to blindly explore websites and infer what they can do, the idea is to make those capabilities easier to discover in a structured way. We covered the adjacent browser-side standard in What Is Google WebMCP and What It Means for Marketers.

Cloudflare is solving another part of the same problem with Markdown for Agents. An AI agent can request a markdown version of a webpage rather than processing the full HTML, reducing the amount of noise and tokens required to understand the page.

We already wrote a deeper explanation of the difference between making a website readable and making it recommendable in our Cloudflare Markdown for Agents guide.

Shopify is moving even further toward agentic commerce, creating infrastructure that makes product data easier for AI systems to discover and use in shopping workflows.

These companies are solving different technical problems, but the direction is the same: AI bots and agents need a cleaner way to interact with websites.

Structured data is important, but it is not the whole solution

I think this is where many companies will make the mistake of treating structured data as another SEO checkbox.

Structured data is a very good foundation, but agents need more than schema. They need to know where information lives, what actions are available, which data is current, and how to move efficiently through the site.

That is why LightSite has continued extending its infrastructure beyond JSON-LD. The platform can publish structured business information through an LLM Discovery API, maintain machine-readable website context, expose Skills and agent manifests, and measure how AI bots actually use those resources.

Our Skills and agent access layer is based on a simple idea: instead of making an agent guess how to search a website, find products, retrieve FAQs, or understand the business, make those capabilities explicit.

We have already seen bots change their behavior when these paths become available. That does not prove that a manifest automatically improves AI rankings, but it does show that agent behavior changes when websites provide infrastructure designed specifically for machines.

How should companies prepare?

You do not need to redesign your website for robots. The human site should remain optimized for humans. But underneath it, I would make sure AI systems can access the site, understand the main entities, retrieve important information through clean machine-readable formats, discover useful actions, and do all of this without unnecessary friction.

Just as importantly, marketing teams should measure what happens after implementation. Which bots are visiting? Which pages are they consuming? Which structured resources are actually being used? Does machine activity increase after a new campaign, new content, or a technical change?

This is where AI bot analytics becomes useful. Instead of deploying technical infrastructure and hoping that AI systems use it, you can see how the machines actually interact with your site. Our AI Bot Behavior Index shows the crawl patterns we observe across engines.

You can build this internally, but it should not become another six-month project

There is nothing stopping a company from building this infrastructure internally. A large organization can involve engineering, technical SEO, data teams, or an agency to deploy structured data, build endpoints, configure crawler access, and keep up with new agent standards.

The problem is usually not whether it can be done. It is how long it takes, how many teams need to be involved, and whether marketing can actually see how the infrastructure performs once it is live.

That is the problem LightSite was built to remove. We add a dynamic machine-readable layer to an existing website in minutes, keep it synchronized as the website changes, and continuously adapt it as AI discovery standards evolve. The same platform also measures how AI bots use the infrastructure, so implementation and measurement sit in one place instead of becoming separate technical projects. If you are still comparing vendors, our rundown of the top GEO platforms in 2026 is a good starting point.

The web is becoming agentic

I do not think human websites are disappearing. Humans will continue to care about design, experience, brand, trust, and emotion.

But websites now have a second audience: machines that increasingly sit between the company and the buyer. Sometimes they retrieve information, sometimes they recommend a vendor, and increasingly they will take actions on the user’s behalf.

The exact protocols will change, but the direction is already clear. The websites that win will still be built for humans on the surface, while becoming much easier for machines to understand and use underneath.

If you want to see how ready your website is today, run the free GEO Checker.

If you want the infrastructure deployed without turning it into another long technical project, start with LightSite AI.