On this page

daydream journal

notes on AI, growth, and the journey from 0→n

All Resources

Insights

How OpenAI Crawls and Indexes Your Website

How OpenAI crawls, indexes, and trains on your content—and how to prepare for all three

May 5

・

Thenuka Karunaratne

How OpenAI Crawls and Indexes Your Website

The architecture of online visibility is being rewritten.

For decades, search visibility meant optimizing for Google’s index: ensuring crawlability, earning backlinks, and ranking in blue links. As large language models (LLMs) become default interfaces for information retrieval, a new layer of discoverability is emerging—one that blends training data, indexed sources, and real-time retrieval.

OpenAI is at the forefront of this shift. With three separate bots operating across its platform, visibility today means understanding the distinct roles each one plays:

GPTBot crawls content for model training.
OAI-SearchBot indexes content for search results inside ChatGPT.
ChatGPT-User accesses content on demand during user-initiated browsing or plug-in activity.

Each bot has different rules, capabilities, and strategic implications. To ensure your site is visible in this new environment, you need to consider how your content feeds, shapes, and surfaces within a trillion-token ecosystem.

Crawling and Indexing 101 (OpenAI Edition)

Before OpenAI can cite, summarize, or retrieve your content in tools like ChatGPT, it has to discover and process it first. That’s where crawling and indexing come in.

Crawling: How GPTBot Finds Your Content

GPTBot (OpenAI’s crawler) uses a combination of:

Backlinks from other websites
Publicly accessible URLs
Shared links in user queries
Possibly sitemap.xml and structured references

Unlike Googlebot, GPTBot doesn't use a full browser or render JavaScript—so what it sees is the raw HTML response. That makes server-side rendering (SSR) a must for visibility.

Indexing: What Gets Stored and Used

OpenAI maintains its own internal index for retrieval and synthesis. This isn’t a traditional search engine index, but a curated set of text snippets and metadata used by:

ChatGPT’s Browsing tool
GPT-4’s internal grounding memory
Products like ChatGPT Enterprise or API RAG pipelines

If your content is well-structured and crawlable, it’s more likely to be fetched and stored in a format that can be recalled accurately.

Note: OpenAI doesn’t provide a Search Console-like tool (yet), so server logs and crawl monitoring tools are your best bet for visibility checks.

If OpenAI can’t access your content directly, it may still cite you based on third-party summaries (like Reddit, Wikipedia, or news aggregators). But to ensure it pulls your words, you need to make them crawlable.

1. Training: Inclusion in OpenAI’s foundational models

GPTBot is OpenAI’s crawler for model training. It collects publicly available data to expand the model’s understanding of the world, improving its ability to generate accurate, comprehensive responses across topics.

By default, GPTBot respects your site’s robots.txt file. If your goal is inclusion, ensure it has access:

User-agent: GPTBot
Allow: /

If you prefer to exclude your site from training, use:

User-agent: GPTBot
Disallow: /

You can also allow or disallow specific directories. For example:

User-agent: GPTBot
Allow: /docs/
Disallow: /checkout/

Note: Blocking GPTBot only affects future training runs. If your content was previously ingested, it remains part of the model.

Being included in GPTBot’s crawl isn’t just about visibility. It shapes how your brand is represented in outputs. We’ve seen tools and frameworks earn default mentions in generated answers, without ever ranking in Google, because they were well-represented in training data.

2. Indexing: Real-time visibility in ChatGPT search

OAI-SearchBot supports ChatGPT’s live search capabilities, including inline citations and real-time answers. This bot builds and maintains an internal index that supplements the model’s knowledge with up-to-date web data.

This is where source attribution happens. For example, when ChatGPT returns a cited paragraph with a clickable link, that’s OAI-SearchBot at work.

Like GPTBot, it can be allowed or disallowed independently:

User-agent: OAI-SearchBot
Allow: /

Updates to robots.txt are typically honored within 24 hours.

Optimizing for OAI-SearchBot requires attention to:

Clear, scannable content structure
High-authority backlinks and mentions
Fast-loading pages with server-rendered content and minimal reliance on JS for primary content

3. Browsing: On-demand retrieval via user interaction

ChatGPT-User is triggered when a user asks a Custom GPT to fetch content, uses a plug-in, or interacts with external web tools inside ChatGPT. While this isn’t a crawler in the traditional sense, it functions like a browser agent for LLM users.

You can control access in robots.txt just like the others:

User-agent: ChatGPT-User
Allow: /

This type of access powers functionality like ChatGPT’s web browsing tool, or integrations that pull real-time product specs, documentation, or support content.

Allowing this bot ensures your site can respond to direct user-driven requests within the ChatGPT interface.

Controlling Indexing with Meta Robots Tags

While robots.txt controls whether OpenAI’s bots (like GPTBot and OAI-SearchBot) can crawl a page, it does not control whether that page appears in their internal index or training datasets if it’s discoverable via external links. To explicitly control indexing behavior, you’ll need to use the robots meta tag.

Add the following to the <head> section of your HTML page:

<meta name="robots" content="noindex">

This tag tells compliant bots—including OpenAI’s—to exclude the page from being indexed or cited in generative results. You can combine multiple directives as needed:

<meta name="robots" content="noindex, nofollow">

Here’s what key directives do:

noindex: Prevents the page from being included in search indexes or LLM citation databases.
nofollow: Prevents crawlers from following outbound links on the page.
nosnippet: Prevents display of text or media snippets from the page in responses.
noarchive: Blocks cached versions.
unavailable_after: [date/time]: Automatically expires visibility after a specific date.

You can also apply these to non-HTML assets (like PDFs or videos) using HTTP headers:

X-Robots-Tag: noindex

Important Caveats:

Do not combine noindex with a Disallow rule in robots.txt. If a crawler is blocked from accessing a page, it won’t see the meta tag at all, and may index the page anyway if it’s linked externally.
Changes to meta tags are typically respected within 24–48 hours of recrawling.

In short, if you want to prevent OpenAI from indexing or citing a page, but not block crawling entirely, meta robots is the right tool. Use it to fine-tune which pages show up in generative results, and which stay private or transient.

OpenAI doesn’t render JavaScript

There’s one major constraint that often gets overlooked in AI indexing conversations: OpenAI’s crawlers can’t render JavaScript.

Unlike Googlebot, which fetches, parses, and executes scripts to render dynamic content, OpenAI’s ecosystem of bots (GPTBot, OAI-SearchBot, and ChatGPT-User) only sees what’s present in the initial HTML. That means anything rendered client-side, such as product details, documentation tabs, or even your primary article content, may never be visible to OpenAI at all.

Recent data from Vercel and MERJ makes this painfully clear. Their joint analysis tracked over half a billion GPTBot fetches and found zero evidence of JavaScript execution. Even when GPTBot downloads JS files (which it does—about 11.5% of the time), it doesn’t run them. The same goes for Anthropic’s ClaudeBot, Meta’s ExternalAgent, ByteDance’s Bytespider, and PerplexityBot. No execution. No hydration. No client-rendered content.

If your core content depends on JavaScript to appear, it might as well not exist as far as OpenAI is concerned.

What does this mean for visibility?

If your site is built with frameworks like React, Vue, or Next.js, you’re not automatically in trouble, but you do need to be intentional.

OpenAI can only index what’s included in the raw HTML it receives. Anything rendered later by JavaScript won’t be seen. That’s why your rendering strategy matters:

✅ Server-side rendering (SSR): The HTML is generated on the server and sent fully formed to the browser. Crawlers (and users) see the final page right away.

✅ Incremental static regeneration (ISR): Think of it as SSR with caching. Pages are pre-rendered and served as static files, but they’re updated periodically behind the scenes.

✅ Static site generation (SSG): The page is built ahead of time during deployment. What gets served is a plain HTML file, no rendering needed on the server or client.

⚠️ Client-side rendering (CSR): The browser loads a mostly empty HTML shell, then uses JavaScript to fetch data and build the page. OpenAI’s bots don’t execute JavaScript, so they’ll miss anything built this way.

This doesn’t mean you have to give up interactivity. JavaScript can still power modals, hover effects, live search, and dynamic enhancements. Note that your foundational content (articles, product specs, and docs) needs to be present at page load.

If it’s not, OpenAI’s bots won’t see it. And if they can’t see it, they can’t cite it.

This also affects how your brand shows up in LLM outputs.

If OpenAI’s bots can’t access your core pages, then:

GPTBot won’t include your site in training data.
OAI-SearchBot won’t surface you in real-time answers.
ChatGPT-User won’t retrieve your content during browsing sessions.

Worse still, if your competitors serve equivalent content via SSR or SSG, their answers may be the only ones referenced, regardless of whether they’re more accurate, more current, or better written.

The good news: This is fixable.

You don’t need to abandon your JavaScript framework. You just need to serve meaningful HTML at load.

Here’s what’s recommended:

Ensure all critical content is included in your initial HTML response.
Use SSR or pre-rendered pages wherever your stack allows.
Test your site with curl or wget to confirm what’s visible without JS.
Avoid placing key content, like product descriptions, article bodies, or documentation, inside components that only render after JavaScript loads(such as hydration-only or dynamically imported components).
For Next.js: Use getServerSideProps or getStaticProps for content-heavy routes.

In short: if a bot can’t see your value, neither can the model. And in the LLM era, invisibility is worse than irrelevance.

Structuring content for LLMs

Whether content is trained on, indexed, or retrieved in real time, visibility isn’t just about access. It’s about interpretability.

OpenAI’s systems parse full page renders—including HTML, JavaScript, images (via OCR), and transcripts. However, LLMs are selective. They prioritize content that is structured.

Clarity in structure helps AI systems quickly understand what your content is and how it should be interpreted. This is where schema markup comes in.

How to use schema markup for LLM visibility

Schema markup provides machine-readable context about your content. It can be implemented using:

Microdata: Small metadata tags directly embedded within your HTML elements (e.g. <span itemprop="author">Jane Doe</span>).
JSON-LD: A <script type="application/ld+json"> block that aggregates all your metadata in one place.

Google prefers JSON-LD, and it’s typically easier to manage and validate.

Schema helps with:

Interpretability: LLMs benefit from context-rich metadata that clarifies relationships between content types (e.g. BlogPosting > author > datePublished).
Citation and training: Structured content is more likely to be surfaced accurately in tools like ChatGPT and Perplexity.
Search visibility: Google uses schema to enhance rich results, which in turn influences your likelihood of being seen, cited, or retrieved in AI interfaces.

Example: BlogPosting schema (JSON-LD)

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "How OpenAI Crawls and Indexes Content",
  "author": {
    "@type": "Person",
    "name": "Jane Doe"
  },
  "datePublished": "2025-05-03",
  "image": "https://example.com/images/openai-crawlers.jpg",
  "publisher": {
    "@type": "Organization",
    "name": "Daydream",
    "logo": {
      "@type": "ImageObject",
      "url": "https://example.com/logo.png"
    }
  },
  "description": "A technical deep dive into how OpenAI crawlers work and how to optimize your site for visibility."
}
</script>

Example: BlogPosting schema (Microdata)

<article itemscope itemtype="https://schema.org/BlogPosting">
  <h1 itemprop="headline">How OpenAI Crawls and Indexes Content</h1>
  <p>By <span itemprop="author">Jane Doe</span></p>
  <p>Published on <time itemprop="datePublished" datetime="2025-05-03">May 3, 2025</time></p>
  <img src="https://example.com/images/openai-crawlers.jpg" itemprop="image" />
  <p itemprop="description">A technical deep dive into how OpenAI crawlers work and how to optimize your site for visibility.</p>
</article>

You can validate your markup using Google's Rich Results Test or Schema.org’s validator.

At a minimum, consider marking up:

Articles or blog posts
Product pages
FAQ sections
How-to guides

These structures help both traditional search engines and AI agents surface your most important content correctly and keep it aligned with the intent behind the page.

What you should know about llms.txt

A new file format—llms.txt—has been circulating in AI indexing discussions. It's proposed as a way to help LLMs understand website content more efficiently, by providing a markdown-formatted "table of contents" for your domain (e.g., yourdomain.com/llms.txt).

In theory, it acts like a sitemap for AI models, listing relevant resources like documentation, guides, and product specs.

In practice? Adoption is limited. No major LLM provider has officially stated that they use llms.txt for crawling or inference.

Even Anthropic, which publishes a public llms.txt file, hasn’t confirmed their crawlers rely on it. The same goes for Claude, Perplexity, and OpenAI. There’s currently no technical evidence (e.g. server logs or bot behavior) showing that these files are parsed, prioritized, or indexed by any LLM.

So is it worth doing?

Maybe, but with caveats. If you already maintain structured documentation, compiling an llms.txt is easy. It’s a low-lift, no-risk addition. But there’s no proven upside. It won’t guarantee better citations, faster visibility, or more accurate summaries in ChatGPT or Claude.

Here’s a recommended example of the format:

# Title
Brief description of the site.

## Section Name
- [Link Title](https://link_url): Optional description
- [Link Title](https://link_url/sub_path): Optional description

## Another Section
- [Link Title](https://link_url): Optional description

Bottom line:

If you're curious, it’s safe to experiment.
If you're resourcing for AI visibility, focus on fundamentals: clean HTML, structured schema, working sitemaps, and server-side rendering.
If llms.txt ever becomes a real standard, you’ll be ahead—but for now, it’s not one.

Closing the loop: Prepare for every layer

Visibility in OpenAI isn’t about blue links. It’s about presence in the model’s worldview.

Your content is now a training input, a citation source, and an interactive response layer. It informs the answers given to millions of users across enterprise tools, developer environments, and general search.

Consider this: when users ask ChatGPT for "the best open-source web analytics platforms," the model isn’t just retrieving links. It’s synthesizing. If your brand has been seen, cited, and structured well enough to be included, you become part of the answer.

At daydream, we help companies prepare their content for the new era of AI discovery. From crawlability to LLM-specific formatting, our team ensures your site is both accessible and influential in model-generated answers. If you’re ready to optimize for how people actually search today, let’s talk.