seoder

See what crawlers get from your page

Paste a URL. seoder fetches it the way ten search and AI crawlers do, reads your robots.txt and llms.txt, follows the redirects, and compares the raw HTML against the rendered page so you can tell which parts need JavaScript.

Use the exact URL you want indexed, including the path. Checks run against public pages only.

Report

Nothing to show yet. The report lands here and stays on this page, so you can copy the URL and send it to whoever owns the robots.txt.

Fetching the page. This takes a few seconds.

The check did not finish.

What the report covers

Six answers, from the same crawl. Each one links to the page explaining how it was measured, so a number you show a colleague can be checked rather than trusted.

  • The bot matrix: what each named crawler receives, one row each, with the status code and whether robots.txt allows the fetch.
  • JavaScript dependence, measured by comparing the raw HTML against the rendered page rather than guessing from the framework.
  • Your robots.txt rules, parsed per crawler, with the line that matched.
  • Your llms.txt, if you have one, checked against the format.
  • The redirect chain, with every hop and its status code.
  • The structured data the page carries, and whether it parses.

Read how each verdict is decided, including request counts, timeouts and what the tool will not do.

Crawlers in the matrix

Each one has a page with its user agent string, its robots token, how to verify a request really came from it, and what blocking it costs you.

  • Amazonbot

    Crawling to improve Amazon products and services, including answers surfaced to customers, and it may be used to train Amazon AI models.

  • Applebot-Extended

    Controlling whether content already crawled by Applebot may be used to train Apple's generative foundation models.

  • Bingbot

    Crawling for the Bing index, which also backs Copilot answers and several licensed search products.

  • Bytespider

    Collecting web content for ByteDance. The operator publishes no statement of purpose, so any description of what it feeds is inference.

  • CCBot

    Building the free public Common Crawl archive, which many research projects and several model training pipelines read downstream.

  • ClaudeBot

    Collecting public web content for model development. Search and user-directed retrieval run under separate tokens.

  • Google-Extended

    Controlling whether crawled content may be used for training Gemini models and for grounding in Gemini Apps and Vertex AI.

  • Googlebot

    Crawling for Google Search indexing and the Search-derived surfaces built on that index.

  • GPTBot

    Collecting web content used to train OpenAI's foundation models. It does not drive ChatGPT search citations.

  • Meta-ExternalAgent

    Crawling for use cases such as training foundation AI models and improving products by indexing content directly.

  • PerplexityBot

    Indexing pages so they can appear as cited sources in Perplexity answers. Perplexity states it does not collect content for model training.

Common questions

How many requests does a check send to my site?

One request per mode. A full report is a raw fetch, a rendered fetch, and one fetch per crawler in the matrix, so a default run is under ten requests in total. The counts and the politeness rules are on the how it works page.

Does the checker try to get around a block?

No. Each fetch identifies itself as the crawler being tested and follows the robots.txt rules that apply to it. A blocked result means blocked, which is the only way the report is worth reading.

Do I need an account?

No. The checker is free and takes no sign up. Rate limits apply per address and per target host so one person cannot point the tool at one site all day.

Can I check a staging site behind a login?

No. The checker fetches the page the way a public crawler would, with no cookies and no credentials, so anything behind an auth wall comes back as whatever the crawler would get, usually a redirect or a 403.