Name: Robots.txt API: Sitemap Parser and AI Crawler Check
Brand: leadharbor
SKU: api-leadharbor-robots-txt-sitemap
Availability: InStock
Author: API Market

Mohammed Samaan

# Robots.txt and Sitemap API

## Overview

The Robots.txt and Sitemap API reads a website's robots.txt and sitemaps and turns them into structured JSON. It has one endpoint, `/v1/market/robots`, with three operations chosen by the `op` parameter. `op=robots` fetches and parses a site's robots.txt and returns the rule groups, crawl-delay, the declared sitemaps and a policy summary for 22 well-known AI crawler tokens such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and PerplexityBot. `op=check` answers one question: may this user agent fetch this path, and which rule decided it. `op=sitemap` reads a sitemap, or the sitemaps the site declares in robots.txt, including gzip files and sitemap indexes, and lists the URLs with their lastmod, changefreq and priority. Rule matching and status handling follow RFC 9309.

It is built for teams that run crawlers, scrapers and AI agents and need to respect site rules before fetching, SEO tools that audit crawl rules and sitemaps, and publishers who want to see how their robots.txt treats AI crawlers. You pass a site URL or a bare domain. `op=robots` returns the parsed groups plus an `ai_crawlers` list where each entry names the operator, the crawler's purpose, whether it is allowed, whether the file names it explicitly, its disallowed paths and whether the operator documents that it honours robots.txt. `op=check` returns `allowed` as true or false with the `matched_rule` and `matched_agent`. `op=sitemap` returns the sitemap `kind`, the URL list, the total `url_count` and a `truncated` flag. Every response also carries a `meta` object that you can ignore.

## Features

*   **1 endpoint, 3 operations**: `op=robots` parses robots.txt, `op=check` tests one path for one bot, and `op=sitemap` reads sitemaps.
*   **Simple inputs**: A full site URL (`url`) or a bare domain (`domain`), plus `user_agent` and `path` for checks and `limit` and `max_children` for sitemaps.
*   **AI crawler policy**: One call shows how a site treats 22 AI-related crawler tokens, with the operator, kind (training, search, user fetch, control token or general), purpose and a link to the operator's own crawler documentation.
*   **RFC 9309 matching**: Longest-match rule resolution with `*` wildcards and `$` end anchors. A missing robots.txt (4xx) means no restrictions, and a server error means everything is treated as disallowed.
*   **Sitemap reading**: Handles plain and gzip sitemaps and sitemap indexes on the same host, reading up to 10 child sitemaps and returning up to 5,000 URLs.
*   **Data source**: The API fetches the live robots.txt and sitemap files from the site at request time. The AI crawler list is built from each operator's published documentation.
*   **Rate limit**: 10 requests per second on every plan.

## Use Cases

*   **Polite crawling**: Check `op=check` before each fetch so your crawler or agent skips paths the site disallows.
*   **AI crawler audits**: Show a publisher which AI bots their robots.txt blocks, which it allows by default and which it never mentions.
*   **SEO audits**: Review crawl rules, crawl-delay and declared sitemaps for a client site in one call.
*   **Sitemap discovery**: Pull the URL list of a site from its declared sitemaps to seed a crawl or a content inventory.
*   **Competitor research**: See how other sites in your market treat GPTBot, ClaudeBot or Google-Extended.
*   **Monitoring**: Re-run `op=robots` on a schedule and alert when a site starts blocking a crawler you rely on.

## API Documentation

### Base URL

All requests go through the API.market gateway at this base URL:

```
https://prod.api.market/api/v1/leadharbor/robots-txt-sitemap
```

### Authentication

Every request needs your API.market key in the `x-api-market-key` header:

```
x-api-market-key: YOUR_API_KEY
```

Replace `YOUR_API_KEY` with your actual API key.

### Endpoints

All three operations use the same endpoint and the same parameters. The `op` parameter selects the operation, and each example below shows one operation.

#### 1. Read robots.txt rules and sitemaps

Returns the parsed robots.txt of a site (`op=robots`), a yes or no answer for one bot and one path (`op=check`), or the URLs in a site's sitemaps (`op=sitemap`).

**Method:** `GET`

**Endpoint:** `/v1/market/robots`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `op` | query | Operation: `robots`, `check` or `sitemap`. | string | No | `robots` | - |
| `url` | query | A site URL, for example `https://www.nytimes.com`. For `op=sitemap` this can be the URL of a sitemap file. Send `url` or `domain`. | string | No | - | - |
| `domain` | query | A bare domain such as `nytimes.com`, instead of `url`. | string | No | - | - |
| `user_agent` | query | `op=check` only: the crawler name to test, for example `GPTBot`. | string | No | - | - |
| `path` | query | `op=check` only: the URL path to test. | string | No | `/` | - |
| `limit` | query | `op=sitemap` only: maximum number of URLs to return, 1 to 5000. | integer | No | `1000` | 5000 |
| `max_children` | query | `op=sitemap` only: number of child sitemaps to read from a sitemap index, 1 to 10. | integer | No | `5` | 10 |

**Example Request (op=robots):**

```bash
curl -X GET "https://prod.api.market/api/v1/leadharbor/robots-txt-sitemap/v1/market/robots?op=robots&url=https%3A%2F%2Fwww.nytimes.com" \
  -H "x-api-market-key: YOUR_API_KEY"
```

**Example Response (200 OK, abridged):**

```json
{
  "op": "robots",
  "url": "https://www.nytimes.com/robots.txt",
  "final_url": "https://www.nytimes.com/robots.txt",
  "status": 200,
  "exists": true,
  "effective": "rules",
  "sitemaps": [
    "https://www.nytimes.com/sitemaps/new/news.xml.gz",
    "https://www.nytimes.com/sitemaps/new/sitemap.xml.gz"
  ],
  "groups": [
    {
      "agents": [
        "*",
        "googlebot"
      ],
      "rules": [
        {
          "type": "disallow",
          "path": "/ads/"
        },
        {
          "type": "disallow",
          "path": "/adx/bin/"
        },
        {
          "type": "disallow",
          "path": "/athletic/wp/wp-admin/"
        }
      ],
      "crawl_delay": null
    },
    {
      "agents": [
        "gptbot"
      ],
      "rules": [
        {
          "type": "disallow",
          "path": "/"
        }
      ],
      "crawl_delay": null
    }
  ],
  "truncated": false,
  "ai_crawlers": [
    {
      "token": "GPTBot",
      "operator": "OpenAI",
      "kind": "training",
      "purpose": "crawls content that may be used to train generative AI models",
      "honors_robots_txt": "documented",
      "docs_url": "https://developers.openai.com/api/docs/bots",
      "allowed": false,
      "explicit": true,
      "disallowed_paths": [
        "/"
      ]
    },
    {
      "token": "OAI-SearchBot",
      "operator": "OpenAI",
      "kind": "search",
      "purpose": "surfaces sites in ChatGPT search",
      "honors_robots_txt": "documented",
      "docs_url": "https://developers.openai.com/api/docs/bots",
      "allowed": false,
      "explicit": true,
      "disallowed_paths": [
        "/"
      ]
    }
  ]
}
```

The full response lists all 25 declared sitemaps, all 63 rule groups and every AI crawler entry, and ends with the `meta` object. In `ai_crawlers`, `explicit: true` means the file names that crawler in its own group, `explicit: false` means the crawler falls back to the `*` group, and `honors_robots_txt` is `documented`, `not_guaranteed` or `not_stated` according to the operator's documentation.

**Example Request (op=check):**

```bash
curl -X GET "https://prod.api.market/api/v1/leadharbor/robots-txt-sitemap/v1/market/robots?op=check&domain=nytimes.com&user_agent=GPTBot&path=%2F2026%2F01%2F01%2Fworld%2Fstory.html" \
  -H "x-api-market-key: YOUR_API_KEY"
```

**Example Response (200 OK):**

```json
{
  "op": "check",
  "url": "https://nytimes.com/robots.txt",
  "status": 200,
  "effective": "rules",
  "user_agent": "GPTBot",
  "path": "/2026/01/01/world/story.html",
  "allowed": false,
  "matched_rule": {
    "type": "disallow",
    "path": "/"
  },
  "matched_agent": "gptbot",
  "crawl_delay": null,
  "meta": {
    "user": null
  }
}
```

`matched_rule` is the rule that decided the answer and `matched_agent` is the group it came from. Here GPTBot has its own group that disallows `/`, so the story path is blocked.

**Example Request (op=sitemap):**

```bash
curl -X GET "https://prod.api.market/api/v1/leadharbor/robots-txt-sitemap/v1/market/robots?op=sitemap&domain=github.com&limit=5&max_children=2" \
  -H "x-api-market-key: YOUR_API_KEY"
```

**Example Response (200 OK):**

```json
{
  "op": "sitemap",
  "discovered_from": "robots.txt",
  "declared_sitemaps": [
    "https://github.com/sitemap_marketing.xml"
  ],
  "url": "https://github.com/sitemap_marketing.xml",
  "kind": "urlset",
  "urls": [
    {
      "loc": "https://github.com/",
      "lastmod": null,
      "changefreq": null,
      "priority": null
    },
    {
      "loc": "https://github.com/about",
      "lastmod": null,
      "changefreq": null,
      "priority": null
    },
    {
      "loc": "https://github.com/about/diversity",
      "lastmod": null,
      "changefreq": null,
      "priority": null
    },
    {
      "loc": "https://github.com/about/diversity/report",
      "lastmod": null,
      "changefreq": null,
      "priority": null
    },
    {
      "loc": "https://github.com/about/leadership",
      "lastmod": null,
      "changefreq": null,
      "priority": null
    }
  ],
  "url_count": 376,
  "truncated": true,
  "children": [],
  "meta": {
    "user": null
  }
}
```

`discovered_from: "robots.txt"` means the sitemap was found through the site's robots.txt. `url_count` is the total number of URLs in the sitemap (376 here), and `truncated: true` means only the first `limit` URLs were returned. When the sitemap is an index, `children` lists the child sitemaps that were read. The `meta` object is metadata you can ignore.

## Status Codes

| Code | Status | Description |
|---|---|---|
| 200 | OK | The request succeeded. A site with no robots.txt still returns `200`, treated as no restrictions under RFC 9309. |
| 400 | Bad Request | A field is invalid or missing (for example no `url` or `domain`, or an unknown `op`), or the address is not public. |
| 401 | Unauthorized | The `x-api-market-key` value is invalid or expired. |
| 403 | Forbidden | Your subscription does not include this endpoint. |
| 404 | Not Found | The request path is wrong. Check that it is `/v1/market/robots` under the base URL. |
| 422 | Unprocessable Entity | The document fetched for `op=sitemap` is not a readable sitemap. |
| 429 | Too Many Requests | You went over 10 requests per second, or you have used all the calls in your plan. |
| 500 | Internal Server Error | Something failed while processing the request. Retry after a short delay. |
| 502 | Bad Gateway | The target site could not be reached. Check the URL or retry later. |

## Rate Limits

The Free trial, Starter and PAYG plans each allow 10 requests per second. The 7-day Free trial includes 500 calls, Starter includes 10,000 calls with extra calls at $0.001 each, and PAYG charges $0.001 per call with no subscription. Every call counts as one call whichever `op` you use. Requests above the per-second limit, or after the Free trial's calls are used up, return `429 Too Many Requests`. Back off before retrying and honour the `Retry-After` header, which says how many seconds to wait. Each call fetches the target site live, so response time depends on that site; in our tests the three operations answered in 0.6 to 1.0 seconds.

## FAQ

**Does a block in robots.txt stop an AI crawler from fetching the page?**

No. robots.txt is a request to crawlers, not access control. The `honors_robots_txt` field shows what each operator says: user-initiated fetchers such as ChatGPT-User and Perplexity-User are marked `not_guaranteed`, so a block may not be respected when a person asks the assistant to open a page.

**What happens when a site has no robots.txt or its server fails?**

The API follows RFC 9309. A missing robots.txt (a 4xx answer) means there are no restrictions, so every path is allowed. A server error (5xx) means everything is treated as disallowed. If the site cannot be reached at all, the API returns `502`.

**How does `op=check` decide between conflicting rules?**

It picks the group that matches the `user_agent` (falling back to `*`), then applies the longest matching rule, with `*` wildcards and `$` end anchors. The response shows the `matched_rule` and `matched_agent` so you can see why a path was allowed or blocked.

**Is the AI crawler list complete?**

No. It covers 22 well-known AI-related tokens taken from the operators' own documentation. It is not exhaustive and operators sometimes rename crawlers, so also check the raw `groups` for any bot you care about.

**Should I send `url` or `domain`?**

Either works for `op=robots` and `op=check`. For `op=sitemap`, send `domain` (or a site URL) to read the sitemaps declared in robots.txt, or send the URL of a specific sitemap file to read just that one.

**How many sitemap URLs can I get in one call?**

Up to 5,000 with `limit` (default 1,000). For a sitemap index, `max_children` reads up to 10 child sitemaps on the same host (default 5). `url_count` gives the total and `truncated` tells you whether more URLs exist than were returned.

**Can I check internal or private hosts?**

No. Only public addresses are fetched. A private, local or otherwise non-public address returns `400`.

## Contact Us

If you have any questions or need assistance, you can reach out to us through the embedded chat on the bottom right corner of the page or send an email to hello@magicapi.com.


Robots.txt API that parses crawl rules, checks if GPTBot or any bot may fetch a path, lists 22 AI crawler policies and reads gzip sitemaps.

Robots.txt API: Sitemap Parser and AI Crawler Check

Free trial