Name: Indexability Checker API: Noindex Canonical robots.txt
Brand: glowhalo
SKU: api-glowhalo-indexability-checker-api
Availability: InStock
Author: API Market

# Indexability Checker API

## Overview

The Indexability Checker API tells you whether a URL can be indexed by Google and, if not, which declared signal blocks it. Give it a URL and it fetches the page and the site's robots.txt, and returns `indexable` (true, false or null when undetermined), a verdict and reason, the evidence, a grade from A+ to F and coded issues. Every finding carries a stable code, the specific evidence and the Google Search Central page it is based on.

It checks the signals a page and its server declare. It does not query Google's index, does not render JavaScript and does not detect soft 404 pages, so a clean report is not a guarantee of indexing. Not affiliated with Google.

## Features

*   **2 endpoints**: Check a live URL, or analyze HTML, headers and robots.txt you supply (offline, nothing is fetched).
*   **HTTP status and redirects**: Up to 5 redirects followed, with the chain reported.
*   **X-Robots-Tag**: noindex, none, nofollow, noarchive, nosnippet, unavailable_after and bot-specific forms such as `googlebot: noindex`.
*   **Meta robots**: `robots`, `googlebot` and `bingbot` meta directives.
*   **Canonical**: From the HTML head and the Link header, with self, other, missing, multiple, conflicting and relative cases, and a check of the canonical target.
*   **robots.txt**: Google's rules (user-agent group selection, longest match wins, Allow wins ties, wildcards and the end anchor).
*   **Coded findings**: Stable `code` values with a severity and a Google documentation link.
*   **Caching**: Results are cached for 15 minutes; `fresh=1` bypasses the cache.

## Use Cases

*   **SEO audits**: Find out which pages are blocked from indexing and by which signal.
*   **Publishing tools**: Warn editors before a page goes live with a stray noindex or a robots.txt block.
*   **CI pipelines**: Validate staging HTML and headers offline before a release.
*   **Migration checks**: Verify canonicals, redirects and robots rules after a site move.
*   **Monitoring**: Re-check key URLs on a schedule and alert when a page becomes non-indexable.

## API Documentation

### Base URL

All requests go through the API.market gateway at this base URL:

```
https://prod.api.market/api/v1/glowhalo/indexability-checker-api
```

### Authentication

Every request needs your API.market key in the `x-api-market-key` header:

```
x-api-market-key: YOUR_API_KEY
```

Replace `YOUR_API_KEY` with your actual API key.

### Endpoints

#### 1. Check a URL

Fetches the page (up to 5 redirects, first 1 MB, 10 second timeout, private networks refused), the site's robots.txt and, when the page canonicalizes elsewhere, the canonical target (one hop). Returns `indexable` (true, false or null when undetermined), a `verdict`, a `reason`, the evidence and issues. A value without a scheme such as `example.com` is treated as https. Results are cached for 15 minutes.

**Method:** `GET`

**Endpoint:** `/v1/check`

**Parameters:**

| Name | In | Description | Type | Required |
|---|---|---|---|---|
| `url` | query | Page to check, for example `https://example.com`. The aliases `u`, `uri`, `link`, `target`, `site`, `website`, `domain` and `host` are also accepted. | string | Yes |
| `userAgent` | query | Crawler whose robots.txt group and bot-specific directives are applied: `googlebot` (default), `bingbot` or `any`. | string | No |
| `fresh` | query | Set to `1` to bypass the 15-minute cache. | string | No |

**Example Request:**

```bash
curl -s "https://prod.api.market/api/v1/glowhalo/indexability-checker-api/v1/check?url=https://example.com" \
  -H "x-api-market-key: YOUR_API_KEY"
```

**Example Response (abridged):**

```json
{
  "indexable": true,
  "verdict": "indexable",
  "grade": "A+",
  "score": 100,
  "summary": { "errors": 0, "warnings": 0, "infos": 0 },
  "issues": [],
  "evidence": { "http": { "status": 200 } }
}
```

#### 2. Validate HTML, headers and robots.txt you send in

Runs the same analysis offline on data you send in: page HTML, response headers, robots.txt text, URL and status. Nothing is fetched, so it works on staging, password-protected or unpublished pages. Maximum 512 KB. Send at least one of `html`, `headers` or `robotsTxt`.

**Method:** `POST`

**Endpoint:** `/v1/validate`

**Body:** JSON with `html`, `headers` (object), `robotsTxt`, `url`, `status` and `userAgent`. A `url` is needed with `robotsTxt` and to resolve relative canonicals.

**Example Request:**

```bash
curl -s -X POST "https://prod.api.market/api/v1/glowhalo/indexability-checker-api/v1/validate" \
  -H "x-api-market-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/private/page", "html": "<html><head><meta name=\"robots\" content=\"noindex\"><link rel=\"canonical\" href=\"/page\"></head></html>", "headers": {"X-Robots-Tag": "googlebot: noindex"}, "robotsTxt": "User-agent: *\nDisallow: /private/", "status": 200}'
```

**Example Response (200 OK):**

```json
{
  "url": "https://example.com/private/page",
  "statusCode": 200,
  "userAgent": "googlebot",
  "indexable": false,
  "verdict": "blocked_by_robots_txt",
  "reason": "robots.txt disallows crawling this URL, so Google cannot index its content.",
  "canonicalizedTo": "https://example.com/page",
  "grade": "F",
  "score": 4,
  "summary": {
    "errors": 3,
    "warnings": 2,
    "infos": 1
  },
  "issues": [
    {
      "severity": "error",
      "code": "robots_txt_blocked",
      "message": "robots.txt disallows googlebot from crawling /private/page (matched Disallow: /private/, group \"*\"; the longest matching rule wins and Allow wins ties). Google cannot read a blocked page, so its content is not indexed; the bare URL can still appear in results if other sites link to it. Basis: Google Search Central, \"How Google interprets the robots.txt specification\".",
      "docs": "https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt"
    },
    {
      "severity": "error",
      "code": "noindex_x_robots_tag",
      "message": "The X-Robots-Tag header (googlebot: noindex) contains bot-specific \"googlebot: noindex\". Google drops the page from search results when it crawls it. Basis: Google Search Central, \"Block Search indexing with noindex\".",
      "docs": "https://developers.google.com/search/docs/crawling-indexing/block-indexing"
    },
    {
      "severity": "error",
      "code": "noindex_meta_robots",
      "message": "<meta name=\"robots\" content=\"noindex\"> contains noindex, which applies to every crawler. Google drops the page from search results when it crawls it. Basis: Google Search Central, \"Block Search indexing with noindex\".",
      "docs": "https://developers.google.com/search/docs/crawling-indexing/block-indexing"
    },
    {
      "severity": "warning",
      "code": "noindex_unseen_due_to_robots",
      "message": "The page declares noindex but robots.txt blocks crawling it. Google cannot fetch a blocked page, so it will never see the noindex and the URL can stay indexed (without content). To deindex a page, allow crawling and keep the noindex. Basis: Google Search Central, \"Block Search indexing with noindex\".",
      "docs": "https://developers.google.com/search/docs/crawling-indexing/block-indexing"
    },
    {
      "severity": "info",
      "code": "canonical_relative",
      "message": "The rel=canonical (/page) is a relative URL, resolved to https://example.com/page. Google supports it, but absolute URLs are recommended because a wrong base resolves to the wrong page. Basis: Google Search Central, \"What is URL canonicalization\".",
      "docs": "https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls"
    },
    {
      "severity": "warning",
      "code": "canonical_points_elsewhere",
      "message": "rel=canonical points to https://example.com/page (another path), not to this URL (https://example.com/private/page). The page marks itself as a duplicate, so Google usually indexes the canonical URL instead of this one. canonical is a hint, not a block. Basis: Google Search Central, \"What is URL canonicalization\".",
      "docs": "https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls"
    }
  ],
  "evidence": {
    "http": {
      "status": 200,
      "contentType": null,
      "redirects": []
    },
    "xRobotsTag": {
      "present": true,
      "raw": "googlebot: noindex",
      "directives": [
        {
          "bot": "googlebot",
          "directive": "noindex",
          "appliesToUserAgent": true
        }
      ]
    },
    "metaRobots": {
      "present": true,
      "tags": [
        {
          "name": "robots",
          "content": "noindex",
          "appliesToUserAgent": true
        }
      ]
    },
    "canonical": {
      "found": true,
      "declarations": [
        {
          "source": "html",
          "href": "/page",
          "resolved": "https://example.com/page",
          "relative": true
        }
      ],
      "target": "https://example.com/page",
      "relation": "different_path",
      "pointsTo": "other"
    },
    "robotsTxt": {
      "checked": true,
      "url": null,
      "httpStatus": null,
      "userAgentGroup": "*",
      "rulesInGroup": 1,
      "crawlAllowed": false,
      "matchedRule": {
        "directive": "Disallow",
        "pattern": "/private/"
      },
      "truncated": false
    }
  },
  "assumptions": {
    "url": "https://example.com/private/page",
    "note": "Validated offline: nothing was fetched, so the canonical target and the live robots.txt were not checked."
  },
  "checkedAt": "2026-10-21T06:38:25.199Z",
  "note": "Reports the indexing signals the page and its server declare (HTTP status, X-Robots-Tag, meta robots, canonical, robots.txt). It does not query Google's index, does not render JavaScript (tags added by scripts are not seen), does not detect soft 404s, and is not a guarantee that Google will or will not index the page. The page is fetched with this API's own user agent; userAgent only selects which rules are applied."
}
```

### Issue codes

**Errors:** `robots_txt_blocked`, `noindex_x_robots_tag`, `noindex_meta_robots`, `unavailable_after_expired`, `http_not_found`, `http_gone`, `http_client_error`, `http_access_denied`, `http_rate_limited`, `http_server_error`, `redirect_not_followable`. `http_access_denied`, `http_rate_limited` and `http_server_error` leave `indexable` null, as do `robots_txt_unavailable` and `robots_txt_unreachable`.

**Warnings:** `noindex_unseen_due_to_robots`, `robots_txt_noindex_unsupported`, `robots_txt_unavailable`, `robots_txt_unreachable`, `unknown_robots_directive`, `unavailable_after_invalid`, `empty_response`, `canonical_points_elsewhere`, `canonical_conflict`, `canonical_empty`, `canonical_invalid`, `canonical_target_redirects`, `canonical_target_error`, `canonical_target_noindex`, `canonical_target_points_elsewhere`, `canonical_target_blocked_by_robots`.

**Infos:** `redirected`, `non_html_content`, `noindex_for_other_bot`, `unavailable_after_set`, `nofollow_directive`, `noarchive_directive`, `nosnippet_directive`, `html_truncated`, `canonical_missing`, `canonical_relative`, `canonical_duplicate`, `canonical_target_not_checked`.

Verdict values: `indexable`, `canonicalized_elsewhere`, `blocked_by_robots_txt`, `noindex`, `http_error`, `unavailable_after_expired`, `undetermined`.

## Status Codes

| Code | Meaning |
|---|---|
| 200 | The report was produced. |
| 400 | Bad input (for example `missing_url`, `invalid_url`, `forbidden_host` for private-network targets). The body has `error`, `message` and a usage hint. |
| 404 | Route not found. |
| 413 | `payload_too_large` on `/v1/validate` (body over 512 KB). |
| 502 | `fetch_failed`: the target page could not be fetched. |
| 429 | The trial or monthly quota is used up (hard limit). Back off before retrying and honour any `Retry-After` header. |

## Rate Limits

The Free trial includes 500 requests. The Basic plan includes 25,000 requests per month as a hard limit: when the quota is used up the API returns `429 Too Many Requests` instead of billing overage.

## FAQ

**Does it tell me whether Google has indexed the URL?**

No. It checks the signals a page and its server declare (status, X-Robots-Tag, meta robots, canonical, robots.txt). It does not query Google's index, and a clean report is not a guarantee that Google will index the page.

**Does it render JavaScript?**

No. Robots tags or canonicals added by scripts are not seen, and soft 404 pages are not detected.

**Is the page fetched as Googlebot?**

No. The page is fetched with this API's own user agent. The `userAgent` parameter only selects which robots.txt group and bot-specific directives are applied, always interpreted the way Google documents them.

**What does `indexable: null` mean?**

The result could not be determined: HTTP 401, 403, 429 or 5xx, or a robots.txt that was unavailable. If the canonical points to another URL, `indexable` stays true with the verdict `canonicalized_elsewhere` and a warning.

**How are the grade and score calculated?**

100 minus 25 per error, 10 per warning and 1 per info (at most 5), mapped to A+ 97, A 90, B 80, C 70, D 60, F. The `indexable` field and the verdict carry the decision, and the grade only summarizes the number of findings: a single noindex scores 75.

## Contact Us

If you have any questions or need assistance, you can reach out to us through the embedded chat on the bottom right corner of the page or send an email to hello@magicapi.com.


Indexability checker API that reads a URL's meta robots, X-Robots-Tag, canonical and robots.txt and returns an indexable verdict, coded issues and a grade.

Indexability Checker API: Noindex Canonical robots.txt