Name: Robots.txt Analyzer API: Parse & Validate Crawl Rules
Brand: sitetrace
SKU: api-sitetrace-robots-analyzer
Availability: InStock
Author: API Market

Edy Appenzeller

# Robots.txt Analyzer API — Parse & Validate robots.txt Rules

## Overview
The Robots.txt Analyzer API is a powerful tool designed to fetch, parse, and validate `robots.txt` files for any given domain. It allows developers and SEO professionals to instantly extract crucial crawl rules, identify disallowed paths, discover declared sitemaps, and understand various bot directives. This API exposes a focused endpoint through api.market, providing comprehensive insights into a website's crawlability. Use it to validate crawl rules, discover sitemap URLs, and detect potential crawler blocking issues, ensuring optimal search engine visibility and compliance.

## Features
*   **Comprehensive Robots.txt Fetching**: Reliably retrieve `robots.txt` files from any public HTTP or HTTPS domain.
*   **Detailed Rule Parsing**: Parse user-agent groups, `Allow`, `Disallow`, and `Crawl-delay` rules.
*   **Sitemap Discovery**: Automatically extract all declared `Sitemap` URLs from the `robots.txt` file.
*   **Crawlability Insights**: Gain actionable insights into potential crawl blocks, allowed paths, and general bot directives.
*   **Structured JSON Output**: Receive well-organized, machine-readable JSON responses for seamless integration into dashboards, automations, and backend workflows.
*   **Asynchronous Processing**: Efficiently handle analysis requests, providing a prediction ID for status tracking and result retrieval.

## Use Cases
1.  **SEO Auditing & Compliance**: For SEO agencies and in-house teams to quickly audit website crawlability, identify restrictive rules, and ensure that critical pages are discoverable by search engine bots. This helps in pre-launch checks, ongoing site health monitoring, and ensuring compliance with SEO best practices.
2.  **Web Development & Deployment**: Developers can integrate the API into CI/CD pipelines to automatically validate `robots.txt` configurations before and after new deployments. This prevents accidental blocking of essential site sections or unintended exposure of private areas, safeguarding website integrity and SEO performance.
3.  **Content Management System (CMS) Integration**: CMS platforms can embed this API to provide users with an instant overview of their site's crawl rules, enabling easier management of search engine visibility for their content. This is particularly useful for large-scale content publishers to ensure new articles or sections are properly indexed.
4.  **Competitor Analysis & Market Research**: Analysts can utilize the API to understand the crawling strategies of competitor websites, discover their sitemap structures, and identify patterns in how they manage bot access. This provides valuable intelligence for refining one's own SEO and content strategies.
5.  **Security Monitoring & Bot Management**: Security teams can use the API to monitor for unauthorized changes to `robots.txt` files that might indicate a compromise or misconfiguration. It also helps in validating that sensitive or internal directories are correctly disallowed from public crawlers, enhancing overall web security posture.


## API Documentation

### Base URL
All API requests should be made to: `https://prod.api.market/api/v1/sitetrace/robots-analyzer`

### Authentication
Authentication is handled via API key. Include your API key in the request header:

```
x-api-market-key: YOUR_API_MARKET_KEY
```
## cURL Example

```bash
curl -X POST "https://prod.api.market/v1/sitetrace/robots-analyzer/api/robots-analyzer" \
  -H "x-api-market-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'
```

## Example Response

```json
{
  "success": true
}
```


## Status Codes
A comprehensive list of HTTP status codes and their meanings:

*   **200 OK**: Request successful. This status code is returned for a successful prediction submission, successful status checks, and when the analysis result is ready.
*   **400 Bad Request**: Invalid input parameters or a malformed request body. This occurs if the `url` field is missing, malformed, or points to an unsupported protocol or a private/local network address.
    *   **Handling:** Validate your request payload and ensure all required fields are correctly formatted.
*   **401 Unauthorized**: Missing or invalid API key. Your `x-api-market-key` header is either not provided or incorrect.
    *   **Handling:** Ensure you include a valid API key in the `x-api-market-key` header for every request.
*   **403 Forbidden**: Valid API key but insufficient permissions for the requested operation or resource, typically due to your subscription plan's limitations.
    *   **Handling:** Check your API.market subscription details or contact support if you believe this is in error.
*   **404 Not Found**: The requested resource (e.g., a prediction with a specific ID) does not exist, or the endpoint URL is incorrect.
    *   **Handling:** Verify the prediction ID and the endpoint path.
*   **429 Too Many Requests**: Rate limit exceeded for your active API.market plan.
    *   **Handling:** Respect the `Retry-After` header and implement exponential backoff.
*   **500 Internal Server Error**: A generic server-side error occurred on API.market or the underlying service.
    *   **Handling:** This usually indicates a problem beyond your control. You may retry the request after a short delay. If the issue persists, contact support.
*   **503 Service Unavailable**: The service is temporarily unable to handle the request, often due to maintenance or overload.
    *   **Handling:** Implement retry logic with exponential backoff.

## Rate Limits
The Robots.txt Analyzer API operates under rate limits controlled by api.market, which vary based on your active subscription plan (e.g., Free, Pro, Ultra, Mega). When your requests exceed the allotted limit within a minute window, you will receive a `429 Too Many Requests` HTTP status code.




## Contact Us

If you have any questions or need assistance, you can reach out to us through the embedded chat on the bottom right corner of the page or send an email to hello@magicapi.com.

Fetch, parse, and validate robots.txt files quickly. Analyze crawl rules, sitemaps, and bot directives for any domain to prevent SEO issues and ensure proper in

Robots.txt Analyzer API: Parse & Validate Crawl Rules

Basic