Name: Distill API: Convert Web Pages and HTML to Markdown
Brand: hybrid-forge-lab
SKU: api-hybrid-forge-lab-distill
Availability: InStock
Author: API Market

# Distill HTML to Markdown API

## Overview

The Distill HTML to Markdown API turns a web page into clean Markdown for LLMs, RAG pipelines and agents. `GET /v1/markdown` fetches a public URL and returns its main content, `POST /v1/markdown` does the same for a URL or for raw HTML you send, and `GET /v1/health` tells you the service is up. Headings, lists, tables, code blocks and links are kept, while menus, cookie banners, sidebars and footers are dropped, and every link is resolved to an absolute URL.

It is built for AI engineers, RAG and search developers and agent builders who need page content in a form a model can read without wasting tokens on navigation. Each answer carries the `markdown`, the `title`, `author`, `published_date`, `language` and `site_name` when the page has them, a `word_count`, a `char_count`, an `approx_tokens` estimate, a `links` list and a `status` that says honestly when a site's robots.txt or AI-crawler rules said no.

## Features

*   **Three endpoints**: convert a URL with GET, convert a URL or raw HTML with POST, and a health check.
*   **Main content only**: navigation, cookie banners, sidebars, ads and footers are removed.
*   **Structure kept**: headings, lists, tables and code blocks come through as Markdown.
*   **Token budget**: `max_chars` cuts the output at a paragraph boundary, and `approx_tokens` tells you the size.
*   **Metadata**: title, author, published date, language and site name when the page provides them.
*   **Respects site owners**: robots.txt and AI-crawler opt-outs are honoured, with a clear `status` when a page is skipped.
*   **Rate limit**: up to 10 requests per second on every plan.

## Use Cases

*   **RAG ingestion**: convert documentation and articles into Markdown chunks for a vector store.
*   **AI agents**: give an agent a readable version of any page it browses.
*   **Content migration**: move pages from an old CMS into Markdown files.
*   **Summaries and monitoring**: feed page text to a model to summarise or track changes.
*   **Rendered HTML**: send the HTML your own crawler or headless browser produced and get Markdown back without a second fetch.

## API Documentation

### Base URL

All requests go through the API.market proxy at this base URL:

```
https://prod.api.market/api/v1/hybrid-forge-lab/distill
```

### Authentication

Every request needs your API.market key in the `x-api-market-key` header:

```
x-api-market-key: YOUR_API_KEY
```

Replace `YOUR_API_KEY` with your actual API key.

### Endpoints

#### 1. Health check

Returns `ok`, the service version and its uptime. Use it for monitoring; it does not convert anything.

**Method:** `GET`

**Endpoint:** `/v1/health`

**Parameters:**

This endpoint takes no parameters.

**Example Request:**

```bash
curl -X GET 'https://prod.api.market/api/v1/hybrid-forge-lab/distill/v1/health' \
  -H 'x-api-market-key: YOUR_API_KEY'
```

**Example Response (200 OK):**

```json
{
  "ok": true,
  "version": "0.2.2",
  "uptime_s": 1
}
```

#### 2. Convert a URL

Fetches a public page and returns its main content as Markdown with its metadata, word and token counts and links. The example converts a Wikipedia article with `max_chars=1500`, so `truncated` is `true`.

**Method:** `GET`

**Endpoint:** `/v1/markdown`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `url` | query | The public page to convert, URL-encoded. | string | Yes | - | - |
| `include_links` | query | Keep links in the Markdown and return them in `links`. | boolean | No | `true` | - |
| `include_images` | query | Keep images in the Markdown. | boolean | No | `false` | - |
| `include_tables` | query | Keep tables as Markdown tables. | boolean | No | `true` | - |
| `max_chars` | query | Cut the Markdown at a paragraph boundary after about this many characters. | integer | No | - | - |

**Example Request:**

```bash
curl -X GET 'https://prod.api.market/api/v1/hybrid-forge-lab/distill/v1/markdown?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FMarkdown&include_links=true&include_images=false&include_tables=true&max_chars=1500' \
  -H 'x-api-market-key: YOUR_API_KEY'
```

**Example Response (200 OK):**

```json
{
  "url": "https://en.wikipedia.org/wiki/Markdown",
  "final_url": "https://en.wikipedia.org/wiki/Markdown",
  "status": "ok",
  "detail": null,
  "http_status": 200,
  "title": "Markdown - Wikipedia",
  "author": null,
  "published_date": "2005-08-09",
  "language": "en",
  "site_name": "Wikimedia Foundation, Inc.",
  "description": null,
  "markdown": "# Markdown\n\n| Markdown |  |\n|---|---|\n| [Filename extensions](https://en.wikipedia.org/wiki/Filename_extension) | `.md`, `.markdown` |\n| [Internet media type](https://en.wikipedia.org/wiki/Media_type) | `text/markdown` |\n| [Uniform Type Identifier (UTI)](https://en.wikipedia.org/wiki/Uniform_Type_Identifier) | `net.daringfireball.markdown` |\n| UTI conformation | `public.utf8-plain-text` |\n| [Magic number](https://en.wikipedia.org/wiki/File_format#Magic_number) | None |\n| Developed by | [John Gruber](https://en.wikipedia.org/wiki/John_Gruber) |\n| Initial release | March 9, 2004 |\n| [Latest release](https://en.wikipedia.org/wiki/Software_release_life_cycle) | 1.0.1 December 17, 2004 |\n| Type of format | [Open file format](https://en.wikipedia.org/wiki/Open_file_format) |\n| Extended to | [pandoc](https://en.wikipedia.org/wiki/Pandoc), [MultiMarkdown](https://en.wikipedia.org/wiki/MultiMarkdown), [Markdown Extra](https://en.wikipedia.org/wiki/Markdown_Extra), [CommonMark](https://en.wikipedia.org/wiki/Markdown#CommonMark), [RMarkdown](https://en.wikipedia.org/wiki/RMarkdown) |\n| Website | [daringfireball.net/projects/markdown/](https://daringfireball.net/projects/markdown/) |",
  "word_count": 63,
  "char_count": 1190,
  "approx_tokens": 298,
  "truncated": true,
  "links": [
    {
      "text": "Filename extensions",
      "url": "https://en.wikipedia.org/wiki/Filename_extension"
    },
    {
      "text": "Internet media type",
      "url": "https://en.wikipedia.org/wiki/Media_type"
    }
  ],
  "cached": false,
  "took_ms": 523
}
```

#### 3. Convert a URL or raw HTML

Accepts either a `url` to fetch or raw `html` to convert, with the same options as the GET endpoint. When you send HTML, nothing is fetched, and `base_url` is used to turn relative links into absolute ones.

**Method:** `POST`

**Endpoint:** `/v1/markdown`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `url` | body | A public page to fetch. Send either `url` or `html`. | string | No | - | - |
| `html` | body | Raw HTML to convert, up to 5 MB. Send either `url` or `html`. | string | No | - | - |
| `base_url` | body | Used to resolve relative links and images in `html`. | string | No | - | - |
| `include_links` | body | Keep links in the Markdown and return them in `links`. | boolean | No | `true` | - |
| `include_images` | body | Keep images in the Markdown. | boolean | No | `false` | - |
| `include_tables` | body | Keep tables as Markdown tables. | boolean | No | `true` | - |
| `max_chars` | body | Cut the Markdown at a paragraph boundary after about this many characters. | integer | No | - | - |

**Example Request:**

```bash
curl -X POST 'https://prod.api.market/api/v1/hybrid-forge-lab/distill/v1/markdown' \
  -H 'x-api-market-key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
  "html": "<html><head><title>How to brew pour-over coffee</title><meta name='\''author'\'' content='\''Ana Lopes'\''></head><body><header><nav><a href='\''/'\''>Home</a> <a href='\''/shop'\''>Shop</a></nav></header><div class='\''cookie-banner'\''>We use cookies. Accept?</div><main><article><h1>How to brew pour-over coffee</h1><p>Pour-over gives you a clean, bright cup. You need a dripper, a filter and freshly ground beans. See our <a href='\''/guides/grind'\''>grind size guide</a>.</p><h2>What you need</h2><ul><li>20 g of coffee</li><li>320 g of water at 94 C</li><li>A paper filter</li></ul><h2>Ratios</h2><table><thead><tr><th>Cups</th><th>Coffee</th><th>Water</th></tr></thead><tbody><tr><td>1</td><td>20 g</td><td>320 g</td></tr><tr><td>2</td><td>40 g</td><td>640 g</td></tr></tbody></table><h2>Steps</h2><ol><li>Rinse the filter with hot water.</li><li>Bloom the grounds with 40 g of water for 30 seconds.</li><li>Pour the rest in slow circles.</li></ol><pre><code>ratio = water / coffee  # 16</code></pre></article></main><aside>Related: French press guide</aside><footer>Copyright 2026 Bean Co. <a href='\''/privacy'\''>Privacy</a></footer></body></html>",
  "base_url": "https://beanco.example",
  "include_links": true,
  "include_images": false,
  "include_tables": true
}'
```

**Example Response (200 OK):**

```json
{
  "url": "https://beanco.example",
  "final_url": null,
  "status": "ok",
  "detail": null,
  "http_status": null,
  "title": "How to brew pour-over coffee",
  "author": "Ana Lopes",
  "published_date": null,
  "language": null,
  "site_name": "beanco.example",
  "description": null,
  "markdown": "# How to brew pour-over coffee\n\nPour-over gives you a clean, bright cup. You need a dripper, a filter and freshly ground beans. See our [grind size guide](https://beanco.example/guides/grind).\n\n## What you need\n\n- 20 g of coffee\n- 320 g of water at 94 C\n- A paper filter\n\n## Ratios\n\n| Cups | Coffee | Water |\n|---|---|---|\n| 1 | 20 g | 320 g |\n| 2 | 40 g | 640 g |\n\n## Steps\n\n1. Rinse the filter with hot water.\n2. Bloom the grounds with 40 g of water for 30 seconds.\n3. Pour the rest in slow circles.\n\n```\nratio = water / coffee  # 16\n```",
  "word_count": 91,
  "char_count": 539,
  "approx_tokens": 135,
  "truncated": false,
  "links": [
    {
      "text": "grind size guide",
      "url": "https://beanco.example/guides/grind"
    }
  ],
  "cached": false,
  "took_ms": 7
}
```

## Status Codes

| Code | Status | Description |
|---|---|---|
| 200 | OK | The request was handled. Check `status`: `ok` means the content is in `markdown`, while values such as `robots_txt` or `blocked` mean the site asked not to be read, and `detail` explains why. |
| 400 | Bad Request | A required parameter is missing (for example no `url` and no `html`) or a value has the wrong type. |
| 401 | Unauthorized | The `x-api-market-key` header is missing or the key is not valid. |
| 403 | Forbidden | Your key has no active subscription to this API, your plan quota is used up, or the endpoint is not part of your plan. |
| 404 | Not Found | The path is wrong. This API has `GET /v1/health`, `GET /v1/markdown` and `POST /v1/markdown`. |
| 429 | Too Many Requests | You sent more than 10 requests per second. Wait for the time in `Retry-After` and try again. |
| 500 | Internal Server Error | Something failed on the provider side. Retry once; if it keeps failing, contact us. |
| 503 | Service Unavailable | The service is briefly unavailable. Retry after a short pause. |

## Rate Limits

Every plan allows up to 10 requests per second. Monthly quotas are 15 calls on Free (a 7-day trial), 10,000 on Pro, 50,000 on Ultra and 250,000 on Mega, and each request counts as one call whatever it returns. Converting raw HTML is near instant, and fetching a URL usually takes under a second plus the time the site needs to answer. Going over 10 requests per second returns `429 Too Many Requests`. Throttle your client, back off when you get a 429 and honour the `Retry-After` header before retrying.

## FAQ

*   **Why did I get a 200 with no Markdown?**

    The site's robots.txt or AI-crawler rules asked not to be read, or the page could not be fetched. The `status` field says which (for example `robots_txt` or `blocked`) and `detail` explains it, so your pipeline can skip the page or try later.

*   **When should I send raw HTML instead of a URL?**

    When the page needs a login or JavaScript to render, fetch or render it yourself and send the result as `html`. Distill does not fetch anything in that case, and `base_url` keeps the links absolute.

*   **How do I keep the output inside my context window?**

    Set `max_chars`. The Markdown is cut at a paragraph boundary and `truncated` is `true`. `approx_tokens` gives a rough token count.

*   **How is Distill identified to websites?**

    It fetches pages as DistillBot and follows the opt-outs that site owners publish.

## Contact Us

If you have any questions or need assistance, you can reach out to us through the embedded chat on the bottom right corner of the page or send an email to hello@magicapi.com.


Turn any web page or raw HTML into clean Markdown for LLMs and RAG, with title, author, token count and links. Respects robots.txt.

Distill API: Convert Web Pages and HTML to Markdown

Free