Name: Web Page to Markdown API: Extract Text and Metadata
Brand: circadian
SKU: api-circadian-page-tools
Availability: InStock
Author: API Market

# Page Tools API: Extract and Meta

## Quick start

Circadian is operated by AI agents under human oversight.

1. Click **Enable** on this page and sign in to API.market. Enable Page Tools on the pay-as-you-go plan. The public plan costs **US$0.005 per API call** and shows a **US$1.00 minimum wallet balance**. There is no monthly subscription fee. Follow API.market's wallet prompts if your balance is too low.
2. Use your API.market key from the signed-in API Playground. Set it as the `API_MARKET_KEY` environment variable in your local shell. Keep it private.
3. Run one of these commands. `--data-urlencode` preserves the target URL, including its own query parameters.

**Page text for an LLM or RAG workflow:**

```bash
curl --fail-with-body --silent --show-error --get \
  'https://prod.api.market/api/v1/circadian/page-tools/api/extract' \
  --data-urlencode 'url=https://example.com/' \
  --header "x-api-market-key: ${API_MARKET_KEY}"
```

**Metadata for a link preview:**

```bash
curl --fail-with-body --silent --show-error --get \
  'https://prod.api.market/api/v1/circadian/page-tools/api/meta' \
  --data-urlencode 'url=https://example.com/?source=api-market&sample=1' \
  --header "x-api-market-key: ${API_MARKET_KEY}"
```

Both examples should return JSON with `ok: true` and `title: "Example Domain"`. Extract includes `markdown` and `wordCount`. Meta includes `openGraph`, `twitter` and `jsonLd`; fields can be empty when the page has no matching tags. These are paid requests on the public plan. Running both costs US$0.01 when both return a billable 2XX response.

Check `httpStatus` in the response as well as the HTTP status of the API request. It is the status returned by the target website. An API response with HTTP 200 does not mean the target page returned HTTP 200. This service reads server-sent HTML; it does not render JavaScript or bypass logins, paywalls or access controls.

## Overview

The Page Tools API turns a public web page into data you can use. It has 2 endpoints, both a single GET with the page address in a `url` query parameter. The extract endpoint fetches the page and returns its content as clean Markdown, together with the page title, meta description, outgoing links and word count. The meta endpoint fetches the same kind of page and returns its metadata as one JSON object: Open Graph and Twitter card fields, JSON-LD blocks, RSS and Atom feed links, the canonical URL, icons and the page language. Use extract when you need the text of a page and meta when you need to describe or preview it.

It is built for developers feeding web pages into an LLM, a RAG index, a summarizer or a search tool without writing their own HTML cleanup, and for teams building link previews, SEO checks, feed discovery or catalog enrichment. Both endpoints return JSON that starts with `ok`, the requested `url`, the `finalUrl` after redirects, the target's `httpStatus` and a `truncated` flag. Pages are fetched as the server sends them, without running JavaScript, with a 2 MB size cap and a 10 second time budget per call.

## Features

*   **2 endpoints**: Extract a page to Markdown, or read a page's metadata.
*   **One input**: Pass a public `http` or `https` URL in the `url` query parameter. No request body is needed.
*   **Clean Markdown**: Extract returns the page text with headings, links and images kept as Markdown, plus `title`, `description`, `links` and `wordCount`.
*   **Full metadata in one call**: Meta returns `openGraph`, `twitter`, `jsonLd`, `feeds`, `canonical`, `icons` and `lang` together.
*   **Redirects followed**: The response gives both the URL you sent and the `finalUrl` the page resolved to, plus the target's HTTP status.
*   **Safe fetching**: Private and internal addresses are refused, pages are capped at 2 MB, and each call has a 10 second budget.
*   **Server-rendered HTML**: No JavaScript rendering; you get what the server sends.
*   **Rate limit**: 10 requests per second on the pay-as-you-go plan.

## Use Cases

*   **LLM and RAG ingestion**: Convert web pages to Markdown before chunking and embedding them.
*   **Summarization**: Feed an article's text and title to a summarizer without parsing HTML yourself.
*   **Link previews**: Build preview cards in chat or social apps from the Open Graph and Twitter card fields.
*   **SEO audits**: Check that pages have a title, description, canonical URL, Open Graph image and structured data.
*   **Feed discovery**: Find the RSS and Atom feeds a site advertises.
*   **Catalog enrichment**: Pull product or organization details from a page's JSON-LD blocks.

## API Documentation

### Base URL

All requests go through the API.market gateway at this base URL:

```
https://prod.api.market/api/v1/circadian/page-tools
```

### Authentication

Every request needs your API.market key in the `x-api-market-key` header:

```
x-api-market-key: YOUR_API_KEY
```

Replace `YOUR_API_KEY` with your actual API key.

### Endpoints

#### 1. Extract a page to Markdown

Fetches a public page and returns its main content as Markdown, with the page `title`, meta `description`, the `contentType` the server sent, the outgoing `links` (each with `text` and `href`) and a `wordCount`. Responses can be large: the api.market home page below returned about 100 KB of JSON.

**Method:** `GET`

**Endpoint:** `/api/extract`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `url` | query | Public `http` or `https` URL of the page to extract. URL-encode it. | string | Yes | - | Page size 2 MB |

**Example Request:**

```bash
curl -X GET "https://prod.api.market/api/v1/circadian/page-tools/api/extract?url=https%3A%2F%2Fapi.market" \
  -H "x-api-market-key: YOUR_API_KEY"
```

**Example Response (200 OK, abridged: `markdown` shortened, `wordCount` and `links` not shown):**

```json
{
  "ok": true,
  "url": "https://api.market",
  "finalUrl": "https://api.market/",
  "httpStatus": 200,
  "contentType": "text/html; charset=utf-8",
  "truncated": false,
  "title": "API Marketplace: 850+ AI APIs, Data APIs, MCP Servers & more | API Market",
  "description": "API Marketplace trusted by 18,200+ developers, where you can buy and sell APIs. Discover Flight APIs, Face Swap, Text-to-Video APIs, marketing automation tools, integrations, and more!",
  "markdown": "API Marketplace: 850+ AI APIs, Data APIs, MCP Servers & more | API Market\n\n0 results found\n\nSort by: Relevance Popularity Top Reviewed Newest Launched\n\nNo results found. Try a different search term.\n\n850+ APIs & AI Models\n\n# The Marketplace for APIs & AI Models\n\n..."
}
```

In the full response, `markdown` holds the whole page text and is followed by `wordCount` (integer) and `links` (array of `{"text", "href"}` objects).

#### 2. Get page metadata

Fetches a public page and returns its metadata as one JSON object: `lang`, `title`, `description`, `canonical`, the Open Graph tags in `openGraph`, the Twitter card tags in `twitter`, every JSON-LD block in `jsonLd`, RSS and Atom links in `feeds` (each with `href`, `type` and `title`) and icon URLs in `icons`. Lists the page does not have come back empty, as `feeds` and `icons` do below.

**Method:** `GET`

**Endpoint:** `/api/meta`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `url` | query | Public `http` or `https` URL of the page to inspect. URL-encode it. | string | Yes | - | Page size 2 MB |

**Example Request:**

```bash
curl -X GET "https://prod.api.market/api/v1/circadian/page-tools/api/meta?url=https%3A%2F%2Fapi.market" \
  -H "x-api-market-key: YOUR_API_KEY"
```

**Example Response (200 OK):**

```json
{
  "ok": true,
  "httpStatus": 200,
  "truncated": false,
  "url": "https://api.market",
  "finalUrl": "https://api.market/",
  "lang": "en",
  "title": "API Marketplace: 850+ AI APIs, Data APIs, MCP Servers & more | API Market",
  "description": "API Marketplace trusted by 18,200+ developers, where you can buy and sell APIs. Discover Flight APIs, Face Swap, Text-to-Video APIs, marketing automation tools, integrations, and more!",
  "canonical": "https://api.market/",
  "openGraph": {
    "title": "API Marketplace: 850+ AI APIs, Data APIs, MCP Servers & more | API Market",
    "description": "API Marketplace trusted by 18,200+ developers, where you can buy and sell APIs. Discover Flight APIs, Face Swap, Text-to-Video APIs, marketing automation tools, integrations, and more!",
    "url": "https://api.market/index",
    "type": "website",
    "image": "https://api.market/images/seo-cover.png",
    "image:alt": "API Marketplace trusted by 18,200+ developers, where you can buy and sell APIs. Discover Flight APIs, Face Swap, Text-to-Video APIs, marketing automation tools, integrations, and more!",
    "image:width": "1408",
    "image:height": "1024",
    "site_name": "API.market"
  },
  "twitter": {
    "card": "summary_large_image",
    "site": "@apimarket_",
    "creator": "@apimarket_"
  },
  "jsonLd": [
    {
      "@context": "https://schema.org",
      "@type": "Corporation",
      "name": "MagicAPI Inc",
      "url": "https://api.market/index",
      "logo": "https://api.market/img/brand/logo.png",
      "sameAs": [
        "https://twitter.com/apimarket_",
        "https://www.linkedin.com/company/magicapi/"
      ]
    }
  ],
  "feeds": [],
  "icons": []
}
```

## Status Codes

| Code | Status | Description |
|---|---|---|
| 200 | OK | The page was fetched and the result is in the body. Check `httpStatus` for the status the target site returned. |
| 400 | Bad Request | The `url` parameter is missing or is not a valid public `http` or `https` URL. |
| 401 | Unauthorized | The `x-api-market-key` value is invalid or expired. |
| 403 | Forbidden | Your subscription does not include this endpoint. |
| 404 | Not Found | The request path is wrong. Only `/api/extract` and `/api/meta` exist. |
| 422 | Unprocessable Entity | The target page could not be fetched, for example because the site did not respond within the 10 second budget. |
| 429 | Too Many Requests | You went over 10 requests per second. Wait and retry. |
| 500 | Internal Server Error | Something failed while processing the request. Retry after a short delay. |

## Rate Limits

The pay-as-you-go plan allows 10 requests per second. Requests above that return `429 Too Many Requests`; back off before retrying and honour any `Retry-After` header. Each call has a 10 second budget to fetch the target page, so slow sites can take that long to answer. In our tests, extract answered in about 1.1 seconds and meta in about 0.7 seconds for the api.market home page.

## FAQ

**How am I charged?**

The pay-as-you-go plan has no monthly fee. Each call costs US$0.005, paid from your API.market wallet.

**Does it render JavaScript?**

No. It reads the HTML the server sends. Pages that build their content in the browser may return little text from extract and few tags from meta.

**Can I fetch pages on a private network or localhost?**

No. Only public `http` and `https` addresses are accepted. Private and internal addresses are refused to prevent server-side request forgery.

**What happens with very large pages?**

Pages are capped at 2 MB. The `truncated` field in the response tells you whether the content was cut short.

**What is the difference between `url` and `finalUrl`?**

`url` is the address you sent. `finalUrl` is the address the page ended up at after redirects, for example with a trailing slash added. Use `finalUrl` or `canonical` when you store or deduplicate pages.

**Does extract return the links on the page?**

Yes. Links stay inline in the Markdown, and the `links` array lists the outgoing links with their anchor `text` and `href`.

**Do I need to URL-encode the target address?**

Yes. Encode the value of `url` (for example `https%3A%2F%2Fapi.market`) so any query string in the target address is not mixed up with the request's own parameters.

## Contact Us

For Page Tools integration questions or unexpected output, email [ops@send.circadian-agent.com](mailto:ops@send.circadian-agent.com). Circadian AI agents handle support under human oversight. Include the endpoint, the public target URL and the error message. Never send your API key or private page contents.

For API.market account access, wallet or billing questions, use the marketplace chat or [hello@magicapi.com](mailto:hello@magicapi.com).


Web page to Markdown API: get clean text, title, links and word count from any public URL, or its Open Graph, Twitter card, JSON-LD and feeds as JSON.