Name: Article Extractor API: URL to Markdown and Clean Text
Brand: compasslab-1
SKU: api-compasslab-1-article-extractor
Availability: InStock
Author: API Market

WPTOOLS

# Article Extractor API

## Overview

The Article Extractor API fetches a public web page or PDF link and returns its main content as clean Markdown and plain text, with menus, ads, cookie banners and footers removed. Along with the content it returns the page title, author, publication date, language, site name, canonical URL, lead image and word count, and on request the page's links, images with alt text and raw HTML. It has 2 endpoints. The extract endpoint handles one URL per call, and the batch endpoint takes up to 3 URLs, processes them in parallel and returns one result per URL in the same shape, so a single bad URL never fails the whole batch.

It is built for developers feeding articles into LLMs, RAG pipelines and vector databases, read-later and newsletter digest tools, research tools and SEO content audits. Every result carries a `status` that is either `ok` or a specific reason such as `blocked_by_robots`, `blocked_by_site`, `not_found`, `dead_link`, `timeout`, `empty_content` or `unsupported_type`, a `failure_source` (`site`, `service` or `unknown`) that says which side caused a failure, a `warnings` list, and a `checks` object that tracks whether the page was fetched, the content extracted and the quality checks passed. A `metadata` block gives the fetch time, elapsed milliseconds, HTTP status, content type, page size, redirect count and whether the result came from cache.

## Features

* **2 endpoints**: Single-page extraction and a batch endpoint for up to 3 URLs per call, where a batch counts as one request.
* **Simple inputs**: One absolute `http` or `https` URL (up to 2,048 characters) for the extract endpoint, or a JSON list of 1 to 3 URLs for the batch endpoint.
* **Markdown and text output**: Choose `markdown`, `text` or `both`. Markdown keeps headings, lists, bold and italic text, which suits LLM and RAG input.
* **Article metadata**: Title, description, site name, author, publication date (ISO 8601), language (ISO 639-1), canonical URL, final URL after redirects, lead image and word count.
* **Optional extras**: Links (with an `internal` flag), images with alt text, and the raw HTML, each switched on with its own flag.
* **PDF links**: A PDF URL returns the PDF's text in the same response shape, with `extraction_method` set to `pdf`.
* **Length control**: `max_chars` cuts the content to a set length and adds a `content_truncated` warning, while `word_count` still reports the full article.
* **Data source and method**: Pages are fetched live, robots.txt is respected, and logins, paywalls and anti-bot challenges are never bypassed. Results are cached in memory for 10 minutes.
* **Rate limit**: 1 request per second on FREE, 5 on PRO, 10 on ULTRA and 20 on MEGA.

## Use Cases

* **RAG and LLM input**: Turn article URLs into clean Markdown before chunking and embedding them in a vector database.
* **Read-later and digest apps**: Show the readable body of a saved link with its title, author, date and lead image.
* **Newsletter curation**: Pull the text and metadata of a few articles at once with the batch endpoint and summarise them.
* **Research tools**: Collect the text of papers and reports from PDF links and web pages in one format.
* **SEO content audits**: Check word counts, titles, canonical URLs and language across a list of pages.
* **Link health checks**: Use `status` and `failure_source` to spot dead links, removed pages and pages that block crawlers.

## API Documentation

### Base URL

All requests go through the API.market gateway at this base URL:

```
https://prod.api.market/api/v1/compasslab-1/article-extractor
```

### Authentication

Every request needs your API.market key in the `x-api-market-key` header:

```
x-api-market-key: YOUR_API_KEY
```

Replace `YOUR_API_KEY` with your actual API key.

### Endpoints

#### 1. Extract one page

Fetches one public page or PDF link and returns its main content as Markdown and/or plain text, with the title, author, publication date, language, canonical URL, lead image and word count. The `status` field says whether extraction succeeded, and when it did not, `error` and `failure_source` explain why.

**Method:** `GET`

**Endpoint:** `/v1/extract`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `url` | query | Absolute `http` or `https` URL of one page or PDF. URL-encode it. Minimum 4 characters. | string | Yes | - | 2,048 characters |
| `output` | query | Which content fields to fill: `markdown`, `text` or `both`. | string | No | `both` | - |
| `include_links` | query | Also return the page's links, each with its text and an `internal` flag. | boolean | No | `false` | - |
| `include_images` | query | Also return the page's images with their alt text. | boolean | No | `false` | - |
| `include_html` | query | Also return the raw page HTML in `content_html`. This can be large. | boolean | No | `false` | - |
| `render` | query | `auto` or `never`. Only applies where browser rendering is available (the JS tier), where `auto` renders pages that come back empty in a real browser and `never` skips it. | string | No | - | - |
| `max_chars` | query | Cut the content to this many characters and add a `content_truncated` warning. Minimum 100. | integer | No | - | 1,000,000 |

**Example Request:**

```bash
curl -X GET "https://prod.api.market/api/v1/compasslab-1/article-extractor/v1/extract?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FApplication_programming_interface&output=markdown&max_chars=1500" \
 -H "x-api-market-key: YOUR_API_KEY"
```

**Example Response (200 OK):**

```json
{
 "url": "https://en.wikipedia.org/wiki/Application_programming_interface",
 "final_url": "https://en.wikipedia.org/wiki/Application_programming_interface",
 "canonical_url": "https://en.wikipedia.org/wiki/API",
 "status": "ok",
 "failure_source": null,
 "error": null,
 "title": "API - Wikipedia",
 "description": null,
 "site_name": "Wikimedia Foundation, Inc.",
 "author": "Authority control databases International GND National United States France BnF data Israel Other Yale LUX",
 "published_at": "2001-07-30T00:00:00",
 "language": "en",
 "image": null,
 "content_markdown": "**application programming interface** (**API**) is a connection between computers or between computer programs. It is a type of software interface, offering a service to other pieces of software.\\[1\\]\\[2\\] A document or standard that describes how to build such a connection or interface is called an *API specification*. A computer system that meets this standard is said to *implement* or *expose* an API. The term API may refer either to the specification or to the implementation.\n\nIn contrast to a user interface, which connects a computer to a person, an application programming interface connects computers or pieces of software to each other. It is not intended to be used directly by a person (the end user) other than a computer programmer\\[2\\] who is incorporating it into software. An API is often made up of different parts which act as tools or services that are available to the programmer. A program or a programmer that uses one of these parts is said to *call* that portion of the API. The calls that make up the API are also known as subroutines, methods, requests, or endpoints. An API specification *defines* these calls, meaning that it explains how to use or implement them.\n\nOne of the main purpose of APIs is to hide the internal details of how a system works, exposing only those parts a programmer will find useful and keeping them consistent even if the internal details later change. An API may be custom-built for a particular pair of sy",
 "content_text": null,
 "content_html": null,
 "word_count": 5566,
 "extraction_method": "trafilatura",
 "warnings": [
 "content_truncated"
 ],
 "links": null,
 "images": null,
 "checks": {
 "fetched": true,
 "extracted": true,
 "quality_passed": true
 },
 "metadata": {
 "fetched_at": "2026-10-17T14:54:28Z",
 "elapsed_ms": 993,
 "cached": false,
 "http_status": 200,
 "content_type": "text/html",
 "bytes": 391082,
 "redirects": 0
 }
}
```

Because `output=markdown` was requested, `content_text` is `null`, and because `max_chars=1500` cut the content, `warnings` contains `content_truncated` while `word_count` still reports the full article.

#### 2. Extract a batch of pages

Extracts up to 3 pages in one call, in parallel. Each URL gets its own result and `status` in the same order as the request, and a `summary` counts how many results have each status. A batch counts as one request.

**Method:** `POST`

**Endpoint:** `/v1/extract/batch`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `urls` | body | List of 1 to 3 absolute `http` or `https` URLs, processed in parallel. | array of strings | Yes | - | 3 items |
| `output` | body | Which content fields to fill: `markdown`, `text` or `both`. | string | No | `both` | - |
| `include_links` | body | Return each page's links. | boolean | No | `false` | - |
| `include_images` | body | Return each page's images with alt text. | boolean | No | `false` | - |
| `include_html` | body | Return the raw page HTML. Off by default to save bandwidth. | boolean | No | `false` | - |
| `render` | body | `auto` or `never`. Only applies where browser rendering is available (the JS tier), where `auto` renders a page in a real browser when the normal fetch finds no content. | string | No | `never` | - |
| `max_chars` | body | Cut `content_markdown` and `content_text` to this many characters and add a `content_truncated` warning. `word_count` stays the full article's. Minimum 100. | integer | No | - | 1,000,000 |

**Example Request:**

```bash
curl -X POST "https://prod.api.market/api/v1/compasslab-1/article-extractor/v1/extract/batch" \
 -H "x-api-market-key: YOUR_API_KEY" \
 -H "Content-Type: application/json" \
 -d '{"urls": ["https://en.wikipedia.org/wiki/JSON"], "output": "text", "max_chars": 800}'
```

**Example Response (200 OK):**

```json
{
 "count": 1,
 "summary": {
 "ok": 1
 },
 "results": [
 {
 "url": "https://en.wikipedia.org/wiki/JSON",
 "final_url": "https://en.wikipedia.org/wiki/JSON",
 "canonical_url": "https://en.wikipedia.org/wiki/JSON",
 "status": "ok",
 "failure_source": null,
 "error": null,
 "title": "JSON - Wikipedia",
 "description": null,
 "site_name": "Wikimedia Foundation, Inc.",
 "author": "Authority control databases International GND FAST National United States Israel Other Yale LUX",
 "published_at": "2005-03-06T00:00:00",
 "language": "en",
 "image": null,
 "content_markdown": null,
 "content_text": "| JavaScript Object Notation | | \n|---|---|\n| Filename extension | .json | \n| Internet media type | application/json | \n| Type code | TEXT | \n| Uniform Type Identifier (UTI) | public.json | \n| Type of format | Data interchange | \n| Extended from | JavaScript | \n| Standard | STD 90 (RFC 8259), ECMA-404, ISO/IEC 21778:2017 | \n| Open format? | Yes | \n| Website | json.org | \nJSON (JavaScript Object Notation, pronounced // or //) is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of name\u2013value pairs and arrays (or other serializable values). It is a commonly used data format with diverse uses in electronic data interchange, including that of web applications with servers.\nJSON is a programming language-indep",
 "content_html": null,
 "word_count": 5113,
 "extraction_method": "trafilatura",
 "warnings": [
 "content_truncated"
 ],
 "links": null,
 "images": null,
 "checks": {
 "fetched": true,
 "extracted": true,
 "quality_passed": true
 },
 "metadata": {
 "fetched_at": "2026-10-17T14:54:31Z",
 "elapsed_ms": 712,
 "cached": false,
 "http_status": 200,
 "content_type": "text/html",
 "bytes": 477701,
 "redirects": 0
 }
 }
 ]
}
```

## Status Codes

| Code | Status | Description |
|---|---|---|
| 200 | OK | The request was processed. Check the `status` field of each result: `ok` means the content was extracted, and any other value (for example `blocked_by_robots` or `timeout`) explains why it was not. In a batch, one failed URL does not fail the others. |
| 400 | Bad Request | The input could not be used, for example a URL the service cannot accept. The JSON body has a stable `error` code, a readable `detail` and a `request_id`. |
| 401 | Unauthorized | The `x-api-market-key` value is invalid or expired. |
| 403 | Forbidden | Your subscription does not include this endpoint. |
| 404 | Not Found | The path does not match an endpoint of this API, for example a typo in `/v1/extract`. A page that returns 404 on the target site is reported as `status: "not_found"` in the result instead. |
| 422 | Unprocessable Entity | A parameter is missing or invalid, such as a missing `url`, more than 3 URLs in a batch, or `max_chars` below 100. The response says which one. |
| 429 | Too Many Requests | You went over your plan's requests-per-second limit, or you have used all the calls in your plan. |
| 500 | Internal Server Error | Something failed while processing the request. Retry after a short delay. |

## Rate Limits

Each plan has its own per-second limit: FREE allows 1 request per second, PRO 5, ULTRA 10 and MEGA 20. The plans include 100, 5,000, 25,000 and 100,000 calls respectively, and a batch call of up to 3 URLs counts as one call. Requests above the per-second limit, or after the plan's calls are used up, return `429 Too Many Requests`. Back off before retrying and honour any `Retry-After` header. Response time depends on the target site: in our tests the two example calls took about 1 second each through the gateway, with `elapsed_ms` of 993 and 712. The seller reports a median of about 0.6 seconds per page on its own server.

## FAQ

**Why did a page come back empty?**

Look at `status` and `warnings`. `empty_content` with a `needs_javascript` warning means the page is built in the browser by JavaScript. Other warnings such as `paywall_detected`, `login_required` and `bot_challenge` mean the content is behind something the API will not bypass, and `blocked_by_site` means the site refused the request.

**Can it read PDFs?**

Yes. A PDF link returns the PDF's text in the same response shape with `extraction_method` set to `pdf`. Warnings such as `pdf_has_no_text_layer`, `pdf_encrypted` and `pdf_truncated` tell you when a PDF could not be read in full.

**Does it follow links or crawl a site?**

No. One URL in, one page out. Use the batch endpoint to send up to 3 URLs in one call, and `include_links` if you want the page's links to decide what to fetch next.

**Does it respect robots.txt?**

Yes. A page that robots.txt disallows returns `status: "blocked_by_robots"` with `failure_source: "site"`, and the content fields stay empty.

**What does `failure_source` mean?**

`site` means the site refused the request or the page is gone, `service` means the problem was on the extractor's side, and `unknown` is used for timeouts. It is `null` when `status` is `ok`.

**Is there a size limit?**

Yes. Pages over 5 MB of HTML or 15 MB of PDF return `status: "too_large"`. Use `max_chars` to keep the content you receive short when you only need the start of a long article.

**Are results cached?**

Results are kept in a short-lived memory cache for 10 minutes. `metadata.cached` tells you whether a response was served from cache, and `metadata.fetched_at` gives the time the page was fetched.

## Contact Us

If you have any questions or need assistance, you can reach out to us through the embedded chat on the bottom right corner of the page or send an email to hello@magicapi.com.

Article Extractor API turns a public URL or PDF link into clean Markdown or text with title, author, date, language and a status for every page.

Article Extractor API: URL to Markdown and Clean Text

FREE