Name: Webcontent API: Extract, Summarize & Screenshot URLs
Brand: quickpod
SKU: api-quickpod-aiquick-speech-to-text
Availability: InStock
Author: API Market

Anil Singh

# Webcontent API

## Overview
The Webcontent API provides robust tools for interacting with web content, offering capabilities for URL article extraction, extractive summarization, and full-page screenshot capture. This API empowers developers to programmatically transform live web pages into structured data and visual artifacts.

Whether you need to capture a visual snapshot for quality assurance, extract clean article text for content analysis, or generate concise summaries from lengthy articles, the Webcontent API streamlines these processes. It’s ideal for automating data collection, monitoring web changes, enriching applications with web content, and generating documentation from live URLs.

## Features
- **Full-Page Screenshot Capture:** Render and capture high-fidelity PNG screenshots of any URL, including support for dynamic content.
- **Configurable Browser Sizing:** Define custom viewport `width` and `height` for screenshots, enabling responsive design testing and diverse visual outputs.
- **Page-Wait Controls:** Specify `wait_ms` to ensure all dynamic elements, JavaScript, and asynchronously loaded content are rendered before a screenshot is taken.
- **URL Article Extraction:** Fetch a URL and automatically extract the main article body, title, and essential page metadata, stripping away navigation, ads, and other irrelevant elements.
- **Extractive Summarization (LSA):** Generate intelligent, concise summaries of web articles using Latent Semantic Analysis (LSA), with control over the number of sentences.
- **Binary PNG Output for Screenshots:** Receive screenshot data directly as binary PNG, ready for storage, attachment, or further image processing.
- **Robust URL Handling:** Supports redirects and handles most standard web pages.

## Use Cases
- **Digital Marketing & SEO:** Automate competitor analysis by capturing screenshots of their landing pages and extracting content updates. Monitor SEO performance by regularly snapshotting SERP results or tracking changes to your own website's content and layout.
- **QA & Web Testing:** Integrate screenshot capabilities into automated testing pipelines to perform visual regression testing, verify responsive designs across different viewports, and document UI bugs with precise visual evidence.
- **Content Curation & Analysis:** Build news aggregators or research tools that automatically extract clean articles and generate summaries, enabling quick content review, sentiment analysis, or populating knowledge bases.
- **Archiving & Compliance:** Create a historical archive of web pages for legal documentation, regulatory compliance, or digital preservation, ensuring you have immutable records of website content at specific points in time.
- **Website Monitoring:** Set up automated alerts that trigger screenshots or content extraction when specific changes are detected on critical web pages, helping to track defacements, content updates, or performance issues.

## API Documentation

### Base URL
All API requests should be made to:
`https://prod.api.market/api/v1/quickpod/aiquick-speech-to-text`

### Authentication
Authentication is handled via API key. Include your API key in the request header:
```
x-api-market-key: YOUR_API_MARKET_KEY
```

### Endpoints

#### 1. Service health check
Check the health status of the API service.

- **Endpoint:** `GET /api/v1/quickpod/aiquick-speech-to-text/health`
- **Summary:** Service health check
- **Description:** Returns a simple status indicating if the service is operational.
- **Authentication:** `x-api-market-key` header required.

**Request Example (cURL):**
```bash
curl -X GET "https://prod.api.market/api/v1/quickpod/aiquick-speech-to-text/health" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY"
```

**Response (200 OK):**
```json
{
  "ok": true,
  "service": "Webcontent API"
}
```

#### 2. Fetch a URL and return clean article text with metadata
Extract the main article content, title, and metadata from any given URL.

- **Endpoint:** `POST /api/v1/quickpod/aiquick-speech-to-text/v1/url/extract`
- **Summary:** Fetch a URL and return clean article text with metadata
- **Description:** Fetches the URL with a browser-like User-Agent, extracts the main article body using `trafilatura`, and returns the title, clean text, and basic metadata. Handles redirects and most JavaScript-free pages.
- **Authentication:** `x-api-market-key` header required.

**Request Body:**
```json
{
  "url": "string"
}
```
- `url` (string, required): The URL of the article to extract.
  - Example: `https://techcrunch.com/2026/07/22/article`

**Request Example (cURL):**
```bash
curl -X POST "https://prod.api.market/api/v1/quickpod/aiquick-speech-to-text/v1/url/extract" \
  -H "Content-Type: application/json" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY" \
  -d '{
    "url": "https://techcrunch.com/2026/07/22/article"
  }'
```

**Response (200 OK):**
```json
{
  "url": "https://example.com/article",
  "title": "Example Article Title",
  "text": "The article body text extracted from the page...",
  "length": 1842,
  "content_type": "text/html; charset=utf-8",
  "final_url": "https://example.com/article"
}
```
- **413 Request Entity Too Large:** Response exceeds 5 MB content limit.
- **502 Bad Gateway:** Failed to fetch the URL or HTTP error from target.

#### 3. Fetch, extract, and return an LSA extractive summary
Summarize the main content of a URL using Latent Semantic Analysis.

- **Endpoint:** `POST /api/v1/quickpod/aiquick-speech-to-text/v1/url/summarize`
- **Summary:** Fetch, extract, and return an LSA extractive summary
- **Description:** Fetches the URL, extracts article text, then applies Latent Semantic Analysis (LSA) to select the most informative sentences as a summary. Falls back to first N sentences if LSA fails on short content.
- **Authentication:** `x-api-market-key` header required.

**Request Body:**
```json
{
  "url": "string",
  "sentences": "integer"
}
```
- `url` (string, required): The URL of the article to summarize.
  - Example: `https://bbc.com/news/article-123`
- `sentences` (integer, optional): Number of sentences in the extractive summary.
  - Default: `5`
  - Minimum: `1`, Maximum: `20`

**Request Example (cURL):**
```bash
curl -X POST "https://prod.api.market/api/v1/quickpod/aiquick-speech-to-text/v1/url/summarize" \
  -H "Content-Type: application/json" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY" \
  -d '{
    "url": "https://bbc.com/news/article-123",
    "sentences": 3
  }'
```

**Response (200 OK):**
```json
{
  "url": "https://bbc.com/news/article-123",
  "title": "Article Title",
  "summary": "Key sentence one. Key sentence two. Key sentence three.",
  "summary_sentences": 3,
  "input_length": 3200,
  "summary_length": 210
}
```
- **422 Unprocessable Entity:** No extractable text found at URL.
- **502 Bad Gateway:** Failed to fetch URL.

#### 4. Capture a full-page PNG screenshot of any URL
Generate a PNG image of a specified web page.

- **Endpoint:** `GET /api/v1/quickpod/aiquick-speech-to-text/v1/screenshot`
- **Summary:** Capture a full-page PNG screenshot of any URL
- **Description:** Uses headless Chromium via Playwright to render the page and capture a full-page or viewport screenshot. Waits for DOM content to load plus an optional additional delay for dynamic content.
- **Authentication:** `x-api-market-key` header required.

**Query Parameters:**
- `url` (string, required): The URL to capture.
  - Example: `https://example.com`
- `width` (integer, optional): Viewport width in pixels.
  - Default: `1280`
  - Minimum: `320`, Maximum: `1920`
- `height` (integer, optional): Viewport height in pixels.
  - Default: `800`
  - Minimum: `240`, Maximum: `1080`
- `full_page` (boolean, optional): Set to `true` to capture the entire scrollable page height, or `false` to capture only the visible viewport.
  - Default: `true`
- `wait_ms` (integer, optional): Additional milliseconds to wait after page load for dynamic content to render.
  - Default: `1500`
  - Minimum: `0`, Maximum: `8000`

**Request Example (cURL):**
```bash
curl -X GET "https://prod.api.market/api/v1/quickpod/aiquick-speech-to-text/v1/screenshot?url=https://example.com&width=1024&height=768&full_page=false&wait_ms=2000" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY" \
  --output "screenshot.png"
```

**Response (200 OK):**
- **Content-Type:** `image/png` (binary data)
- **Headers:**
    - `X-URL`: The URL that was screenshotted.
    - `X-Width`: The width used for the screenshot.
    - `X-Full-Page`: `true` or `false` indicating if full page was captured.

- **422 Unprocessable Entity:** Failed to render URL (invalid URL, timeout, or network error).

## Status Codes
- **200 OK:** The request was successful, and the response body contains the requested data.
- **201 Created:** A resource was successfully created (not typically used by this API's current endpoints).
- **400 Bad Request:** The request was malformed or contained invalid input parameters. Check your JSON body or query parameters.
- **401 Unauthorized:** Missing or invalid `x-api-market-key` in the request header. Ensure your API key is correct and present.
- **403 Forbidden:** Valid API key but insufficient permissions to access the requested resource.
- **404 Not Found:** The requested resource or endpoint path does not exist.
- **413 Request Entity Too Large:** (Specific to `/v1/url/extract`) The extracted content from the target URL exceeded the 5 MB limit.
- **422 Unprocessable Entity:** (Specific to `/v1/url/summarize` and `/v1/screenshot`) The server understood the request but could not process it due to semantic errors, such as no extractable text found at the URL for summarization or failure to render the URL for a screenshot.
- **429 Too Many Requests:** You have exceeded the API's rate limits. Slow down your requests and implement retry logic.
- **500 Internal Server Error:** A generic error occurred on the API server. This typically indicates a problem with the service itself. If persistent, please contact support.
- **502 Bad Gateway:** (Specific to `/v1/url/extract` and `/v1/url/summarize`) The API failed to fetch the target URL, or the target server returned an HTTP error.
- **503 Service Unavailable:** The API service is temporarily unable to handle the request, often due to maintenance or overload. Retry after a short delay.

## Rate Limits
To ensure fair usage and service stability, the Webcontent API implements rate limiting. If you exceed the allowed number of requests within a given timeframe, you will receive a `429 Too Many Requests` HTTP status code.

It is recommended to implement exponential backoff with jitter in your client-side retry logic when encountering `429` responses. For higher throughput requirements, please contact API.market support for custom plans.

## Error Handling
Effective error handling is crucial for building robust applications with the Webcontent API.

- **Interpreting Error Responses:** When an error occurs, the API will return a non-2xx HTTP status code. For most errors, the response body will be a JSON object containing an `error` field with details about the issue, helping you diagnose and resolve problems.
- **Common Errors & Resolution:**
    - **401 Unauthorized:** Double-check your `x-api-market-key`. Ensure it's correctly placed in the header and is valid.
    - **400 Bad Request:** Review the endpoint's documentation for required parameters and correct data types. Check your request body/query parameters for typos or missing fields.
    - **422 Unprocessable Entity:** For `/v1/url/extract` or `/v1/url/summarize`, this might mean the URL provided doesn't contain extractable article content. For `/v1/screenshot`, it could mean the URL is invalid, timed out during rendering, or had a network error. Try the URL in a browser to confirm accessibility.
    - **5xx Server Errors (500, 502, 503):** These indicate issues on the server side or with upstream services. These are often transient. Implement retry logic for these errors.
- **Recommended Retry Strategies:** For transient errors like `429 Too Many Requests`, `500 Internal Server Error`, `502 Bad Gateway`, and `503 Service Unavailable`, implement a retry mechanism with exponential backoff.
    - Start with a small delay (e.g., 1 second).
    - Increase the delay exponentially for subsequent retries (e.g., 1s, 2s, 4s, 8s...).
    - Add a small random jitter to delays to prevent "thundering herd" issues.
    - Set a maximum number of retries or a total retry timeout to prevent indefinite waiting.

**Error Response Example:**
```json
{
  "error": {
    "code": "invalid_parameter",
    "message": "The parameter 'url' is required",
    "status": 400
  }
}
```



#### cURL Example
This example demonstrates a synchronous request to the `/v1/url/extract` endpoint using cURL.

```bash
# cURL example for synchronous API calls

# Example: Extract article content
curl -X POST "https://prod.api.market/api/v1/quickpod/aiquick-speech-to-text/v1/url/extract" \
  -H "Content-Type: application/json" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY" \
  -d '{
    "url": "https://techcrunch.com/2026/07/22/article"
  }'

# Example: Get a screenshot (output to a file)
curl -X GET "https://prod.api.market/api/v1/quickpod/aiquick-speech-to-text/v1/screenshot?url=https://example.com&width=1400&height=900&full_page=true&wait_ms=2500" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY" \
  --output "example_screenshot.png"

# Example: Summarize an article
curl -X POST "https://prod.api.market/api/v1/quickpod/aiquick-speech-to-text/v1/url/summarize" \
  -H "Content-Type: application/json" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY" \
  -d '{
    "url": "https://www.theguardian.com/world/2023/jan/01/new-year-message-ukraine-russia",
    "sentences": 4
  }'
```


## FAQ

**Q: What types of URLs can the API process?**

A: The API can process most public web URLs. For extraction and summarization, it works best with article-like content. For screenshots, it can render any accessible web page. Private pages or those requiring complex authentication (beyond simple headers or basic auth) are not directly supported.

**Q: Can I use this API for commercial purposes?**

A: Yes, the libraries used within the Webcontent API are open-source and permitted for commercial use (Apache-2.0, MIT licenses).

**Q: Is there a limit to the size of articles or web pages?**

A: For article extraction, there is a content limit of 5 MB for the extracted text. For screenshots, while there isn't a strict "size" limit on the page content itself, very large or complex pages might take longer to render, potentially hitting a timeout if `wait_ms` is too low or a client-side timeout is reached.

**Q: How accurate is the extractive summarization?**

A: The API uses the Latent Semantic Analysis (LSA) algorithm for extractive summarization. LSA identifies the most semantically important sentences. Its effectiveness can vary depending on the article's structure and complexity, but it generally provides a good, fact-preserving summary.

**Q: Can I specify the browser version or rendering engine?**

A: The API uses headless Chromium via Playwright, providing a consistent rendering environment. Specific browser versions or alternative engines cannot be chosen by the user.

**Q: What if a URL requires JavaScript to render content?**

A: The screenshot endpoint fully supports JavaScript rendering, as it uses a headless browser. You can use the `wait_ms` parameter to ensure dynamic content has loaded before the screenshot is taken. For the `extract` and `summarize` endpoints, they are primarily designed for static or server-side rendered content and may not capture text fully loaded by client-side JavaScript.

## Contact Us
If you have any questions or need assistance, you can reach out to us through the embedded chat on the bottom right corner of the page or send an email to hello@magicapi.com.

Leverage the Webcontent API to programmatically extract clean article text, generate LSA summaries, and capture full-page PNG screenshots from any URL. Perfect 

Webcontent API: Extract, Summarize & Screenshot URLs

Free