Name: AI News Content Crawler API: Google News Article Text
Brand: long-bear
SKU: api-long-bear-ai-news-content-crawler
Availability: InStock
Author: API Market

Fabio Borsotti

# AI News Content Crawler API

## Overview

The AI News Content Crawler searches Google News and returns the full article text from each publisher page. Send a search query or a predefined topic with one or more country and language editions, and the API finds matching news through Google News RSS, opens each publisher page in a headless browser and extracts the clean article text, title, publication date, domain and thumbnail. Results come back in one JSON response.

It is built for developers of AI agents, RAG pipelines, news monitoring tools, media analytics and model training datasets who need article text, not just headlines and links. In our tests a search for `renewable energy` in the US English edition over two days returned 3 articles in about 16 seconds, including the full text of a Nature Climate Change paper.

## Features

*   **1 endpoint**: one POST call per crawl.
*   **Search or topic**: a Google News query with operators such as `site:`, `intitle:` and quotes, or a predefined topic.
*   **Many editions**: pass several `CC:ll` pairs such as `US:en` and `IT:it` in one call.
*   **Full article text**: text is extracted from the publisher page, with a plain HTTP fallback.
*   **Rich metadata**: title, domain, published date, snippet, thumbnail and HTTP status for each article.
*   **Tunable**: `maxSites`, `daysBack` and `maxConcurrency` control size and speed.

## Use Cases

*   **RAG pipelines**: feed fresh article text into a vector store.
*   **AI agents**: let an agent read today's news on a subject.
*   **Media monitoring**: track coverage of a company, product or topic.
*   **Training datasets**: collect news text by language and country.
*   **Newsletters**: gather full articles for a daily summary.

## API Documentation

### Base URL

All requests use the following base URL.

```
https://prod.api.market/api/v1/long-bear/ai-news-content-crawler
```

### Authentication

All requests are authenticated with the `x-api-market-key` header.

```
x-api-market-key: YOUR_API_KEY
```

Replace `YOUR_API_KEY` with your actual API key.

### Endpoints

#### 1. Crawl news

Finds news for `searchQuery` or `topic` in each edition and returns the extracted articles. Send exactly one of `searchQuery` or `topic`.

**Method:** `POST`

**Endpoint:** `/crawl`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `searchQuery` | body | Google News search query. Operators such as `site:`, `intitle:`, `-` and quotes are supported. Do not send with `topic`. | string | One of | - | - |
| `topic` | body | Predefined topic: `WORLD`, `NATION`, `BUSINESS`, `TECHNOLOGY`, `ENTERTAINMENT`, `SPORTS`, `SCIENCE` or `HEALTH`. Do not send with `searchQuery`. | string | One of | - | - |
| `languageAndRegion` | body | Editions as `CC:ll` pairs, for example `US:en` or `IT:it` | array of strings | Yes | `["IT:it"]` | - |
| `maxSites` | body | Maximum unique news sites per edition | integer | No | `50` | 1000 |
| `daysBack` | body | How many days back to search. Search queries only. | integer | No | `7` | 90 |
| `chunkDays` | body | Days per search chunk | integer | No | `7` | 30 |
| `maxConcurrency` | body | Pages opened at the same time while crawling | integer | No | `5` | 10 |
| `decodeInterval` | body | Milliseconds between Google News link decodings | integer | No | `1` | 5000 |

**Example Request:**

```bash
curl -X POST 'https://prod.api.market/api/v1/long-bear/ai-news-content-crawler/crawl' \
  -H 'x-api-market-key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"searchQuery": "renewable energy", "languageAndRegion": ["US:en"], "maxSites": 3, "daysBack": 2}'
```

**Example Response (200 OK):**

```json
{
  "count": 3,
  "data": [
    {
      "acceptLanguage": "en",
      "ceid": "US:en",
      "domain": "http://www.nature.com",
      "error": null,
      "extractor": "playwright",
      "gl": "US",
      "googleNewsUrl": "https://news.google.com/rss/articles/CBMiX0FVX3lxTFB5ajRLRzZ...",
      "hl": "en",
      "htmlLang": "en",
      "inputMode": "search",
      "langs": [
        "en"
      ],
      "position": 2,
      "published": "Tue, 10 Nov 2026 02:10:16 GMT",
      "query": "renewable energy after:2026-11-08 before:2026-11-10",
      "snippet": "This work projects a decline in global emissions in 2026 as a result of clean energy and in response to conflicts. Energy demand has increased, and will continue to, with electric vehicles and solar and wind deployment mitigating emissions; accelerated deployment is needed to...",
      "statusCode": 200,
      "success": true,
      "text": "Brief Communication\nOpen access\nPublished: 10 November 2026\nA further acceleration of clean energy is needed for global CO2 emissions to peak and decline\nCorinne Le Quéré, Robbie M. Andrew, Josep G. Canadell, Pierre Friedlingstein, Robert B. Jackson & Glen P. Peters \n\nNature Climate Change (2026)...",
      "thumbnail": "data:image/jpeg;base64,/9j/4AAQSkZJRgABA...",
      "title": "A further acceleration of clean energy is needed for global CO2 emissions to peak and decline | Nature Climate Change - Nature",
      "topic": null,
      "topicRssUrl": null,
      "url": "https://www.nature.com/articles/s41558-026-02821-8"
    }
  ]
}
```

The response above is shortened to one of the three articles, and `text`, `thumbnail` and `googleNewsUrl` are cut. The full response for this call was about 280 KB.

## Status Codes

| Code | Status | Description |
|---|---|---|
| 200 | OK | The request succeeded and the JSON result is in the response body. |
| 400 | Bad Request | The body is not valid JSON, both or neither of `searchQuery` and `topic` were sent, or a value is out of range. |
| 401 | Unauthorized | The `x-api-market-key` header is missing or the key is not valid. |
| 403 | Forbidden | Your key does not have an active subscription that includes this endpoint. |
| 404 | Not Found | The path does not match an endpoint of this API. Check the Base URL and path. |
| 429 | Too Many Requests | You went over the requests per second of your plan or used up its quota. |
| 500 | Internal Server Error | The request failed on the server. Retry once, then contact us if it continues. |
| 503 | Service Unavailable | The service is temporarily unavailable. Retry after a short wait. |

## Rate Limits

The Free plan allows 1 request per second, and Hobby, Pro, Business and Ultra allow 10. Each plan has a monthly quota. A crawl opens many pages, so one call can take 15 seconds or more. When you go over a limit, the API returns `429 Too Many Requests`. Throttle your requests, back off before retrying, and honour the `Retry-After` header when it is present.

## FAQ

*   **Why does an article say success true with statusCode 403?**
    The publisher blocked the page and returned a 403 page, which was extracted as text such as `403 Forbidden`. Check `statusCode` on each article and skip anything that is not 200.

*   **Can I send searchQuery and topic together?**
    No. Send exactly one of them, or the call is refused.

*   **How long does a crawl take?**
    It depends on `maxSites` and the number of editions. Our call with 3 sites took about 16 seconds. Set your client timeout to a few minutes for large crawls.

*   **Does daysBack work with topic?**
    No. `daysBack` only applies to `searchQuery`. A topic reads the current Google News topic feed.

*   **Why is the response so large?**
    It includes full article text and base64 thumbnails. Lower `maxSites` to keep it small.

## Contact Us

If you have any questions or need support, reach out through the chat embedded on this page or email us at hello@magicapi.com.


Search Google News by query or topic and get clean article text, titles, dates and thumbnails from publisher pages for AI and RAG pipelines.

AI News Content Crawler API: Google News Article Text

Free