Name: Morsel: RAG Chunking and Token Count API
Brand: hybrid-forge-lab
SKU: api-hybrid-forge-lab-morsel
Availability: InStock
Author: API Market

# Morsel RAG Chunking and Token Count API

## Overview

The Morsel API splits text and Markdown into chunks that never go over `max_tokens`, by tokens, sentences or Markdown headings, with overlap. Each chunk comes with its exact token count, its offsets in the original text and the headings it sits under.

It also counts tokens and truncates text to a token budget with the same key. It is built for anyone feeding documents to a model, and nothing is stored.

## Features

*   **Four endpoints**: chunking, token counting, truncation and a health check.
*   **Never over budget**: every chunk's own token count is at most `max_tokens`, checked by encoding the chunk on its own.
*   **Three strategies**: `tokens` for fixed windows, `sentences` for whole sentences packed up to the budget, and `headings` for one chunk per Markdown section.
*   **Heading paths**: each chunk knows where it lives, for example `["Handbook", "Setup", "First boot"]`. ATX and setext headings are understood, and anything inside code fences is ignored.
*   **Offsets you can trust**: `start` and `end` point into the original text, and `text == original[start:end]` for every chunk. Nothing is trimmed or rewritten.
*   **Real overlap**: with `tokens`, consecutive chunks share exactly `overlap` tokens of the document. With `sentences` and `headings`, the overlap is whole sentences.
*   **Exact counts**: `o200k_base` and `cl100k_base` are identical to tiktoken. Other models get a documented approximation (`chars4` or `chars3`), flagged `exact: false`.

## Use Cases

*   **RAG ingestion**: chunk docs, wikis and knowledge bases before embedding, with the section path stored as metadata.
*   **Prompt budgeting**: count the tokens of a prompt or retrieved context, and truncate it to fit the context window.
*   **Highlighting and citations**: offsets map every chunk back to the source, so an answer can highlight the exact passage it came from.
*   **One splitter for every stack**: the same chunking from Python, Node.js, Go, PHP or a no-code tool, without a tokenizer in every service.

## API Documentation

### Base URL

All requests go through the API.market proxy at this base URL:

```
https://prod.api.market/api/v1/hybrid-forge-lab/morsel
```

### Authentication

Every request needs your API.market key in the `x-api-market-key` header:

```
x-api-market-key: YOUR_API_KEY
```

Replace `YOUR_API_KEY` with your actual API key.

### Endpoints

#### 1. Health check

Returns `ok`, the service version and its uptime.

**Method:** `GET`

**Endpoint:** `/v1/health`

**Parameters:**

This endpoint takes no parameters.

**Example Request:**

```bash
curl -X GET 'https://prod.api.market/api/v1/hybrid-forge-lab/morsel/v1/health' \
  -H 'x-api-market-key: YOUR_API_KEY'
```

**Example Response (200 OK):**

```json
{
  "ok": true,
  "version": "0.2.1",
  "uptime_s": 361
}
```

#### 2. Split text or Markdown into chunks

Splits the text into chunks of at most `max_tokens` tokens, with offsets and heading paths. The example sends a short Markdown document with two headings and the `headings` strategy.

**Method:** `POST`

**Endpoint:** `/v1/chunk`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `text` | body | The text or Markdown to split. | string | Yes | - | 1,000,000 characters |
| `format` | body | `markdown` gives every chunk its `heading_path`. `text` is plain text, and `heading_path` is always `[]`. | string | No | `markdown` | - |
| `strategy` | body | `headings` (Markdown only): one chunk per section, large sections split by sentences. `tokens`: fixed windows; `sentences`: whole sentences packed up to `max_tokens`. | string | No | `headings` | - |
| `max_tokens` | body | Largest chunk, in tokens (minimum 1). | integer | No | 512 | 32,768 |
| `overlap` | body | Tokens repeated between consecutive chunks. Must be smaller than `max_tokens`. | integer | No | 0 | `max_tokens` - 1 |
| `tokenizer` | body | `o200k_base` or `cl100k_base` (exact), or `chars4` or `chars3` (approximations). | string | No | `o200k_base` | - |

**Example Request:**

```bash
curl -X POST 'https://prod.api.market/api/v1/hybrid-forge-lab/morsel/v1/chunk' \
  -H 'x-api-market-key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"text": "# Setup\n\nPlug the router into the modem. The light turns blue after 90 seconds.\n\n# Wi-Fi\n\nUse 5 GHz in the same room. Use 2.4 GHz farther away.\n", "format": "markdown", "strategy": "headings", "max_tokens": 64}'
```

**Example Response (200 OK):**

```json
{
  "chunks": [
    {
      "index": 0,
      "text": "# Setup\n\nPlug the router into the modem. The light turns blue after 90 seconds.\n\n",
      "tokens": 19,
      "heading_path": [
        "Setup"
      ],
      "start": 0,
      "end": 81
    },
    {
      "index": 1,
      "text": "# Wi-Fi\n\nUse 5 GHz in the same room. Use 2.4 GHz farther away.\n",
      "tokens": 22,
      "heading_path": [
        "Wi-Fi"
      ],
      "start": 81,
      "end": 144
    }
  ],
  "chunk_count": 2,
  "total_tokens": 41,
  "tokenizer": "o200k_base",
  "exact": true
}
```

#### 3. Count tokens

Counts the tokens of a text. With `o200k_base` and `cl100k_base` the result is identical to tiktoken.

**Method:** `POST`

**Endpoint:** `/v1/count`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `text` | body | The text to count. An empty string has 0 tokens. | string | Yes | - | 1,000,000 characters |
| `tokenizer` | body | `o200k_base`, `cl100k_base`, `chars4` or `chars3`. | string | No | `o200k_base` | - |

**Example Request:**

```bash
curl -X POST 'https://prod.api.market/api/v1/hybrid-forge-lab/morsel/v1/count' \
  -H 'x-api-market-key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"text": "Morsel counts tokens exactly like the model does.", "tokenizer": "o200k_base"}'
```

**Example Response (200 OK):**

```json
{
  "tokens": 11,
  "exact": true,
  "tokenizer": "o200k_base"
}
```

#### 4. Truncate text to a token budget

Cuts a text to at most `max_tokens` tokens, at a token boundary. Use it to fit retrieved context or a long prompt into a context window.

**Method:** `POST`

**Endpoint:** `/v1/truncate`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `text` | body | The text to cut. | string | Yes | - | 1,000,000 characters |
| `max_tokens` | body | Keep at most this many tokens (minimum 1). | integer | Yes | - | 32,768 |
| `tokenizer` | body | `o200k_base`, `cl100k_base`, `chars4` or `chars3`. | string | No | `o200k_base` | - |

**Example Request:**

```bash
curl -X POST 'https://prod.api.market/api/v1/hybrid-forge-lab/morsel/v1/truncate' \
  -H 'x-api-market-key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"text": "Morsel counts tokens exactly like the model does.", "max_tokens": 4, "tokenizer": "o200k_base"}'
```

**Example Response (200 OK):**

```json
{
  "text": "Morsel counts",
  "tokens": 4,
  "truncated": true,
  "exact": true,
  "tokenizer": "o200k_base"
}
```

## Status Codes

| Code | Status | Description |
|---|---|---|
| 200 | OK | The request succeeded. |
| 401 | Unauthorized | The `x-api-market-key` header is missing or the key is not valid. |
| 403 | Forbidden | Your key has no active subscription to this API, your plan quota is used up, or the endpoint is not part of your plan. |
| 404 | Not Found | The path is wrong. This API has `GET /v1/health`, `POST /v1/chunk`, `POST /v1/count` and `POST /v1/truncate`. |
| 413 | Payload Too Large | `text` is longer than 1,000,000 characters, or the request body is over 8 MB. |
| 415 | Unsupported Media Type | The body is not sent as JSON with `Content-Type: application/json`. |
| 422 | Unprocessable Entity | A field is invalid, `strategy=headings` was sent with `format=text`, or `overlap` is not below `max_tokens`. It is also returned for over 10,000 chunks or a request over the 20-second processing budget. |
| 429 | Too Many Requests | You sent more requests per second than your plan allows. Back off and try again. |

## Rate Limits

Each request counts as one call. Calls stop when the quota runs out until the next month.

*   **Free**: 15 requests, a 7-day free trial, once per customer, 2 requests per second.
*   **Pro**: 25,000 requests per month, 10 requests per second.
*   **Ultra**: 150,000 requests per month, 20 requests per second.
*   **Mega**: 1,000,000 requests per month, 50 requests per second.

Going over your plan's requests per second returns `429 Too Many Requests`. Back off when you get a 429 and retry after a short pause.

## FAQ

*   **Which tokenizer should I use?**

    `o200k_base` for GPT-4o, GPT-4.1 and the o-series, and `cl100k_base` for GPT-4, GPT-3.5 and text-embedding-3. For other models, `chars4` is a fair estimate for English prose and `chars3` a safer bound for code or other languages.

*   **What exactly do `start` and `end` mean?**

    They are offsets into the text you sent, counted in Unicode code points (Python string indices), with `end` exclusive. JavaScript strings count UTF-16 units, so for emoji index with `[...text]` or `Array.from(text)`.

*   **How does overlap work?**

    With `strategy` `tokens`, each chunk starts `overlap` tokens of the document before the previous one ends. With `sentences` and `headings`, Morsel repeats as many whole trailing sentences as fit in `overlap` tokens, so a chunk never starts mid-sentence.

*   **Why is a chunk's token count below `max_tokens`?**

    Chunks end at a boundary: a sentence, a section or a character. `tokens` is the count of the chunk's own text, encoded on its own, and it is never above `max_tokens`.

*   **How much text can I send?**

    Up to 1,000,000 characters per request, `max_tokens` from 1 to 32,768 and up to 10,000 chunks. A million characters takes well under a second.

*   **Do you store my documents?**

    No. Text is processed in memory and discarded, and there are no request logs.

## Contact Us

If you have any questions or need assistance, you can reach out to us through the embedded chat on the bottom right corner of the page or send an email to hello@magicapi.com.


Split text and Markdown into chunks under a token budget, with overlap, offsets and heading paths. Count and truncate tokens with o200k_base or cl100k_base.

Morsel: RAG Chunking and Token Count API

Morsel RAG Chunking and Token Count