Name: Token Counter API for OpenAI o200k and cl100k
Brand: leadharbor
SKU: api-leadharbor-token-counter
Availability: InStock
Author: API Market

Mohammed Samaan

# Token Counter API

## Overview

The Token Counter API counts the tokens in a text before you send it to a language model. It uses OpenAI's published BPE vocabularies: `o200k_base`, used by the GPT-4o family and newer models, and `cl100k_base`, used by GPT-4 and GPT-3.5. It has 2 endpoints on the same path. `GET /v1/market/tokens` takes the text in the query string and suits short prompts. `POST /v1/market/tokens` takes the same fields in a JSON body and accepts long documents of up to 400,000 characters. Both return the token count and character count, and if you pass a `limit`, whether the text fits within it.

It is built for developers of LLM applications who need to check prompt length, split documents into chunks, estimate cost or enforce a context-window budget before they call a model. The count is exact for OpenAI models that use the chosen encoding and only an estimate for other model families, such as Claude, Gemini and Llama, which tokenize differently; every response says so in `note`. Each response is a flat JSON object with `tokens`, `characters`, `encoding`, `limit` and `within_limit` when a limit was sent, `note`, and a `meta` object that you can ignore.

## Features

*   **2 endpoints**: `GET` for short text in the query string and `POST` with a JSON body for long text.
*   **Two OpenAI encodings**: `o200k_base` (default, GPT-4o family and newer) and `cl100k_base` (GPT-4 and GPT-3.5).
*   **Limit check in the same call**: Pass `limit` and the response adds `within_limit`, so you can reject or trim a prompt without doing the comparison yourself.
*   **Long documents**: The `POST` endpoint accepts text up to 400,000 characters.
*   **Safe with special tokens**: Text that looks like a special token, such as `<|endoftext|>`, is counted as ordinary text instead of failing.
*   **Clear accuracy note**: Counts are exact for OpenAI models that use the chosen encoding and approximate for other model families, and each response states this.
*   **Rate limit**: 10 requests per second on every plan.

## Use Cases

*   **Prompt length checks**: Confirm a prompt fits a model's context window before you send it, and trim it if `within_limit` is `false`.
*   **Cost estimates**: Count input tokens to estimate the cost of a request to an OpenAI model.
*   **Document chunking**: Measure documents and sections to split them into chunks of a target token size for RAG pipelines.
*   **User quotas**: Enforce per-user token budgets in a chat or writing app.
*   **Model comparison**: Compare the token count of the same text under `o200k_base` and `cl100k_base`.

## API Documentation

### Base URL

All requests go through the API.market gateway at this base URL:

```
https://prod.api.market/api/v1/leadharbor/token-counter
```

### Authentication

Every request needs your API.market key in the `x-api-market-key` header:

```
x-api-market-key: YOUR_API_KEY
```

Replace `YOUR_API_KEY` with your actual API key.

### Endpoints

#### 1. Count the tokens in a text

Counts the tokens in a short text passed in the query string and returns `tokens`, `characters`, `encoding` and `note`. When you pass `limit`, the response also carries `limit` and `within_limit`. Use the `POST` endpoint for long text.

**Method:** `GET`

**Endpoint:** `/v1/market/tokens`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `text` | query | The text to count. URL-encode it. | string | Yes | - | - |
| `encoding` | query | Vocabulary to count with: `o200k_base` or `cl100k_base`. | string | No | `o200k_base` | - |
| `limit` | query | Token limit to check the text against. When set, the response adds `within_limit`. | integer | No | - | - |

**Example Request (with a limit):**

```bash
curl -X GET "https://prod.api.market/api/v1/leadharbor/token-counter/v1/market/tokens?text=Hello%2C%20world!&encoding=o200k_base&limit=8000" \
  -H "x-api-market-key: YOUR_API_KEY"
```

**Example Response (200 OK):**

```json
{
  "tokens": 4,
  "characters": 13,
  "encoding": "o200k_base",
  "limit": 8000,
  "within_limit": true,
  "note": "Counted with OpenAI BPE vocabularies. Exact for OpenAI models that use this encoding; other model families (Claude, Gemini, Llama) tokenize differently, so treat the number as an estimate for them.",
  "meta": {
    "user": null
  }
}
```

**Example Request (cl100k_base, no limit):**

```bash
curl -X GET "https://prod.api.market/api/v1/leadharbor/token-counter/v1/market/tokens?text=Hello%2C%20world!&encoding=cl100k_base" \
  -H "x-api-market-key: YOUR_API_KEY"
```

**Example Response (200 OK):**

```json
{
  "tokens": 4,
  "characters": 13,
  "encoding": "cl100k_base",
  "note": "Counted with OpenAI BPE vocabularies. Exact for OpenAI models that use this encoding; other model families (Claude, Gemini, Llama) tokenize differently, so treat the number as an estimate for them.",
  "meta": {
    "user": null
  }
}
```

**Response fields:**

| Field | Type | Description |
|---|---|---|
| `tokens` | integer | Number of tokens in the text under the chosen encoding. |
| `characters` | integer | Number of characters in the text. |
| `encoding` | string | The encoding used. |
| `limit` | integer | The limit you sent. Present only when `limit` was set. |
| `within_limit` | boolean | `true` when `tokens` is within `limit`. Present only when `limit` was set. |
| `note` | string | States that the count is exact for OpenAI models using this encoding and an estimate for other model families. |
| `meta` | object | Metadata that you can ignore. |

#### 2. Count tokens in a longer text

Counts the tokens in a text sent in a JSON body, for prompts and documents too long for a query string (up to 400,000 characters). It takes the same fields as the `GET` endpoint and returns the same response.

**Method:** `POST`

**Endpoint:** `/v1/market/tokens`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `text` | body | The text to count. | string | Yes | - | 400,000 characters |
| `encoding` | body | Vocabulary to count with: `o200k_base` or `cl100k_base`. | string | No | `o200k_base` | - |
| `limit` | body | Token limit to check the text against. When set, the response adds `within_limit`. | integer | No | - | - |

**Example Request:**

```bash
curl -X POST "https://prod.api.market/api/v1/leadharbor/token-counter/v1/market/tokens" \
  -H "x-api-market-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text": "Hello, world! This is a test.", "encoding": "o200k_base", "limit": 8000}'
```

**Example Response (200 OK):**

```json
{
  "tokens": 9,
  "characters": 29,
  "encoding": "o200k_base",
  "limit": 8000,
  "within_limit": true,
  "note": "Counted with OpenAI BPE vocabularies. Exact for OpenAI models that use this encoding; other model families (Claude, Gemini, Llama) tokenize differently, so treat the number as an estimate for them.",
  "meta": {
    "user": null
  }
}
```

## Status Codes

| Code | Status | Description |
|---|---|---|
| 200 | OK | The text was counted. When `limit` was sent, read `within_limit` for the result. |
| 400 | Bad Request | `text` is missing, `encoding` is not `o200k_base` or `cl100k_base`, or `limit` is not a valid integer. |
| 401 | Unauthorized | The `x-api-market-key` value is invalid or expired. |
| 403 | Forbidden | Your subscription does not include this endpoint. |
| 404 | Not Found | The path is wrong. Check that you are calling `/v1/market/tokens` under the base URL. |
| 429 | Too Many Requests | You went over 10 requests per second, or you have used all the calls in your plan. |
| 500 | Internal Server Error | Something failed while processing the request. Retry after a short delay. |

## Rate Limits

The Free trial, Starter and PAYG plans each allow 10 requests per second. The Free trial is a 7-day trial with 500 calls and a hard stop, Starter includes 10,000 calls with extra calls billed at $0.001 each, and PAYG bills $0.001 per call with no subscription. Each request counts as one call whether you use `GET` or `POST`. Requests above the per-second limit, or after the Free trial's calls are used up, return `429 Too Many Requests`. Back off before retrying and honour the `Retry-After` header, which says how many seconds to wait.

## FAQ

**Is the count exact for every model?**

It is exact for OpenAI models that use the chosen encoding. Other model families, such as Claude, Gemini and Llama, use their own tokenizers, so for them treat the number as an estimate. Every response repeats this in `note`.

**Which encoding should I use?**

Use `o200k_base`, the default, for the GPT-4o family and newer OpenAI models. Use `cl100k_base` for GPT-4 and GPT-3.5.

**When should I use POST instead of GET?**

Use `GET` for short prompts. Use `POST` with a JSON body for long text, since query strings have practical length limits. The `POST` endpoint accepts up to 400,000 characters.

**What does `within_limit` tell me?**

When you send `limit`, the response includes `within_limit`, which is `true` when the token count is within that limit. If you do not send `limit`, neither `limit` nor `within_limit` appears in the response.

**What happens with text that contains special tokens like `<|endoftext|>`?**

It is counted as ordinary text, so the request does not fail.

**Does the count include tokens added by a chat format?**

The API counts the text you send. Chat formats can add a few tokens per message for roles and separators, so leave some headroom when you set `limit` for a chat request.

## Contact Us

If you have any questions or need assistance, you can reach out to us through the embedded chat on the bottom right corner of the page or send an email to hello@magicapi.com.


Token counter API that returns token and character counts for OpenAI o200k_base and cl100k_base, with an optional limit check. GET or POST up to 400k chars.

Token Counter API for OpenAI o200k and cl100k

Free trial