Name: Whisper Diarization API: Transcripts with Speaker Labels
Brand: magicapi
SKU: api-magicapi-speech-to-text-diarization-api-rafaelgalle-whisper-diarization-advanced
Availability: InStock
Author: API Market

Shashank Agarwal

# 🎙️ Speech-to-Text with Speaker Diarization API (rafaelgalle Whisper Diarization Advanced)

## Overview
Whisper Diarization Advanced transcribes speech with Whisper and labels each part of the transcript with the speaker who said it. Its developer describes it as fast, customizable speech-to-text and speaker diarization for noisy, multi-speaker audio.

Send audio as a URL (`file_path` or `file_url`) or as Base64 (`file_string`). Optionally set `num_speakers`, `language`, a `prompt` with names and acronyms, and `translate` to get English. For noisy recordings, `preprocess` adds sanitizing, filtering, noise reduction and loudness normalization in steps. The output lists the detected language, the number of speakers and segments with speaker, start and end times, text and per-word timestamps.

## Features
- **Speaker diarization**: each segment and word is labelled with a speaker such as `SPEAKER_00`.
- **Word timestamps**: every word has `start`, `end` and `probability`.
- **Three ways to send audio**: `file_path` (URL), `file_url` (direct URL) or `file_string` (Base64).
- **Speaker count**: set `num_speakers` (1 to 50) or let the model detect it.
- **Translation**: `translate` returns the transcript in English.
- **Vocabulary hints**: list names and acronyms that occur in the audio in `prompt`.
- **Audio preprocessing**: `preprocess` levels 0 to 4 with highpass and lowpass filters, noise reduction (`prop_decrease`, `stationary`) and loudness normalization (`target_dBFS`).

## Use Cases
1. **Meeting notes**: transcribe calls and meetings with each speaker's lines separated.
2. **Podcasts and interviews**: produce speaker-labelled transcripts for show notes and captions.
3. **Call center audio**: transcribe noisy two-party calls with noise reduction turned on.
4. **Subtitles**: use word timestamps to time captions.

## Visual Examples
### Example Input
A 90-second English interview clip, with a prompt describing the topic and no preprocessing (`preprocess` `0`).

```json
{
  "prompt": "A conversation about open-source AI.",
  "language": "en",
  "file_path": "https://blog.api.market/wp-content/uploads/2026/10/whisper-diarization-input-api-market.mp3",
  "translate": false,
  "preprocess": 0,
  "stationary": true,
  "target_dBFS": -18,
  "lowpass_freq": 8000,
  "highpass_freq": 45,
  "prop_decrease": 0.3
}
```

### Example Output
The model detected 2 speakers and returned 7 segments:

| Speaker | Start (s) | End (s) | Text |
| --- | --- | --- | --- |
| SPEAKER_01 | 2.74 | 14.24 | Let me ask you about AI. It seems like this year, for the entirety of the human civilization, is an interesting year for the development of artificial intelligence. A lot of interesting stuff is happening. |
| SPEAKER_01 | 15.56 | 21.94 | So meta is a big part of that. Meta has developed LLAMA, which is a 65 billion parameter model. |
| SPEAKER_01 | 24.06 | 38.94 | There's a lot of interesting questions I can ask here, one of which has to do with open-source. But first, can you tell the story of developing of this model and making the complicated decision of how to release it? |
| SPEAKER_00 | 38.94 | 52.38 | Yeah, sure. I think you're right, first of all, that in the last year, there have been a bunch of advances on scaling up these large transformer models. So there's the language equivalent of it with large language models. |
| SPEAKER_00 | 52.38 | 61.4 | There's sort of the image generation equivalent with these large diffusion models. There's a lot of fundamental research that's gone into this. |
| SPEAKER_00 | 62.7 | 82.82 | Meta has taken the approach of being quite open and academic in our development of AI. Part of this is we want to have the best people in the world researching this, and a lot of the best people want to know that they're going to be able to share their work. |
| SPEAKER_00 | 82.82 | 89.98 | So that's part of the deal that we have, is that if you're one of the top AI researchers in the world, you're going to be able to share your work. |

Each segment also includes `duration`, `avg_logprob` and a `words` list with per-word `start`, `end`, `probability` and `speaker`.

## Input Parameters
| Field | Type | Required | Default | Allowed values | Description |
| --- | --- | --- | --- | --- | --- |
| `file_path` | string (URL) | No | none |  | Audio file URL |
| `file_url` | string | No | none |  | Direct URL to the audio |
| `file_string` | string | No | none |  | Base64-encoded audio |
| `num_speakers` | integer | No | none | `1` to `50` | Number of speakers. Leave empty to detect automatically |
| `translate` | boolean | No | `false` |  | Translate the transcript to English |
| `language` | string | No | none |  | Language code, for example 'en' or 'pt'. Leave empty to detect automatically |
| `prompt` | string | No | none |  | Custom prompt with names and acronyms, separated by punctuation |
| `preprocess` | integer | No | `0` | `0` to `4` | Preprocessing level: 0 = none, 1 = sanitize, 2 = + filter, 3 = + reduce noise, 4 = + normalize |
| `highpass_freq` | integer | No | `45` |  | Highpass filter frequency in Hz |
| `lowpass_freq` | integer | No | `8000` |  | Lowpass filter frequency in Hz |
| `prop_decrease` | number | No | `0.3` | `0` to `1` | Noise reduction proportion |
| `stationary` | boolean | No | `true` |  | Use stationary noise reduction |
| `target_dBFS` | number | No | `-18` |  | Target loudness in dBFS |

## Pricing
- **Pay As You Go (PAYG) pricing**: you only pay for what you use.
- **Price**: $0.078 per API unit.
- **1 API unit = 1 request with up to 72 seconds of processing time.** A request that takes longer is billed one unit per started 72-second block (for example, 100 seconds of processing is 2 units, $0.156).
- Longer recordings and heavier preprocessing take longer, so very long files can use more than one unit.
- **Example cost**: the example's 90 seconds of audio took 5.04 seconds of processing: 1 API unit = **$0.078**.
- **Failed or cancelled predictions are not billed.**
- **GET requests to check prediction status** are free and don't count toward API unit usage.
- **No monthly subscription fee.**

## API Documentation

### Base URL
All API requests go to: `https://prod.api.market/api/v1/magicapi/speech-to-text-diarization-api-rafaelgalle-whisper-diarization-advanced`

### Authentication
Send your api.market API key in the `x-api-market-key` header:
```
x-api-market-key: YOUR_API_MARKET_KEY
```

### Asynchronous Processing
**IMPORTANT**: This API is asynchronous. The flow is:
1. Send a POST request to `/predictions`. A successful request returns **201 Created** with a prediction `id` and a `status` of `starting` or `processing`.
2. Poll GET `/predictions/{id}` with that ID.
3. Keep polling while `status` is `starting` or `processing`.
4. Stop when `status` is `succeeded` (read `output`), `failed` or `canceled` (read `error`).

A polling interval of 1 second and an overall timeout of 600 seconds (10 minutes) work well.

### Endpoints

#### Create a Prediction
```bash
curl -X POST "https://prod.api.market/api/v1/magicapi/speech-to-text-diarization-api-rafaelgalle-whisper-diarization-advanced/predictions" \
  -H "Content-Type: application/json" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY" \
  -d '{
    "version": "68dd22041e73a120096d341482a691626f48cb1cd19b48230b8ee420db487462",
    "input": {
      "prompt": "A conversation about open-source AI.",
      "language": "en",
      "file_path": "https://blog.api.market/wp-content/uploads/2026/10/whisper-diarization-input-api-market.mp3",
      "translate": false,
      "preprocess": 0,
      "stationary": true,
      "target_dBFS": -18,
      "lowpass_freq": 8000,
      "highpass_freq": 45,
      "prop_decrease": 0.3
    }
  }'
```

Response (201 Created):
```json
{
  "id": "cn85mryx8drmy0d1cd68g4kwn8",
  "version": "68dd22041e73a120096d341482a691626f48cb1cd19b48230b8ee420db487462",
  "status": "starting",
  "created_at": "2026-10-22T11:09:04.356Z"
}
```

#### Check Prediction Status
```bash
curl -X GET "https://prod.api.market/api/v1/magicapi/speech-to-text-diarization-api-rafaelgalle-whisper-diarization-advanced/predictions/{id}" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY"
```

Response when the prediction has finished (200 OK):
```json
{
  "id": "cn85mryx8drmy0d1cd68g4kwn8",
  "status": "succeeded",
  "output": {
    "language": "en",
    "num_speakers": 2,
    "segments": [
      {
        "avg_logprob": -0.2942341608060917,
        "duration": 11.5,
        "end": 14.24,
        "speaker": "SPEAKER_01",
        "start": 2.74,
        "text": "Let me ask you about AI. It seems like this year, for the entirety of the human civilization, is an interesting year for the development of artificial intelligence. A lot of interesting stuff is happening.",
        "words": [
          {
            "end": 3.16,
            "probability": 0.50244140625,
            "speaker": "SPEAKER_01",
            "start": 2.74,
            "word": " Let"
          },
          {
            "end": 3.3,
            "probability": 0.9033203125,
            "speaker": "SPEAKER_01",
            "start": 3.16,
            "word": " me"
          }
        ]
      },
      {
        "avg_logprob": -0.2288161072020347,
        "duration": 13.440000000000005,
        "end": 52.38,
        "speaker": "SPEAKER_00",
        "start": 38.94,
        "text": "Yeah, sure. I think you're right, first of all, that in the last year, there have been a bunch of advances on scaling up these large transformer models. So there's the language equivalent of it with large language models.",
        "words": [
          {
            "end": 39.78,
            "probability": 0.67578125,
            "speaker": "SPEAKER_00",
            "start": 38.94,
            "word": " Yeah,"
          },
          {
            "end": 40.22,
            "probability": 0.80810546875,
            "speaker": "SPEAKER_00",
            "start": 39.84,
            "word": " sure."
          }
        ]
      }
    ]
  },
  "error": null,
  "created_at": "2026-10-22T11:09:04.356Z",
  "started_at": "2026-10-22T11:10:30.041Z",
  "completed_at": "2026-10-22T11:10:35.080Z"
}
```

`output` is an object with `language`, `num_speakers` and `segments` (shortened above to two segments with two words each). Each segment has `speaker`, `start`, `end`, `duration`, `text`, `avg_logprob` and `words`.

### Status Codes
- **201 Created**: prediction created (POST `/predictions`). This is the normal success response.
- **200 OK**: prediction details returned (GET `/predictions/{id}`).
- **400 Bad Request**: invalid input, for example a `num_speakers` above 50 or a `preprocess` level above 4.
- **401 Unauthorized**: missing or invalid API key.
- **404 Not Found**: no prediction with that ID.
- **429 Too Many Requests**: rate limit exceeded.
- **500 Internal Server Error** or **503 Service Unavailable**: temporary server problem; retry later.

## Rate Limits
All plans allow 10 requests per second. Requests above that return `429 Too Many Requests`; back off before retrying and honour any `Retry-After` header. Polling a prediction with GET is free but still counts toward the per-second limit, so poll every 1 to 5 seconds rather than in a tight loop.

## Error Handling
- Check the HTTP status code of every request and read the response body for the error message.
- A prediction can also end with `status` set to `failed`; the `error` field then explains why. Failed predictions are not billed.
- Retry 429, 500 and 503 responses with exponential backoff. Fix the input before retrying a 400.

## Implementation Guide
### Example Code

#### JavaScript
```javascript
const apiKey = "YOUR_API_MARKET_KEY";
const baseUrl = "https://prod.api.market/api/v1/magicapi/speech-to-text-diarization-api-rafaelgalle-whisper-diarization-advanced";

async function createAndWaitForPrediction(input) {
  const response = await fetch(`${baseUrl}/predictions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-api-market-key": apiKey
    },
    body: JSON.stringify({ version: "68dd22041e73a120096d341482a691626f48cb1cd19b48230b8ee420db487462", input })
  });

  if (response.status !== 201) {
    throw new Error(`API error: ${response.status} ${await response.text()}`);
  }

  const prediction = await response.json();
  return pollForResults(prediction.id);
}

const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function pollForResults(predictionId) {
  const maxTimeout = 600000;
  const pollInterval = 1000;
  const startTime = Date.now();

  while (Date.now() - startTime < maxTimeout) {
    const response = await fetch(`${baseUrl}/predictions/${predictionId}`, {
      headers: { "x-api-market-key": apiKey }
    });

    if (!response.ok) {
      throw new Error(`API error: ${response.status} ${await response.text()}`);
    }

    const prediction = await response.json();

    if (prediction.status === "succeeded") {
      return prediction.output;
    }

    if (prediction.status === "failed" || prediction.status === "canceled") {
      throw new Error(`Prediction ${prediction.status}: ${prediction.error || "Unknown error"}`);
    }

    await delay(pollInterval);
  }

  throw new Error("Prediction timed out after 10 minutes");
}

createAndWaitForPrediction({
    "prompt": "A conversation about open-source AI.",
    "language": "en",
    "file_path": "https://blog.api.market/wp-content/uploads/2026/10/whisper-diarization-input-api-market.mp3",
    "translate": false,
    "preprocess": 0,
    "stationary": true,
    "target_dBFS": -18,
    "lowpass_freq": 8000,
    "highpass_freq": 45,
    "prop_decrease": 0.3
  }).then((output) => console.log(output));
```

#### Python
```python
import time
import requests

API_KEY = "YOUR_API_MARKET_KEY"
BASE_URL = "https://prod.api.market/api/v1/magicapi/speech-to-text-diarization-api-rafaelgalle-whisper-diarization-advanced"


def create_and_wait_for_prediction(input_data):
    headers = {"Content-Type": "application/json", "x-api-market-key": API_KEY}
    response = requests.post(
        f"{BASE_URL}/predictions",
        headers=headers,
        json={"version": "68dd22041e73a120096d341482a691626f48cb1cd19b48230b8ee420db487462", "input": input_data},
    )

    if response.status_code != 201:
        raise Exception(f"API error: {response.status_code} {response.text}")

    return poll_for_results(response.json()["id"])


def poll_for_results(prediction_id):
    max_timeout = 600
    poll_interval = 1
    headers = {"x-api-market-key": API_KEY}
    start_time = time.time()

    while time.time() - start_time < max_timeout:
        response = requests.get(f"{BASE_URL}/predictions/{prediction_id}", headers=headers)

        if response.status_code != 200:
            raise Exception(f"API error: {response.status_code} {response.text}")

        prediction = response.json()

        if prediction["status"] == "succeeded":
            return prediction["output"]

        if prediction["status"] in ("failed", "canceled"):
            raise Exception(f"Prediction {prediction['status']}: {prediction.get('error') or 'Unknown error'}")

        time.sleep(poll_interval)

    raise Exception("Prediction timed out after 10 minutes")


print(create_and_wait_for_prediction({
    "prompt": "A conversation about open-source AI.",
    "language": "en",
    "file_path": "https://blog.api.market/wp-content/uploads/2026/10/whisper-diarization-input-api-market.mp3",
    "translate": False,
    "preprocess": 0,
    "stationary": True,
    "target_dBFS": -18,
    "lowpass_freq": 8000,
    "highpass_freq": 45,
    "prop_decrease": 0.3
}))
```

## FAQ
- **How do I send the audio?**
  - Pass a URL in `file_path` (as in the example) or `file_url`, or send the audio as Base64 in `file_string`.
- **Do I need to set the number of speakers?**
  - No. Leave `num_speakers` empty to detect it automatically, or set 1 to 50 if you know it.
- **Can I get an English transcript of non-English audio?**
  - Yes. Set `translate` to `true`.
- **When should I use `preprocess`?**
  - For noisy recordings. Level 1 sanitizes the audio, 2 adds filtering, 3 adds noise reduction and 4 adds loudness normalization. The default is 0 (none).
- **How is a request priced?**
  - $0.078 per started 72 seconds of processing. The 90-second example took 5.04 seconds, so it cost 1 unit.
- **What if my prediction fails?**
  - Failed predictions are not billed. Read the `error` field, adjust the input and try again.

## Contact Us
If you have any questions, reach out to us through the chat on the bottom right of the page. You can also email our support team at hello@magicapi.com.


Transcribe audio with Whisper and speaker diarization: speaker-labelled segments with word timestamps, optional translation to English and noise reduction.