Name: Meta SAM 2 Video API: Video Object Segmentation
Brand: magicapi
SKU: api-magicapi-video-object-segmentation-meta-sam-2-video
Availability: InStock
Author: API Market

Shashank Agarwal

# 🎯 Video Object Segmentation API (Meta SAM 2 Video)

## Overview
SAM 2 (Segment Anything v2) is Meta's segmentation model for images and videos. This version segments and tracks objects through a video.

Send an `input_video` and one or more click points in `click_coordinates`. Each click can be marked as foreground or background (`click_labels`), placed on a specific frame (`click_frames`) and given an object label (`click_object_ids`). The model follows the selected objects through the video and returns them as `binary`, `highlighted` or `greenscreen` masks, with optional bounding boxes. Set `output_video` to `true` for one video, or leave it `false` for an image sequence.

## Features
- **Click prompts**: select objects with `[x,y]` points; label each as foreground (`1`) or background (`0`).
- **Several objects**: name the objects you click with `click_object_ids`, for example `person,dog`.
- **Clicks on any frame**: `click_frames` sets the frame each click belongs to.
- **Mask styles**: `binary` (default, black and white), `highlighted` (colored overlay) or `greenscreen`.
- **Boxes**: `annotation_type` draws a `mask`, a `box` or `both` (for highlighted masks).
- **Video or frames**: `output_video` returns one video at `video_fps` (1 to 60); otherwise an image sequence in `webp`, `jpg` or `png`, optionally every Nth frame.

## Use Cases
1. **Background replacement**: produce a `greenscreen` or `binary` matte of a person for compositing.
2. **Highlighting subjects**: overlay a colored mask on a player, product or vehicle in a clip.
3. **Object tracking data**: get per-frame masks or boxes of selected objects for analysis.
4. **Video editing masks**: create masks to blur, recolor or cut out an object in an editor.

## Visual Examples
### Example Input
The example video, with one foreground click at `[320,240]` on frame 1 labelled `person`. The request asks for a `highlighted` mask as a 25 fps video.

```json
{
  "mask_type": "highlighted",
  "video_fps": 25,
  "input_video": "https://blog.api.market/wp-content/uploads/2026/10/robust_video_matting-input-api-market.mp4",
  "click_frames": "1",
  "click_labels": "1",
  "output_video": true,
  "output_format": "webp",
  "output_quality": 80,
  "annotation_type": "mask",
  "click_object_ids": "person",
  "click_coordinates": "[320,240]",
  "output_frame_interval": 1
}
```

### Example Output
[Watch the example output](https://blog.api.market/wp-content/uploads/2026/10/sam-2-video-output-x69ehax6-api-market.mp4)

## Input Parameters
| Field | Type | Required | Default | Allowed values | Description |
| --- | --- | --- | --- | --- | --- |
| `input_video` | string (URL) | Yes |  |  | URL of the video to segment |
| `click_coordinates` | string | Yes |  |  | Click points as '[x,y],[x,y],...'. The number of points sets the number of clicks |
| `click_labels` | string | No | `1` |  | Click types as '1,1,0,1' (1 = foreground, 0 = background). Extended automatically if shorter than the list of coordinates |
| `click_frames` | string | No | `0` |  | Frame index for each click as '0,0,150,0'. Extended automatically if shorter than the list of coordinates |
| `click_object_ids` | string | No | `""` |  | Object label for each click as 'person,dog,cat'. Generated automatically if missing or incomplete |
| `mask_type` | string | No | `binary` | `binary`, `highlighted`, `greenscreen` | Mask style: binary (black and white), highlighted (colored overlay) or greenscreen |
| `annotation_type` | string | No | `mask` | `mask`, `box`, `both` | Annotation: mask only, bounding box only, or both. Ignored for binary and greenscreen masks |
| `output_video` | boolean | No | `false` |  | true returns a video, false returns an image sequence |
| `video_fps` | integer | No | `30` | `1` to `60` | Frame rate of the output video. Ignored for image sequences |
| `output_format` | string | No | `webp` | `webp`, `jpg`, `png` | Image format of the image sequence. Ignored for video output |
| `output_quality` | integer | No | `80` | `0` to `100` | JPG and WebP compression quality (0 to 100). Ignored for PNG and video output |
| `output_frame_interval` | integer | No | `1` |  | Output every Nth frame: 1 = all frames, 2 = every other frame, and so on |

## Pricing
- **Pay As You Go (PAYG) pricing**: you only pay for what you use.
- **Price**: $0.02 per API unit.
- **1 API unit = 1 request with up to 18 seconds of processing time.** A request that takes longer is billed one unit per started 18-second block (for example, 30 seconds of processing is 2 units, $0.04).
- Longer and higher-resolution videos take longer to process, so they can use several units.
- **Example cost**: the example took 45.2 seconds of processing, 3 started 18-second blocks: 3 API units = **$0.06**.
- **Failed or cancelled predictions are not billed.**
- **GET requests to check prediction status** are free and don't count toward API unit usage.
- **No monthly subscription fee.**

## API Documentation

### Base URL
All API requests go to: `https://prod.api.market/api/v1/magicapi/video-object-segmentation-meta-sam-2-video`

### Authentication
Send your api.market API key in the `x-api-market-key` header:
```
x-api-market-key: YOUR_API_MARKET_KEY
```

### Asynchronous Processing
**IMPORTANT**: This API is asynchronous. The flow is:
1. Send a POST request to `/predictions`. A successful request returns **201 Created** with a prediction `id` and a `status` of `starting` or `processing`.
2. Poll GET `/predictions/{id}` with that ID.
3. Keep polling while `status` is `starting` or `processing`.
4. Stop when `status` is `succeeded` (read `output`), `failed` or `canceled` (read `error`).

A polling interval of 2 seconds and an overall timeout of 600 seconds (10 minutes) work well.

### Endpoints

#### Create a Prediction
```bash
curl -X POST "https://prod.api.market/api/v1/magicapi/video-object-segmentation-meta-sam-2-video/predictions" \
  -H "Content-Type: application/json" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY" \
  -d '{
    "version": "33432afdfc06a10da6b4018932893d39b0159f838b6d11dd1236dff85cc5ec1d",
    "input": {
      "mask_type": "highlighted",
      "video_fps": 25,
      "input_video": "https://blog.api.market/wp-content/uploads/2026/10/robust_video_matting-input-api-market.mp4",
      "click_frames": "1",
      "click_labels": "1",
      "output_video": true,
      "output_format": "webp",
      "output_quality": 80,
      "annotation_type": "mask",
      "click_object_ids": "person",
      "click_coordinates": "[320,240]",
      "output_frame_interval": 1
    }
  }'
```

Response (201 Created):
```json
{
  "id": "x69ehax669rmw0d1cd5tk5nk34",
  "version": "33432afdfc06a10da6b4018932893d39b0159f838b6d11dd1236dff85cc5ec1d",
  "status": "starting",
  "created_at": "2026-10-22T11:07:44.820Z"
}
```

#### Check Prediction Status
```bash
curl -X GET "https://prod.api.market/api/v1/magicapi/video-object-segmentation-meta-sam-2-video/predictions/{id}" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY"
```

Response when the prediction has finished (200 OK):
```json
{
  "id": "x69ehax669rmw0d1cd5tk5nk34",
  "status": "succeeded",
  "output": [
    "https://blog.api.market/wp-content/uploads/2026/10/sam-2-video-output-x69ehax6-api-market.mp4"
  ],
  "error": null,
  "created_at": "2026-10-22T11:07:44.820Z",
  "started_at": "2026-10-22T11:08:04.432Z",
  "completed_at": "2026-10-22T11:08:49.680Z"
}
```

`output` is a list of URLs. With `output_video` set to `true` it holds one video; otherwise it holds one image per output frame.

### Status Codes
- **201 Created**: prediction created (POST `/predictions`). This is the normal success response.
- **200 OK**: prediction details returned (GET `/predictions/{id}`).
- **400 Bad Request**: invalid input, for example a missing `input_video` or `click_coordinates`, or a `mask_type` that is not in the list.
- **401 Unauthorized**: missing or invalid API key.
- **404 Not Found**: no prediction with that ID.
- **429 Too Many Requests**: rate limit exceeded.
- **500 Internal Server Error** or **503 Service Unavailable**: temporary server problem; retry later.

## Rate Limits
All plans allow 10 requests per second. Requests above that return `429 Too Many Requests`; back off before retrying and honour any `Retry-After` header. Polling a prediction with GET is free but still counts toward the per-second limit, so poll every 1 to 5 seconds rather than in a tight loop.

## Error Handling
- Check the HTTP status code of every request and read the response body for the error message.
- A prediction can also end with `status` set to `failed`; the `error` field then explains why. Failed predictions are not billed.
- Retry 429, 500 and 503 responses with exponential backoff. Fix the input before retrying a 400.

## Implementation Guide
### Example Code

#### JavaScript
```javascript
const apiKey = "YOUR_API_MARKET_KEY";
const baseUrl = "https://prod.api.market/api/v1/magicapi/video-object-segmentation-meta-sam-2-video";

async function createAndWaitForPrediction(input) {
  const response = await fetch(`${baseUrl}/predictions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-api-market-key": apiKey
    },
    body: JSON.stringify({ version: "33432afdfc06a10da6b4018932893d39b0159f838b6d11dd1236dff85cc5ec1d", input })
  });

  if (response.status !== 201) {
    throw new Error(`API error: ${response.status} ${await response.text()}`);
  }

  const prediction = await response.json();
  return pollForResults(prediction.id);
}

const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function pollForResults(predictionId) {
  const maxTimeout = 600000;
  const pollInterval = 2000;
  const startTime = Date.now();

  while (Date.now() - startTime < maxTimeout) {
    const response = await fetch(`${baseUrl}/predictions/${predictionId}`, {
      headers: { "x-api-market-key": apiKey }
    });

    if (!response.ok) {
      throw new Error(`API error: ${response.status} ${await response.text()}`);
    }

    const prediction = await response.json();

    if (prediction.status === "succeeded") {
      return prediction.output;
    }

    if (prediction.status === "failed" || prediction.status === "canceled") {
      throw new Error(`Prediction ${prediction.status}: ${prediction.error || "Unknown error"}`);
    }

    await delay(pollInterval);
  }

  throw new Error("Prediction timed out after 10 minutes");
}

createAndWaitForPrediction({
    "mask_type": "highlighted",
    "video_fps": 25,
    "input_video": "https://blog.api.market/wp-content/uploads/2026/10/robust_video_matting-input-api-market.mp4",
    "click_frames": "1",
    "click_labels": "1",
    "output_video": true,
    "output_format": "webp",
    "output_quality": 80,
    "annotation_type": "mask",
    "click_object_ids": "person",
    "click_coordinates": "[320,240]",
    "output_frame_interval": 1
  }).then((output) => console.log(output));
```

#### Python
```python
import time
import requests

API_KEY = "YOUR_API_MARKET_KEY"
BASE_URL = "https://prod.api.market/api/v1/magicapi/video-object-segmentation-meta-sam-2-video"


def create_and_wait_for_prediction(input_data):
    headers = {"Content-Type": "application/json", "x-api-market-key": API_KEY}
    response = requests.post(
        f"{BASE_URL}/predictions",
        headers=headers,
        json={"version": "33432afdfc06a10da6b4018932893d39b0159f838b6d11dd1236dff85cc5ec1d", "input": input_data},
    )

    if response.status_code != 201:
        raise Exception(f"API error: {response.status_code} {response.text}")

    return poll_for_results(response.json()["id"])


def poll_for_results(prediction_id):
    max_timeout = 600
    poll_interval = 2
    headers = {"x-api-market-key": API_KEY}
    start_time = time.time()

    while time.time() - start_time < max_timeout:
        response = requests.get(f"{BASE_URL}/predictions/{prediction_id}", headers=headers)

        if response.status_code != 200:
            raise Exception(f"API error: {response.status_code} {response.text}")

        prediction = response.json()

        if prediction["status"] == "succeeded":
            return prediction["output"]

        if prediction["status"] in ("failed", "canceled"):
            raise Exception(f"Prediction {prediction['status']}: {prediction.get('error') or 'Unknown error'}")

        time.sleep(poll_interval)

    raise Exception("Prediction timed out after 10 minutes")


print(create_and_wait_for_prediction({
    "mask_type": "highlighted",
    "video_fps": 25,
    "input_video": "https://blog.api.market/wp-content/uploads/2026/10/robust_video_matting-input-api-market.mp4",
    "click_frames": "1",
    "click_labels": "1",
    "output_video": True,
    "output_format": "webp",
    "output_quality": 80,
    "annotation_type": "mask",
    "click_object_ids": "person",
    "click_coordinates": "[320,240]",
    "output_frame_interval": 1
}))
```

## FAQ
- **How do I select an object?**
  - Send one or more `[x,y]` points in `click_coordinates`, in pixels of the video frame. Mark points to include with `1` and points to exclude with `0` in `click_labels`.
- **Can I segment several objects at once?**
  - Yes. Add a click for each object and name them in `click_object_ids`, for example `person,dog`. Missing labels are generated automatically.
- **Why do I get many URLs back?**
  - `output_video` defaults to `false`, which returns an image sequence with one image per output frame. Set it to `true` to get a single video.
- **Why does `annotation_type` have no effect?**
  - It is ignored for `binary` and `greenscreen` masks. Use `mask_type` `highlighted` to see boxes.
- **How is a request priced?**
  - $0.02 per started 18 seconds of processing. The example took 45.2 seconds, so it cost 3 units ($0.06).
- **What if my prediction fails?**
  - Failed predictions are not billed. Read the `error` field, adjust the input and try again.

## Contact Us
If you have any questions, reach out to us through the chat on the bottom right of the page. You can also email our support team at hello@magicapi.com.


Segment and track objects through a video with Meta's SAM 2 from click points, with binary, highlighted or greenscreen masks and optional boxes.