Name: Florence-2 API: Image Captioning, Object Detection and OCR
Brand: magicapi
SKU: api-magicapi-unified-vision-representation-api-lucataco-florence-2-large
Availability: InStock
Author: API Market

Shashank Agarwal

# 🔎 Image Captioning, Object Detection and OCR API (Microsoft Florence-2 Large)

## Overview
Florence-2 Large is a vision model released by Microsoft. Its paper title describes it as a unified representation for a variety of vision tasks: one model handles captioning, detection, grounding and text reading.

Send an `image` and pick a task in `task_input`: `Caption`, `Detailed Caption`, `More Detailed Caption`, `Caption to Phrase Grounding`, `Object Detection`, `Dense Region Caption`, `Region Proposal`, `OCR` or `OCR with Region`. Tasks that take text, such as phrase grounding, read it from `text_input`. The response holds the result as text and, for tasks that find regions, an image with the result drawn on it.

## Features
- **Nine tasks in one model**: captions at three levels of detail, phrase grounding, object detection, dense region captions, region proposals, OCR and OCR with regions.
- **Bounding boxes and labels**: detection tasks return boxes and labels for what the model finds.
- **Text reading**: `OCR` returns the text in the image; `OCR with Region` also returns where each piece of text is.
- **Annotated image**: `img` in the output is an image with the result drawn on it, when the task produces one.
- **Optional text input**: `text_input` passes a phrase to tasks that use text.

## Use Cases
1. **Alt text and image descriptions**: generate captions for images in a CMS or product catalog.
2. **Object detection**: find and label objects in photos, as in the car example below.
3. **Reading text in images**: pull text from signs, screenshots and labels with `OCR` or `OCR with Region`.
4. **Locating a described object**: use `Caption to Phrase Grounding` with a phrase in `text_input` to find where it is.

## Visual Examples
### Example Input
![Input image: a car](https://blog.api.market/wp-content/uploads/2026/10/florence-2-large-input-api-market.jpg)

Task: `task_input` set to `Object Detection`. No `text_input`.

### Example Output
`img`, the input image with the detected boxes drawn on it:

![Florence-2 Large object detection output with boxes around the car and the door handle](https://blog.api.market/wp-content/uploads/2026/10/florence-2-large-output-41vamtss-api-market.png)

`text`, the detection result:

```
{'<OD>': {'bboxes': [[33.599998474121094, 160.55999755859375, 596.7999877929688, 371.7599792480469], [271.67999267578125, 242.1599884033203, 302.3999938964844, 246.95999145507812]], 'labels': ['car', 'door handle']}}
```

The model found two objects, `car` and `door handle`. Each box is four pixel coordinates (x1, y1, x2, y2).

## Input Parameters
| Field | Type | Required | Default | Allowed values | Description |
| --- | --- | --- | --- | --- | --- |
| `image` | string (URL) | Yes | none |  | Input image URL |
| `task_input` | string | No | `Caption` | `Caption`, `Detailed Caption`, `More Detailed Caption`, `Caption to Phrase Grounding`, `Object Detection`, `Dense Region Caption`, `Region Proposal`, `OCR`, `OCR with Region` | Vision task to run on the image |
| `text_input` | string | No | none |  | Optional text input for tasks that use text, such as `Caption to Phrase Grounding` |

## Pricing
- **Pay As You Go (PAYG) pricing**: you only pay for what you use.
- **Price**: $0.0022 per API unit.
- **1 API unit = 1 request with up to 2 seconds of processing.** Longer requests use one unit per started 2-second block (for example, 3 seconds of processing = 2 units).
- Every task is billed the same way.
- **Example cost**: the object detection request above processed in 0.83 seconds = 1 API unit = **$0.0022**.
- **Failed or cancelled predictions are not billed.**
- **GET requests to check prediction status** are free and don't count toward API unit usage.
- **No monthly subscription fee.**

## API Documentation

### Base URL
All API requests go to: `https://prod.api.market/api/v1/magicapi/unified-vision-representation-api-lucataco-florence-2-large`

### Authentication
Send your api.market API key in the `x-api-market-key` header:
```
x-api-market-key: YOUR_API_MARKET_KEY
```

### Asynchronous Processing
**IMPORTANT**: This API is asynchronous. The flow is:
1. Send a POST request to `/predictions`. A successful request returns **201 Created** with a prediction `id` and a `status` of `starting` or `processing`.
2. Poll GET `/predictions/{id}` with that ID.
3. Keep polling while `status` is `starting` or `processing`.
4. Stop when `status` is `succeeded` (read `output`), `failed` or `canceled` (read `error`).

A polling interval of 1 second and an overall timeout of 600 seconds (10 minutes) work well.

### Endpoints

#### Create a Prediction
```bash
curl -X POST "https://prod.api.market/api/v1/magicapi/unified-vision-representation-api-lucataco-florence-2-large/predictions" \
  -H "Content-Type: application/json" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY" \
  -d '{
    "version": "da53547e17d45b9cfb48174b2f18af8b83ca020fa76db62136bf9c6616762595",
    "input": {
      "image": "https://blog.api.market/wp-content/uploads/2026/10/florence-2-large-input-api-market.jpg",
      "task_input": "Object Detection"
    }
  }'
```

Response (201 Created):
```json
{
  "id": "41vamtssmnrmt0d1c6mv9evpfw",
  "version": "da53547e17d45b9cfb48174b2f18af8b83ca020fa76db62136bf9c6616762595",
  "status": "starting",
  "created_at": "2026-10-22T03:30:42.914Z"
}
```

#### Check Prediction Status
```bash
curl -X GET "https://prod.api.market/api/v1/magicapi/unified-vision-representation-api-lucataco-florence-2-large/predictions/{id}" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY"
```

Response when the prediction has finished (200 OK):
```json
{
  "id": "41vamtssmnrmt0d1c6mv9evpfw",
  "status": "succeeded",
  "output": {
    "img": "https://blog.api.market/wp-content/uploads/2026/10/florence-2-large-output-41vamtss-api-market.png",
    "text": "{'<OD>': {'bboxes': [[33.599998474121094, 160.55999755859375, 596.7999877929688, 371.7599792480469], [271.67999267578125, 242.1599884033203, 302.3999938964844, 246.95999145507812]], 'labels': ['car', 'door handle']}}"
  },
  "error": null,
  "created_at": "2026-10-22T03:30:42.914Z",
  "started_at": "2026-10-22T03:30:43.547Z",
  "completed_at": "2026-10-22T03:30:44.372Z"
}
```

`output` is an object. `text` (always present) holds the task result. `img` is the URL of an image with the result drawn on it; it may be absent for text-only tasks such as captions. Note that `text` uses Python dictionary notation with single quotes, not JSON: in Python, read it with `ast.literal_eval`.

### Status Codes
- **201 Created**: prediction created (POST `/predictions`). This is the normal success response.
- **200 OK**: prediction details returned (GET `/predictions/{id}`).
- **400 Bad Request**: invalid input, for example a missing `image` or a `task_input` that is not in the list.
- **401 Unauthorized**: missing or invalid API key.
- **404 Not Found**: no prediction with that ID.
- **429 Too Many Requests**: rate limit exceeded.
- **500 Internal Server Error** or **503 Service Unavailable**: temporary server problem; retry later.

## Rate Limits
All plans allow 10 requests per second. Requests above that return `429 Too Many Requests`; back off before retrying and honour any `Retry-After` header. Polling a prediction with GET is free but still counts toward the per-second limit, so poll every 1 to 5 seconds rather than in a tight loop.
## Error Handling
- Check the HTTP status code of every request and read the response body for the error message.
- A prediction can also end with `status` set to `failed`; the `error` field then explains why. Failed predictions are not billed.
- Retry 429, 500 and 503 responses with exponential backoff. Fix the input before retrying a 400.

## Implementation Guide
### Example Code

#### JavaScript
```javascript
const apiKey = "YOUR_API_MARKET_KEY";
const baseUrl = "https://prod.api.market/api/v1/magicapi/unified-vision-representation-api-lucataco-florence-2-large";

async function createAndWaitForPrediction(input) {
  const response = await fetch(`${baseUrl}/predictions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-api-market-key": apiKey
    },
    body: JSON.stringify({ version: "da53547e17d45b9cfb48174b2f18af8b83ca020fa76db62136bf9c6616762595", input })
  });

  if (response.status !== 201) {
    throw new Error(`API error: ${response.status} ${await response.text()}`);
  }

  const prediction = await response.json();
  return pollForResults(prediction.id);
}

const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function pollForResults(predictionId) {
  const maxTimeout = 600000;
  const pollInterval = 1000;
  const startTime = Date.now();

  while (Date.now() - startTime < maxTimeout) {
    const response = await fetch(`${baseUrl}/predictions/${predictionId}`, {
      headers: { "x-api-market-key": apiKey }
    });

    if (!response.ok) {
      throw new Error(`API error: ${response.status} ${await response.text()}`);
    }

    const prediction = await response.json();

    if (prediction.status === "succeeded") {
      return prediction.output;
    }

    if (prediction.status === "failed" || prediction.status === "canceled") {
      throw new Error(`Prediction ${prediction.status}: ${prediction.error || "Unknown error"}`);
    }

    await delay(pollInterval);
  }

  throw new Error("Prediction timed out after 10 minutes");
}

createAndWaitForPrediction({
    "image": "https://blog.api.market/wp-content/uploads/2026/10/florence-2-large-input-api-market.jpg",
    "task_input": "Object Detection"
  }).then((output) => console.log(output));
```

#### Python
```python
import time
import requests

API_KEY = "YOUR_API_MARKET_KEY"
BASE_URL = "https://prod.api.market/api/v1/magicapi/unified-vision-representation-api-lucataco-florence-2-large"

def create_and_wait_for_prediction(input_data):
    headers = {"Content-Type": "application/json", "x-api-market-key": API_KEY}
    response = requests.post(
        f"{BASE_URL}/predictions",
        headers=headers,
        json={"version": "da53547e17d45b9cfb48174b2f18af8b83ca020fa76db62136bf9c6616762595", "input": input_data},
    )

    if response.status_code != 201:
        raise Exception(f"API error: {response.status_code} {response.text}")

    return poll_for_results(response.json()["id"])

def poll_for_results(prediction_id):
    max_timeout = 600
    poll_interval = 1
    headers = {"x-api-market-key": API_KEY}
    start_time = time.time()

    while time.time() - start_time < max_timeout:
        response = requests.get(f"{BASE_URL}/predictions/{prediction_id}", headers=headers)

        if response.status_code != 200:
            raise Exception(f"API error: {response.status_code} {response.text}")

        prediction = response.json()

        if prediction["status"] == "succeeded":
            return prediction["output"]

        if prediction["status"] in ("failed", "canceled"):
            raise Exception(f"Prediction {prediction['status']}: {prediction.get('error') or 'Unknown error'}")

        time.sleep(poll_interval)

    raise Exception("Prediction timed out after 10 minutes")

print(create_and_wait_for_prediction({
    "image": "https://blog.api.market/wp-content/uploads/2026/10/florence-2-large-input-api-market.jpg",
    "task_input": "Object Detection"
}))
```

## FAQ
- **Which tasks are supported?**
  - `Caption`, `Detailed Caption`, `More Detailed Caption`, `Caption to Phrase Grounding`, `Object Detection`, `Dense Region Caption`, `Region Proposal`, `OCR` and `OCR with Region`. The default is `Caption`.
- **How do I parse the `text` field?**
  - It is written in Python dictionary notation (single quotes), for example `{'<OD>': {'bboxes': [...], 'labels': [...]}}`. In Python use `ast.literal_eval(output["text"])`; in other languages convert it to JSON before parsing.
- **When do I need `text_input`?**
  - For tasks that take a phrase, such as `Caption to Phrase Grounding`. The other tasks only need the image.
- **What are the box coordinates?**
  - Four numbers per box: x1, y1, x2, y2 in pixels of the input image.
- **How much does a request cost?**
  - $0.0022 for each started 2 seconds of processing. The example above took under a second.
- **Is the 201 response from POST an error?**
  - No. 201 Created means the prediction was accepted. Use the returned `id` to poll GET `/predictions/{id}` until `status` is `succeeded`, `failed` or `canceled`.
- **What if my prediction fails?**
  - Failed predictions are not billed. Read the `error` field, adjust the input and try again.

## Contact Us
If you have any questions or need assistance, you can reach out to us through the embedded chat on the bottom right corner of the page or send an email to hello@magicapi.com.


Florence-2 Large API by Microsoft: caption images, detect objects, ground phrases and read text with one vision model. Low cost, pay as you go.