Name: Isaac 0.1 API: Visual Q&A with Object Grounding
Brand: magicapi
SKU: api-magicapi-visual-question-answering-object-grounding-perceptron-ai-inc-isaac-0-1
Availability: InStock
Author: API Market

Shashank Agarwal

# 🔎 Visual Question Answering and Object Grounding API (Perceptron Isaac 0.1)

## Overview
Isaac 0.1 is a vision-language model from Perceptron. Its developer describes it as an open-source, 2B-parameter model built for real-world applications.

Send an `image` and a question or instruction as `prompt`. The model answers in text and, depending on `response`, also locates the objects it refers to as bounding boxes, points or polygons. Each request returns an object with the answer `text`, the located regions as a JSON string in `structured_output`, and an `image` URL with the regions drawn on the input image.

## Features
- **Answers about an image**: ask a question in `prompt` and get a written answer in `text`.
- **Grounded output**: `response` chooses how found objects are returned: `box` (default) for bounding boxes, `point` for coordinates, `polygon` for shapes, or `text` for a description only.
- **Annotated image**: `image` in the output is a picture of the input with the located regions drawn on it.
- **Structured results**: `structured_output` is a JSON string you can parse, with one entry per located object (type, coordinates and the object `mention`).
- **Answer length**: `max_new_tokens` (1 to 512, default 256) caps the generated text.

## Use Cases
1. **Scene questions**: ask whether a situation is safe, what is happening or what is visible, and get the answer with the evidence marked, as in the example.
2. **Object localization**: find and box named objects in photos for inspection or review tools.
3. **Pointing and outlining**: get points or polygons for objects to drive cropping, highlighting or UI overlays.
4. **Image descriptions**: set `response` to `text` for a written description without regions.

## Visual Examples
### Example Input
![Input image: a street scene with a traffic light](https://blog.api.market/wp-content/uploads/2026/10/isaac-0-1-input-api-market.webp)

Prompt: "Determine whether it is safe to cross the street. Look for signage and moving traffic."

`response` is `box` and `max_new_tokens` is `256`.

### Example Output
![Isaac 0.1 output: the street scene with a box drawn around the traffic light](https://blog.api.market/wp-content/uploads/2026/10/isaac-0-1-output-ffcpakb6-api-market.jpg)

`text`: "No, it is not safe to cross the street at this time. The traffic light in the background is showing a red signal, which means it's not safe or legal to cross the street. This red light indicates that vehicles have the right of way, and pedestrians should wait until the light changes before proceeding to cross. It's important to always follow traffic signals for your safety and the safety of others on the road. When in doubt, it's best to wait until the signal changes to green before crossing the street."

`structured_output` (a JSON string, shown parsed):

```json
[
  {
    "type": "box",
    "top_left": {
      "x": 808,
      "y": 248
    },
    "bottom_right": {
      "x": 863,
      "y": 386
    },
    "mention": "traffic light"
  }
]
```

## Input Parameters
| Field | Type | Required | Default | Allowed values | Description |
| --- | --- | --- | --- | --- | --- |
| `response` | string | No | `box` | `text`, `box`, `point`, `polygon` | Response type: `box` for bounding boxes, `point` for coordinates, `polygon` for shapes, `text` for a text description only |
| `image` | string (URL) | Yes | none |  | Input image for analysis (URL) |
| `max_new_tokens` | integer | No | `256` | 1 to 512 | Maximum number of tokens to generate |
| `prompt` | string | No | `Determine whether it is safe to cross the street. Look for signage and moving traffic.` |  | Question or instruction for the model |

## Pricing
- **Pay As You Go (PAYG) pricing**: you only pay for what you use.
- **Price**: $0.0028 per API unit.
- **1 API unit = 1,000 weighted tokens.** An output token counts 1, an input token counts 0.1199.
- That works out to **$0.33572 per million input tokens** and **$2.80 per million output tokens**.
- The input image is counted as input tokens together with the prompt.
- Each request is rounded up to a whole API unit, so the smallest charge for a request is $0.0028.
- **Example cost**: the request below used 1,579 input tokens (image and prompt) and 138 output tokens: 1,579 x 0.1199 + 138 = 327.3221 weighted tokens, rounded up to 1 API unit = **$0.0028**.
- **Failed or cancelled predictions are not billed.**
- **GET requests to check prediction status** are free and don't count toward API unit usage.
- **No monthly subscription fee.**

## API Documentation

### Base URL
All API requests go to: `https://prod.api.market/api/v1/magicapi/visual-question-answering-object-grounding-perceptron-ai-inc-isaac-0-1`

### Authentication
Send your api.market API key in the `x-api-market-key` header:
```
x-api-market-key: YOUR_API_MARKET_KEY
```

### Asynchronous Processing
**IMPORTANT**: This API is asynchronous. The flow is:
1. Send a POST request to `/predictions`. A successful request returns **201 Created** with a prediction `id` and a `status` of `starting` or `processing`.
2. Poll GET `/predictions/{id}` with that ID.
3. Keep polling while `status` is `starting` or `processing`.
4. Stop when `status` is `succeeded` (read `output`), `failed` or `canceled` (read `error`).

A polling interval of 1 second and an overall timeout of 600 seconds (10 minutes) work well.

### Endpoints

#### Create a Prediction
```bash
curl -X POST "https://prod.api.market/api/v1/magicapi/visual-question-answering-object-grounding-perceptron-ai-inc-isaac-0-1/predictions" \
  -H "Content-Type: application/json" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY" \
  -d '{
    "version": "perceptron-ai-inc/isaac-0.1",
    "input": {
      "image": "https://blog.api.market/wp-content/uploads/2026/10/isaac-0-1-input-api-market.webp",
      "prompt": "Determine whether it is safe to cross the street. Look for signage and moving traffic.",
      "response": "box",
      "max_new_tokens": 256
    }
  }'
```

Response (201 Created):
```json
{
  "id": "ffcpakb6ydrmw0d1d189dnfypm",
  "version": "perceptron-ai-inc/isaac-0.1",
  "status": "starting",
  "created_at": "2026-10-23T10:31:02.291Z"
}
```

#### Check Prediction Status
```bash
curl -X GET "https://prod.api.market/api/v1/magicapi/visual-question-answering-object-grounding-perceptron-ai-inc-isaac-0-1/predictions/{id}" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY"
```

Response when the prediction has finished (200 OK):
```json
{
  "id": "ffcpakb6ydrmw0d1d189dnfypm",
  "status": "succeeded",
  "output": {
    "image": "https://blog.api.market/wp-content/uploads/2026/10/isaac-0-1-output-ffcpakb6-api-market.jpg",
    "structured_output": "[\n  {\n    \"type\": \"box\",\n    \"top_left\": {\n      \"x\": 808,\n      \"y\": 248\n    },\n    \"bottom_right\": {\n      \"x\": 863,\n      \"y\": 386\n    },\n    \"mention\": \"traffic light\"\n  }\n]",
    "text": "No, it is not safe to cross the street at this time. The traffic light in the background is showing a red signal, which means it's not safe or legal to cross the street. This red light indicates that vehicles have the right of way, and pedestrians should wait until the light changes before proceeding to cross. It's important to always follow traffic signals for your safety and the safety of others on the road. When in doubt, it's best to wait until the signal changes to green before crossing the street."
  },
  "error": null,
  "created_at": "2026-10-23T10:31:02.291Z",
  "started_at": "2026-10-23T10:31:02.791Z",
  "completed_at": "2026-10-23T10:31:05.643Z"
}
```

`output` is an object: `text` is the written answer, `structured_output` is a JSON string with the located regions (it can be null), and `image` is the URL of the input image with those regions drawn on it.

### Status Codes
- **201 Created**: prediction created (POST `/predictions`). This is the normal success response.
- **200 OK**: prediction details returned (GET `/predictions/{id}`).
- **400 Bad Request**: invalid input, for example a missing `image`, a `response` that is not in the list, or a `max_new_tokens` above 512.
- **401 Unauthorized**: missing or invalid API key.
- **404 Not Found**: no prediction with that ID.
- **429 Too Many Requests**: rate limit exceeded.
- **500 Internal Server Error** or **503 Service Unavailable**: temporary server problem; retry later.

## Rate Limits
All plans allow 10 requests per second. Requests above that return `429 Too Many Requests`; back off before retrying and honour any `Retry-After` header. Polling a prediction with GET is free but still counts toward the per-second limit, so poll every 1 to 5 seconds rather than in a tight loop.

## Error Handling
- Check the HTTP status code of every request and read the response body for the error message.
- A prediction can also end with `status` set to `failed`; the `error` field then explains why. Failed predictions are not billed.
- Retry 429, 500 and 503 responses with exponential backoff. Fix the input before retrying a 400.

## Implementation Guide
### Example Code

#### JavaScript
```javascript
const apiKey = "YOUR_API_MARKET_KEY";
const baseUrl = "https://prod.api.market/api/v1/magicapi/visual-question-answering-object-grounding-perceptron-ai-inc-isaac-0-1";

async function createAndWaitForPrediction(input) {
  const response = await fetch(`${baseUrl}/predictions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-api-market-key": apiKey
    },
    body: JSON.stringify({ version: "perceptron-ai-inc/isaac-0.1", input })
  });

  if (response.status !== 201) {
    throw new Error(`API error: ${response.status} ${await response.text()}`);
  }

  const prediction = await response.json();
  return pollForResults(prediction.id);
}

const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function pollForResults(predictionId) {
  const maxTimeout = 600000;
  const pollInterval = 1000;
  const startTime = Date.now();

  while (Date.now() - startTime < maxTimeout) {
    const response = await fetch(`${baseUrl}/predictions/${predictionId}`, {
      headers: { "x-api-market-key": apiKey }
    });

    if (!response.ok) {
      throw new Error(`API error: ${response.status} ${await response.text()}`);
    }

    const prediction = await response.json();

    if (prediction.status === "succeeded") {
      return prediction.output;
    }

    if (prediction.status === "failed" || prediction.status === "canceled") {
      throw new Error(`Prediction ${prediction.status}: ${prediction.error || "Unknown error"}`);
    }

    await delay(pollInterval);
  }

  throw new Error("Prediction timed out after 10 minutes");
}

createAndWaitForPrediction({
  "image": "https://blog.api.market/wp-content/uploads/2026/10/isaac-0-1-input-api-market.webp",
  "prompt": "Determine whether it is safe to cross the street. Look for signage and moving traffic.",
  "response": "box",
  "max_new_tokens": 256
}).then((output) => console.log(output.text, output.structured_output ? JSON.parse(output.structured_output) : null, output.image));
```

#### Python
```python
import time
import requests

API_KEY = "YOUR_API_MARKET_KEY"
BASE_URL = "https://prod.api.market/api/v1/magicapi/visual-question-answering-object-grounding-perceptron-ai-inc-isaac-0-1"


def create_and_wait_for_prediction(input_data):
    headers = {"Content-Type": "application/json", "x-api-market-key": API_KEY}
    response = requests.post(
        f"{BASE_URL}/predictions",
        headers=headers,
        json={"version": "perceptron-ai-inc/isaac-0.1", "input": input_data},
    )

    if response.status_code != 201:
        raise Exception(f"API error: {response.status_code} {response.text}")

    return poll_for_results(response.json()["id"])


def poll_for_results(prediction_id):
    max_timeout = 600
    poll_interval = 1
    headers = {"x-api-market-key": API_KEY}
    start_time = time.time()

    while time.time() - start_time < max_timeout:
        response = requests.get(f"{BASE_URL}/predictions/{prediction_id}", headers=headers)

        if response.status_code != 200:
            raise Exception(f"API error: {response.status_code} {response.text}")

        prediction = response.json()

        if prediction["status"] == "succeeded":
            return prediction["output"]

        if prediction["status"] in ("failed", "canceled"):
            raise Exception(f"Prediction {prediction['status']}: {prediction.get('error') or 'Unknown error'}")

        time.sleep(poll_interval)

    raise Exception("Prediction timed out after 10 minutes")


output = create_and_wait_for_prediction({
    "image": "https://blog.api.market/wp-content/uploads/2026/10/isaac-0-1-input-api-market.webp",
    "prompt": "Determine whether it is safe to cross the street. Look for signage and moving traffic.",
    "response": "box",
    "max_new_tokens": 256
})
print(output["text"])
print(output["structured_output"])
print(output["image"])
```

## FAQ
- **What does `response` change?**
  - `box` returns bounding boxes, `point` returns coordinates, `polygon` returns shapes and `text` returns only the written answer. The default is `box`.
- **How do I read the located objects?**
  - `structured_output` is a string. Parse it as JSON to get a list with one entry per object, each with its `type`, coordinates and the `mention` it refers to.
- **Why is my answer cut off?**
  - `max_new_tokens` limits the generated text (default 256, maximum 512). Raise it for longer answers.
- **How much does one image cost?**
  - The image is counted as input tokens. In the example, one image plus the prompt came to 1,579 input tokens, which at $0.33572 per million is about $0.0005 before rounding; each request is rounded up to at least one unit ($0.0028).
- **What if my prediction fails?**
  - Failed predictions are not billed. Read the `error` field, adjust the input and try again.

## Contact Us
If you have any questions, reach out to us through the chat on the bottom right of the page. You can also email our support team at hello@magicapi.com.


Ask Perceptron's Isaac 0.1 about an image and get a written answer plus bounding boxes, points or polygons for the objects it mentions.