Name: CogVLM2 – Analyze Video & Images | Free · $5/mo | API.market
Brand: magicapi
SKU: api-magicapi-video-visual-language-understanding-api
Availability: InStock
Author: API Market

Shashank Agarwal

# 🎥 Video and Image Language Understanding API (CogVLM2)

## Overview
Welcome to the Video and Image Language Understanding API (CogVLM2). This API allows you to analyze video and image content in real time to generate rich scene descriptions, object detections, and textual insights. By leveraging visual language modeling, CogVLM2 produces contextualized observations and coherent narratives around your media.

Key highlights:
• Seamlessly handle both video and image inputs  
• Automated scene description and object detection  
• Flexible prompting for custom analysis scenarios  
• Designed for asynchronous processing via a simple predictive workflow  

Below is a comprehensive guide to help you get started, integrate the API, manage asynchronous calls, and handle error conditions.

## Features
- Advanced Multimodal Analysis: Understand video frames and image content at a deep semantic level.  
- Natural Language Descriptions: Generate descriptive paragraphs about scenes, objects, and activities.  
- Flexible Prompting: Customize the analysis by providing specific prompts, enabling targeted insights.  
- Real-Time Operation: Asynchronous architecture suited for handling large media files efficiently.  
- Adjustable Output Quality: Fine-tune outputs using parameters like max tokens, temperature, and top_p.  

## Use Cases
1. Video Content Summaries: Generate an executive summary for any uploaded video, identifying key objects or scenes.  
2. Image Captioning for Accessibility: Provide text-based interpretations for images to assist visually impaired users.  
3. Interactive Teaching Tools: Use context-specific prompts to teach or explain elements in educational videos or images.  
4. Marketing & Branding Analysis: Identify products or brand elements within promotional footage for inventory or reasons of compliance.  
5. Automated Documentation: Generate textual descriptions for large archives of videos and images for indexing and retrieval.

## Visual Examples
Below are examples illustrating the type of media you can provide to the API.

### Example Input Video
Prompt: "Please provide a description of this video."

![Sample input video featuring an elephant in a grassland](https://replicate.delivery/pbxt/LgGGFqrlw37TQMMWbjCTvYwp1vhH916HGqjKIuLxyB5SqiuT/%E5%A4%A7%E8%B1%A1.mp4)

• This video shows a large elephant moving across a grassy field under a clear sky.  
• You might prompt the model to describe the scenery, objects, or the elephant’s behavior.

### Example Image Input
In some code samples, you might see an example referencing an image:
![Sample image input showing a placeholder example](https://example.com/image.jpg)

• This represents any image you want to analyze.  
• You can prompt the model with instructions such as “Describe the objects in the image” or “Identify main colors and scene composition.”

## Pricing
• One API unit equals one video processed.  
• GET requests to check the prediction status are free and do not count toward API usage.  
• Pricing tiers vary based on usage volume (contact us for plan details).

## API Documentation

### Base URL
All API requests should be made to:
```
https://prod.api.market/api/v1/magicapi/video-visual-language-understanding-api
```

### Authentication
Include your API key in every request header:

```
x-magicapi-key: YOUR_API_MARKET_KEY
```

### Asynchronous Processing
**Important**: This API uses asynchronous processing for predictions. The typical flow is:
1. Submit a prediction request with a POST call.  
2. Receive a prediction ID in the response.  
3. Poll the GET endpoint with the prediction ID to check status.  
4. Continue polling until the prediction status is “succeeded” or an error occurs.

Recommended guidelines:
- Poll once every 1 second.  
- Implement a 600-second (10-minute) maximum timeout.  
- Handle status = "failed" or a timeout gracefully.  

### Endpoints

#### 1. Create a Prediction
• URL:  
  POST /api/v1/magicapi/video-visual-language-understanding-api/predictions  

• Description:  
  Creates a new prediction using CogVLM2. Accepts the media URL and additional parameters (e.g., prompt, temperature settings). Returns an ID to track the prediction’s status.

• Example Request:
```bash
curl -X POST "https://prod.api.market/api/v1/magicapi/video-visual-language-understanding-api/predictions" \
  -H "Content-Type: application/json" \
  -H "x-magicapi-key: YOUR_API_MARKET_KEY" \
  -d '{
    "input": {
      "input_video": "https://replicate.delivery/pbxt/LgGGFqrlw37TQMMWbjCTvYwp1vhH916HGqjKIuLxyB5SqiuT/%E5%A4%A7%E8%B1%A1.mp4",
      "prompt": "Describe the events in this video focusing on the elephant",
      "temperature": 0.1,
      "top_p": 0.1,
      "max_new_tokens": 2048
    }
  }'
```

• Example Response (HTTP 201):
```json
{
  "id": "ufawqhfynnddngldkgtslldrkq",
  "version": "9da7e9a554d36bb7b5fec36b43b00e4616dc1e819bc963ded8e053d8d8196cb5",
  "status": "starting",
  "created_at": "2025-04-23T20:02:23.291Z"
}
```
- “id”: unique identifier for tracking this prediction.  
- “status” can be one of [starting, processing, succeeded, failed, canceled].  
- “created_at”: timestamp for when the prediction was initiated.

#### 2. Get Prediction Status
• URL:  
  GET /api/v1/magicapi/video-visual-language-understanding-api/predictions/{id}  

• Description:  
  Retrieves the current status and results (if completed) for a prediction request.

• Example Request:
```bash
curl -X GET "https://prod.api.market/api/v1/magicapi/video-visual-language-understanding-api/predictions/ufawqhfynnddngldkgtslldrkq" \
  -H "x-magicapi-key: YOUR_API_MARKET_KEY"
```

• Example Response (HTTP 200) when succeeded:
```json
{
  "id": "ufawqhfynnddngldkgtslldrkq",
  "status": "succeeded",
  "output": "In the video, we see a large elephant walking across a dry grassland...",
  "created_at": "2023-01-01T12:00:00.000Z",
  "started_at": "2023-01-01T12:00:05.000Z",
  "completed_at": "2023-01-01T12:01:00.000Z"
}
```
- “status”: once it is “succeeded,” “output” contains the final description.  
- “output”: text describing the events, objects, and context of the video or image.

## Status Codes
• 200 OK: Request successful.  
• 201 Created: Resource successfully created (e.g., prediction initiated).  
• 400 Bad Request: Invalid input or malformed request.  
• 401 Unauthorized: Missing or invalid API key.  
• 403 Forbidden: API key valid but lacks necessary permissions.  
• 404 Not Found: Resource could not be located (e.g., invalid prediction ID).  
• 429 Too Many Requests: Rate limit exceeded; slow down requests or upgrade plan.  
• 500 Internal Server Error: A problem occurred on the server side.  
• 503 Service Unavailable: Temporary server overload or maintenance.

## Rate Limits
Requests may be subject to rate limiting depending on your subscription plan. If you receive a 429 Too Many Requests error, wait briefly before retrying or upgrade to a higher tier if your use case consistently exceeds your current plan.

## Error Handling
Always check the HTTP status code and the response payload for error details. A typical error response might look like:
```json
{
  "error": {
    "code": "invalid_parameter",
    "message": "The parameter 'input_video' is required",
    "status": 400
  }
}
```
• Inspect the “message” field for actionable details.  
• In case of 5xx errors, implement a retry strategy with exponential backoff.  
• For 4xx errors, correct your request parameters or authentication details before retrying.

## Implementation Guide

Below are sample implementations in JavaScript, Python, and cURL. They illustrate creating a prediction, polling for status, and handling errors accordingly.

### JavaScript (Node.js)

```javascript
// JavaScript example with proper polling and timeout
const apiKey = "YOUR_API_MARKET_KEY";
const baseUrl = "https://prod.api.market/api/v1/magicapi/video-visual-language-understanding-api";

async function createAndWaitForPrediction(inputData) {
  try {
    // Step 1: Create the prediction
    const response = await fetch(`${baseUrl}/predictions`, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "x-magicapi-key": apiKey
      },
      body: JSON.stringify({ input: inputData })
    });

    if (!response.ok) {
      throw new Error(`API error: ${response.status} ${response.statusText}`);
    }

    const prediction = await response.json();
    const predictionId = prediction.id;

    // Step 2: Poll for results with timeout
    return await pollForResults(predictionId);
  } catch (error) {
    console.error("Error:", error);
    throw error;
  }
}

function delay(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function pollForResults(predictionId) {
  const MAX_TIMEOUT = 600000; // 600 seconds (10 minutes)
  const POLL_INTERVAL = 1000; // 1 second between checks
  const startTime = Date.now();

  while (Date.now() - startTime < MAX_TIMEOUT) {
    // Check prediction status
    const response = await fetch(`${baseUrl}/predictions/${predictionId}`, {
      headers: {
        "x-magicapi-key": apiKey
      }
    });

    if (!response.ok) {
      throw new Error(`API error: ${response.status} ${response.statusText}`);
    }

    const prediction = await response.json();

    // Check status
    if (prediction.status === "succeeded") {
      return prediction.output;
    }
    if (prediction.status === "failed") {
      throw new Error(`Prediction failed: ${prediction.error || "Unknown error"}`);
    }

    // Wait before next poll
    await delay(POLL_INTERVAL);
  }

  throw new Error("Prediction timed out after 10 minutes");
}
```

### Python

```python
# Python example with proper polling and timeout
import requests
import time

API_KEY = "YOUR_API_MARKET_KEY"
BASE_URL = "https://prod.api.market/api/v1/magicapi/video-visual-language-understanding-api"

def create_and_wait_for_prediction(input_data):
    # Step 1: Create the prediction
    headers = {
        "Content-Type": "application/json",
        "x-magicapi-key": API_KEY
    }

    response = requests.post(
        f"{BASE_URL}/predictions",
        headers=headers,
        json={"input": input_data}
    )

    if response.status_code not in [200, 201]:
        raise Exception(f"API error: {response.status_code} {response.text}")

    prediction = response.json()
    prediction_id = prediction["id"]

    # Step 2: Poll for results with timeout
    return poll_for_results(prediction_id)

def poll_for_results(prediction_id):
    MAX_TIMEOUT = 600  # 600 seconds (10 minutes)
    POLL_INTERVAL = 1  # 1 second
    start_time = time.time()

    while time.time() - start_time < MAX_TIMEOUT:
        response = requests.get(
            f"{BASE_URL}/predictions/{prediction_id}",
            headers={"x-magicapi-key": API_KEY}
        )

        if response.status_code != 200:
            raise Exception(f"API error: {response.status_code} {response.text}")

        prediction = response.json()

        if prediction["status"] == "succeeded":
            return prediction["output"]
        if prediction["status"] == "failed":
            error_msg = prediction.get("error", "Unknown error")
            raise Exception(f"Prediction failed: {error_msg}")

        time.sleep(POLL_INTERVAL)

    raise Exception("Prediction timed out after 10 minutes")
```

### cURL

```bash
# Step 1: Create a prediction
curl -X POST "https://prod.api.market/api/v1/magicapi/video-visual-language-understanding-api/predictions" \
  -H "Content-Type: application/json" \
  -H "x-magicapi-key: YOUR_API_MARKET_KEY" \
  -d '{
    "input": {
      "input_video": "https://replicate.delivery/pbxt/LgGGFqrlw37TQMMWbjCTvYwp1vhH916HGqjKIuLxyB5SqiuT/%E5%A4%A7%E8%B1%A1.mp4",
      "prompt": "Describe what happens in the elephant video"
    }
  }'

# Response should include a prediction ID, e.g. {"id": "abcd1234", "status": "starting"}

# Step 2: Poll the status until "status" is "succeeded"
curl -X GET "https://prod.api.market/api/v1/magicapi/video-visual-language-understanding-api/predictions/abcd1234" \
  -H "x-magicapi-key: YOUR_API_MARKET_KEY"
```

### Performance Optimization
- Submit only necessary frames for short videos or reduce image resolution if high detail is not needed.  
- Shorten prompt length if you only need concise answers. Lowering max_new_tokens can also speed responses and save costs.  
- Use a robust caching strategy if analyzing identical content multiple times.

## FAQ
1. **What’s the maximum video size?** The size limit depends on your plan. Contact us if your files exceed standard tiers.  
2. **Can I process multiple videos in one request?** One request handles one video. You can submit multiple requests in parallel if needed.  
3. **What is the typical latency?** Typical processing depends on video length and complexity. Short videos often complete within seconds; longer videos can take up to a few minutes.  
4. **How do I handle partial scene descriptions?** Adjust parameters like temperature and max_new_tokens to refine the output.  

## Contact Us
If you have any questions or need assistance, you can reach out to us through the embedded chat on the bottom right corner of the page or send an email to hello[at]api.market.

Analyze video & images to detect objects and describe scenes. Free 20 calls/mo, from $5/mo. Start free on API.market.

CogVLM2 – Analyze Video & Images | Free · $5/mo | API.market

🎥 Video and Image Language Understanding API (CogVLM2)

FREE

MagicAPI

PAYG

START

ULTRA

ELITE

MEGA

GIGA

BULK

Analyze videos and images with advanced AI for rich scene, object, and text insights in real time using CogVLM2.