Name: MuseTalk – Generate Lipsync Video | Free Plan | API.market
Brand: magicapi
SKU: api-magicapi-ai-lipsync-video-generator
Availability: InStock
Author: API Market

Shashank Agarwal

# 👄 Realistic Lipsync Video API (MuseTalk by tmappdev)

## Overview
The Realistic Lipsync Video API (MuseTalk by tmappdev) allows you to seamlessly synchronize spoken audio with a video portrait, generating an ultra-realistic lipsync effect. This is perfect for dubbing, virtual avatars, and creating automated video content at scale. Simply provide an audio file and a video file, and this API will output a video where lips match the provided audio.

![Sample input video for lipsync](https://replicate.delivery/pbxt/LzlR1mCzrpoMUA5GPk1RzxkriLjW5YMBmyJ2ieXbjARlgEBs/lily-ldwi8a_LdG-2.mp4)

## Features
- High-fidelity lipsync from separate audio tracks  
- Supports various video formats and frame rates  
- Adjustable bounding box shift (bbox_shift) for fine-tuning face alignment  
- Automated asynchronous processing for efficient batch operations  
- Fast turnaround times, typically under a few minutes for short clips  

## Use Cases
1. **Film and TV Dubbing**: Quickly localize video content into different languages while preserving realistic lip movements.  
2. **Virtual Avatars and Education**: Create talking-head avatars for e-learning modules, spokespersons, or interactive experiences.  
3. **Marketing and Advertising**: Launch dynamic ad campaigns where brand ambassadors instantly “speak” new promotions or voiceovers.  
4. **Social Media and Entertainment**: Generate short lipsynced videos for TikTok, Instagram, or YouTube to keep audiences engaged.  
5. **Customer Service and Tutorials**: Add a personal touch to tutorial or instructional videos with natural lipsync from prerecorded audio.

## Visual Examples

### Input Examples
- Audio File (MP3):  
  [Audio input example link](https://storage.googleapis.com/chattuesday/audios/img_9518da8f-71c3-4576-9dd2-4e6fc78ef9f4.mp3)  

- Video File (MP4):  
  ![Video input with a person speaking](https://replicate.delivery/pbxt/LzlR1mCzrpoMUA5GPk1RzxkriLjW5YMBmyJ2ieXbjARlgEBs/lily-ldwi8a_LdG-2.mp4)

### Output Example
Once the lipsync process completes, the API returns a link to a new video file. Below is a sample output video demonstrating how the lipsync aligns the spoken audio with the speaker’s lips:

![Output lipsynced video](https://replicate.delivery/yhqm/5QL3bYQH9y7qMJSy4mIdaeB1e79t4zRNkVGasldAVgerWHlnA/tmpgwijzrc3lily-ldwi8a_LdG-2_tmp5w74vyssimg_9518da8f-71c3-4576-9dd2-4e6fc78ef9f4.mp4)

## Pricing
- 1 API unit equals 1 image processed.  
- GET requests to check prediction status (GET /api/v1/magicapi/ai-lipsync-video-generator/predictions/{id}) are free.  
- Different pricing tiers are available based on usage volume.  

## API Documentation

### Base URL
All API requests should be made to:
```
https://prod.api.market/api/v1/magicapi/ai-lipsync-video-generator
```

### Authentication
Include your API key in the request header:
```
x-magicapi-key: YOUR_API_MARKET_KEY
```

### Asynchronous Processing
This API uses asynchronous processing for predictions:

1. Submit a prediction request to the POST endpoint.  
2. Receive a prediction ID in the response.  
3. Poll the GET endpoint with the prediction ID to check status.  
4. Continue polling until the prediction is complete.  

Processing time varies based on request size and length of the audio/video. Your code should:
- Poll every 1 second.
- Allow a maximum timeout of 600 seconds (10 minutes).
- Handle “starting” or “processing” statuses by continuing to poll.
- Handle “succeeded” status by retrieving the output and finishing.
- Handle “failed” status properly or stop if the timeout is reached.

### Endpoints

#### Create a Prediction
• **URL**: `POST /api/v1/magicapi/ai-lipsync-video-generator/predictions`  
• **Description**: Create a lipsync prediction by submitting audio and video files.  

**Request Body (JSON)**:
```json
{
  "version": "c54ce2fe673ea59b857b91250b3d71a2cd304a78f2370687632805c8405fbf4c",
  "input": {
    "audio_input": "https://storage.googleapis.com/chattuesday/audios/img_9518da8f-71c3-4576-9dd2-4e6fc78ef9f4.mp3",
    "video_input": "https://replicate.delivery/pbxt/LzlR1mCzrpoMUA5GPk1RzxkriLjW5YMBmyJ2ieXbjARlgEBs/lily-ldwi8a_LdG-2.mp4",
    "fps": 25,
    "bbox_shift": 0
  }
}
```
- **version**: model version (default provided above).  
- **input**:  
  - **audio_input** (required): A publicly accessible URL of your audio file (MP3, WAV, etc.).  
  - **video_input** (required): A publicly accessible URL of your source video.  
  - **fps** (optional): frames per second for the output video (default 25).  
  - **bbox_shift** (optional): integer to shift bounding box for better alignment (default 0).

**Example cURL**:
```bash
curl -X POST "https://prod.api.market/api/v1/magicapi/ai-lipsync-video-generator/predictions" \
  -H "Content-Type: application/json" \
  -H "x-magicapi-key: YOUR_API_MARKET_KEY" \
  -d '{
    "version": "c54ce2fe673ea59b857b91250b3d71a2cd304a78f2370687632805c8405fbf4c",
    "input": {
      "audio_input": "https://storage.googleapis.com/chattuesday/audios/img_9518da8f-71c3-4576-9dd2-4e6fc78ef9f4.mp3",
      "video_input": "https://replicate.delivery/pbxt/LzlR1mCzrpoMUA5GPk1RzxkriLjW5YMBmyJ2ieXbjARlgEBs/lily-ldwi8a_LdG-2.mp4",
      "fps": 25,
      "bbox_shift": 0
    }
  }'
```

**Response (JSON)**:
```json
{
  "id": "ufawqhfynnddngldkgtslldrkq",
  "version": "c54ce2fe673ea59b857b91250b3d71a2cd304a78f2370687632805c8405fbf4c",
  "status": "starting",
  "created_at": "2025-04-24T07:49:37.609Z"
}
```
- **id**: Unique identifier of the prediction request.  
- **status**: Starting status.  

#### Retrieve a Prediction
• **URL**: `GET /api/v1/magicapi/ai-lipsync-video-generator/predictions/{id}`  
• **Description**: Check the status and retrieve the result for a previously created prediction.  

**Path Parameter**:
- **id** (string): The unique prediction ID from the initial POST.

**Example cURL**:
```bash
curl -X GET "https://prod.api.market/api/v1/magicapi/ai-lipsync-video-generator/predictions/ufawqhfynnddngldkgtslldrkq" \
  -H "x-magicapi-key: YOUR_API_MARKET_KEY"
```

**Response (JSON)**:
```json
{
  "id": "ufawqhfynnddngldkgtslldrkq",
  "status": "succeeded",
  "output": "https://example.com/your-lipsynced-video.mp4",
  "error": null,
  "created_at": "2025-04-24T07:49:37.609Z",
  "started_at": "2025-04-24T07:49:38.000Z",
  "completed_at": "2025-04-24T07:50:10.000Z"
}
```
- **output**: Direct URL to the lipsynced video upon success.  

## Status Codes
- **200 OK**: Request succeeded and resource retrieved successfully.  
- **201 Created**: Resource created successfully (e.g. prediction initiated).  
- **400 Bad Request**: Invalid parameters or malformed JSON in the request.  
- **401 Unauthorized**: Missing or invalid API key.  
- **403 Forbidden**: API key is valid but lacks required permissions.  
- **404 Not Found**: Requested resource cannot be found (e.g. invalid prediction ID).  
- **429 Too Many Requests**: Rate limit or usage limit reached; try again later.  
- **500 Internal Server Error**: A server-side error occurred.  
- **503 Service Unavailable**: Service is temporarily unavailable or under heavy load.

## Rate Limits
Depending on your plan, there may be a limit on how many predictions you can create in a certain time frame. If you exceed the rate limit, you will receive a 429 Too Many Requests error. To handle this, implement an exponential backoff or similar retry strategy.

## Error Handling
When an error occurs, the API responds with JSON indicating the error reason. For example:

```json
{
  "error": {
    "code": "invalid_parameter",
    "message": "The parameter 'video_input' is required",
    "status": 400
  }
}
```
**Common errors**:
- Missing required fields (audio_input, video_input)
- Invalid URL or inaccessible media files
- Unauthorized request (invalid or missing API key)

**Recommended handling**:
- Check the error message to fix invalid parameters.
- Implement retries for transient errors (429 or 503).
- Log error details for debugging.

## Implementation Guide

Below are code snippets illustrating how to integrate with this API, handle authentication, and properly poll for longer-running jobs.

### Example Code

#### JavaScript (Node.js) Example
```javascript
const apiKey = "YOUR_API_MARKET_KEY";
const baseUrl = "https://prod.api.market/api/v1/magicapi/ai-lipsync-video-generator";

async function createAndWaitForPrediction(inputData) {
  try {
    // Step 1: Create the prediction
    const response = await fetch(`${baseUrl}/predictions`, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "x-magicapi-key": apiKey
      },
      body: JSON.stringify({
        version: "c54ce2fe673ea59b857b91250b3d71a2cd304a78f2370687632805c8405fbf4c",
        input: inputData
      })
    });

    if (!response.ok) {
      throw new Error(`API error: ${response.status} ${response.statusText}`);
    }

    const prediction = await response.json();
    const predictionId = prediction.id;
    
    // Step 2: Poll for results with timeout
    return await pollForResults(predictionId);
  } catch (error) {
    console.error("Error:", error);
    throw error;
  }
}

function delay(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function pollForResults(predictionId) {
  const MAX_TIMEOUT = 600000; // 600 seconds (10 minutes)
  const POLL_INTERVAL = 1000; // 1 second

  const startTime = Date.now();

  while (Date.now() - startTime < MAX_TIMEOUT) {
    // Check prediction status
    const response = await fetch(`${baseUrl}/predictions/${predictionId}`, {
      headers: {
        "x-magicapi-key": apiKey
      }
    });

    if (!response.ok) {
      throw new Error(`API error: ${response.status} ${response.statusText}`);
    }

    const prediction = await response.json();

    // If succeeded, return the output video URL
    if (prediction.status === "succeeded") {
      return prediction.output;
    }

    // If failed, throw an error
    if (prediction.status === "failed") {
      throw new Error(`Prediction failed: ${prediction.error || "Unknown error"}`);
    }

    // Wait before the next poll
    await delay(POLL_INTERVAL);
  }

  throw new Error("Prediction timed out after 10 minutes");
}

// Example usage
(async () => {
  const lipsyncedVideoUrl = await createAndWaitForPrediction({
    audio_input: "https://storage.googleapis.com/chattuesday/audios/img_9518da8f-71c3-4576-9dd2-4e6fc78ef9f4.mp3",
    video_input: "https://replicate.delivery/pbxt/LzlR1mCzrpoMUA5GPk1RzxkriLjW5YMBmyJ2ieXbjARlgEBs/lily-ldwi8a_LdG-2.mp4",
    fps: 25,
    bbox_shift: 0
  });
  console.log("Lipsynced video URL:", lipsyncedVideoUrl);
})();
```

#### Python Example
```python
import requests
import time

API_KEY = "YOUR_API_MARKET_KEY"
BASE_URL = "https://prod.api.market/api/v1/magicapi/ai-lipsync-video-generator"

def create_and_wait_for_prediction(input_data):
    # Step 1: Create the prediction
    headers = {
        "Content-Type": "application/json",
        "x-magicapi-key": API_KEY
    }
    
    response = requests.post(
        f"{BASE_URL}/predictions",
        headers=headers,
        json={
            "version": "c54ce2fe673ea59b857b91250b3d71a2cd304a78f2370687632805c8405fbf4c",
            "input": input_data
        }
    )
    
    if response.status_code not in (200, 201):
        raise Exception(f"API error: {response.status_code} {response.text}")

    prediction = response.json()
    prediction_id = prediction["id"]
    
    # Step 2: Poll for results with timeout
    return poll_for_results(prediction_id)

def poll_for_results(prediction_id):
    MAX_TIMEOUT = 600  # 600 seconds
    POLL_INTERVAL = 1  # 1 second

    headers = {"x-magicapi-key": API_KEY}
    start_time = time.time()

    while time.time() - start_time < MAX_TIMEOUT:
        response = requests.get(
            f"{BASE_URL}/predictions/{prediction_id}",
            headers=headers
        )
        
        if response.status_code != 200:
            raise Exception(f"API error: {response.status_code} {response.text}")
        
        prediction = response.json()
        
        if prediction["status"] == "succeeded":
            return prediction["output"]
        
        if prediction["status"] == "failed":
            error_msg = prediction.get("error", "Unknown error")
            raise Exception(f"Prediction failed: {error_msg}")

        time.sleep(POLL_INTERVAL)

    raise Exception("Prediction timed out after 10 minutes")

# Example usage
if __name__ == "__main__":
    output_url = create_and_wait_for_prediction({
        "audio_input": "https://storage.googleapis.com/chattuesday/audios/img_9518da8f-71c3-4576-9dd2-4e6fc78ef9f4.mp3",
        "video_input": "https://replicate.delivery/pbxt/LzlR1mCzrpoMUA5GPk1RzxkriLjW5YMBmyJ2ieXbjARlgEBs/lily-ldwi8a_LdG-2.mp4",
        "fps": 25,
        "bbox_shift": 0
    })
    print("Lipsynced video available at:", output_url)
```

#### cURL Example
```bash
# Step 1: Create a prediction request
curl -X POST "https://prod.api.market/api/v1/magicapi/ai-lipsync-video-generator/predictions" \
  -H "Content-Type: application/json" \
  -H "x-magicapi-key: YOUR_API_MARKET_KEY" \
  -d '{
    "version": "c54ce2fe673ea59b857b91250b3d71a2cd304a78f2370687632805c8405fbf4c",
    "input": {
      "audio_input": "https://storage.googleapis.com/chattuesday/audios/img_9518da8f-71c3-4576-9dd2-4e6fc78ef9f4.mp3",
      "video_input": "https://replicate.delivery/pbxt/LzlR1mCzrpoMUA5GPk1RzxkriLjW5YMBmyJ2ieXbjARlgEBs/lily-ldwi8a_LdG-2.mp4",
      "fps": 25,
      "bbox_shift": 0
    }
  }'

# Response example:
# {
#   "id": "abcd1234",
#   "version": "c54ce2fe673ea59b857b91250b3d71a2cd304a78f2370687632805c8405fbf4c",
#   "status": "starting",
#   "created_at": "2025-04-24T07:49:37.609Z"
# }

# Step 2: Poll the GET endpoint until status is "succeeded"
curl -X GET "https://prod.api.market/api/v1/magicapi/ai-lipsync-video-generator/predictions/abcd1234" \
  -H "x-magicapi-key: YOUR_API_MARKET_KEY"
```

### Performance Optimization
- Use appropriate video resolutions to balance quality vs. processing speed.  
- Keep audio files short if only a segment of lipsync is required.  
- Use efficient polling intervals (1 second is sufficient in most cases).  
- Consider concurrency limits when creating multiple predictions simultaneously.

## FAQ
1. **Does the API support stereo or multi-channel audio?**  
   The API works best with mono or stereo audio files. Multi-channel files may be downmixed.

2. **Can I specify the output resolution?**  
   Currently, the resolution matches the input video’s native resolution. Bbox shifting can help with face alignment, but not resolution changes.

3. **How long does it take to lipsync a typical clip?**  
   Most short clips (under 30 seconds) complete within 1-2 minutes. Longer clips can take more time but usually finish within a few minutes.

4. **Can I adjust the lipsync sensitivity?**  
   Fine-tuning is handled automatically. Tweaking “bbox_shift” helps in some edge cases.

5. **How do I handle errors like ‘invalid media URL’?**  
   Ensure the audio or video file is publicly accessible at a valid HTTPS URL and re-try.

## Contact Us
If you have any questions or need assistance, you can reach out to us through the embedded chat on the bottom right corner of the page or send an email to hello[at]api.market.

Generate realistic lipsync videos from an audio track and a portrait. High-fidelity results are perfect for dubbing. Start free on API.market.