Name: Llama 3 70B Instruct API: Chat Completion by Meta
Brand: magicapi
SKU: api-magicapi-chat-completions-ai-api-meta-meta-llama-3-70b-instruct
Availability: InStock
Author: API Market

Shashank Agarwal

# 💬 Chat Completion API (Meta Llama 3 70B Instruct)

## Overview
Meta Llama 3 70B Instruct is a 70 billion parameter language model from Meta, fine-tuned for chat completions.

Send a `prompt` (and optionally a `system_prompt`) and the model returns its answer as a list of text chunks. You control the length with `max_tokens` and `min_tokens`, the sampling with `temperature`, `top_p` and `top_k`, repetition with `presence_penalty`, and where generation stops with `stop_sequences`. The `prompt_template` field lets you change how the system prompt and prompt are wrapped in Llama 3's chat format.

## Features
- **System prompt**: `system_prompt` (default `You are a helpful assistant`) is inserted before the prompt by the default template.
- **Length control**: `max_tokens` (default 512, minimum 1) and `min_tokens` (minimum -1; set -1 to disable).
- **Sampling**: `temperature` 0 to 5 (default 0.7), `top_p` 0 to 1 (default 0.95), `top_k` (default 0, minimum -1).
- **Repetition and length penalties**: `presence_penalty` (default 0) and `length_penalty` 0 to 5 (default 1; below 1 favors shorter outputs, above 1 longer ones).
- **Stop sequences**: a comma-separated list in `stop_sequences`.
- **Prompt template**: `prompt_template` must contain `{prompt}`; the default also contains `{system_prompt}`.
- **Seed**: set `seed` to make sampling repeatable.

## Use Cases
1. **Chat assistants**: answer user questions with a `system_prompt` that sets the assistant's role.
2. **Step-by-step reasoning**: work through word problems and explain each step, as in the example.
3. **Summaries and rewrites**: condense or rephrase text pasted into `prompt`.
4. **Drafting**: write emails, descriptions or outlines from short instructions.

## Examples
### Example Input
```json
{
  "top_p": 0.9,
  "prompt": "Work through this problem step by step:\n\nQ: Sarah has 7 llamas. Her friend gives her 3 more trucks of llamas. Each truck has 5 llamas. How many llamas does Sarah have in total?",
  "max_tokens": 512,
  "min_tokens": 0,
  "temperature": 0.6,
  "prompt_template": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "presence_penalty": 1.15,
  "frequency_penalty": 0.2
}
```

This request sets its own `prompt_template` with the system message written into it ("You are a helpful assistant"), so `system_prompt` is not used. It also sends `frequency_penalty`, which is not one of this model's input fields (see the table below).

### Example Output
The joined output text:

```text
Let's break this problem down step by step.

Step 1: Sarah already has 7 llamas.

Step 2: Her friend gives her 3 trucks of llamas. We need to find out how many llamas are in these 3 trucks.

Step 3: Each truck has 5 llamas, so we multiply the number of trucks (3) by the number of llamas per truck (5):

3 trucks x 5 llamas/truck = 3 x 5 = 15 llamas

Step 4: Sarah already had 7 llamas, and now she gets 15 more llamas from the trucks. To find the total number of llamas, we add the two numbers together:

7 llamas (she already had) + 15 llamas (from the trucks) = 7 + 15 = 22 llamas

Therefore, Sarah has 22 llamas in total.
```

## Input Parameters
| Field | Type | Required | Default | Allowed values | Description |
| --- | --- | --- | --- | --- | --- |
| `prompt` | string | Yes |  |  | Prompt to send to the model |
| `system_prompt` | string | No | `You are a helpful assistant` |  | System prompt to send to the model. It is prepended to the prompt and helps guide the model's behavior |
| `max_tokens` | integer | No | `512` | minimum 1 | Maximum number of tokens to generate. A word is generally 2 to 3 tokens |
| `min_tokens` | integer | No | none | minimum -1 | Minimum number of tokens to generate. Set to -1 to disable. A word is generally 2 to 3 tokens |
| `temperature` | number | No | `0.7` | 0 to 5 | Adjusts randomness of outputs: greater than 1 is random and 0 is deterministic; 0.75 is a good starting value |
| `top_p` | number | No | `0.95` | 0 to 1 | When decoding text, samples from the top p percentage of most likely tokens; lower it to ignore less likely tokens |
| `top_k` | integer | No | `0` | minimum -1 | When decoding text, samples from the top k most likely tokens; lower it to ignore less likely tokens |
| `stop_sequences` | string | No | `<\|end_of_text\|>,<\|eot_id\|>` |  | A comma-separated list of sequences to stop generation at. For example, `<end>,<stop>` stops generation at the first instance of `<end>` or `<stop>` |
| `length_penalty` | number | No | `1` | 0 to 5 | Controls how long the outputs are. Below 1 the model tends to generate shorter outputs, above 1 longer outputs |
| `presence_penalty` | number | No | `0` |  | Penalizes repeated tokens regardless of how often they appear. Higher values make the model less likely to repeat tokens |
| `seed` | integer | No | none |  | Random seed. Leave blank to randomize the seed |
| `prompt_template` | string | No | see below |  | Template for formatting the prompt. Can be any string, but must contain the substring `{prompt}` |
| `log_performance_metrics` | boolean | No | `false` |  | Log performance metrics for the request |
| `max_new_tokens` | integer | No | none | minimum 1 | Older name for `max_tokens`, kept for backwards compatibility. Use `max_tokens` instead; do not set both |
| `min_new_tokens` | integer | No | none | minimum -1 | Older name for `min_tokens`, kept for backwards compatibility. Use `min_tokens` instead; do not set both |

Default `prompt_template`:
```text
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
```

## Pricing
- **Pay As You Go (PAYG) pricing**: you only pay for what you use.
- **Price**: $0.0035 per API unit.
- **1 API unit = 1,000 weighted tokens.** An output token counts 1, an input token counts 0.2364.
- That works out to **$0.8274 per million input tokens** and **$3.50 per million output tokens**.
- Each request is rounded up to a whole API unit, so the smallest charge for a request is $0.0035.
- **Example cost**: the request below used 68 input tokens and 190 output tokens: 68 x 0.2364 + 190 = 206.0752 weighted tokens, rounded up to 1 API unit = **$0.0035**.
- **Failed or cancelled predictions are not billed.**
- **GET requests to check prediction status** are free and don't count toward API unit usage.
- **No monthly subscription fee.**

## API Documentation

### Base URL
All API requests go to: `https://prod.api.market/api/v1/magicapi/chat-completions-ai-api-meta-meta-llama-3-70b-instruct`

### Authentication
Send your api.market API key in the `x-api-market-key` header:
```
x-api-market-key: YOUR_API_MARKET_KEY
```

### Asynchronous Processing
**IMPORTANT**: This API is asynchronous. The flow is:
1. Send a POST request to `/predictions`. A successful request returns **201 Created** with a prediction `id` and a `status` of `starting` or `processing`.
2. Poll GET `/predictions/{id}` with that ID.
3. Keep polling while `status` is `starting` or `processing`.
4. Stop when `status` is `succeeded` (read `output`), `failed` or `canceled` (read `error`).

A polling interval of 1 second and an overall timeout of 600 seconds (10 minutes) work well.

### Endpoints

#### Create a Prediction
```bash
curl -X POST "https://prod.api.market/api/v1/magicapi/chat-completions-ai-api-meta-meta-llama-3-70b-instruct/predictions" \
  -H "Content-Type: application/json" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY" \
  -d '{
    "version": "meta/meta-llama-3-70b-instruct",
    "input": {
      "top_p": 0.9,
      "prompt": "Work through this problem step by step:\n\nQ: Sarah has 7 llamas. Her friend gives her 3 more trucks of llamas. Each truck has 5 llamas. How many llamas does Sarah have in total?",
      "max_tokens": 512,
      "min_tokens": 0,
      "temperature": 0.6,
      "prompt_template": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
      "presence_penalty": 1.15,
      "frequency_penalty": 0.2
    }
  }'
```

Response (201 Created):
```json
{
  "id": "dckb68vnssrne0d1c6sswtzs3c",
  "version": "meta/meta-llama-3-70b-instruct",
  "status": "starting",
  "created_at": "2026-10-22T03:41:53.682Z"
}
```

#### Check Prediction Status
```bash
curl -X GET "https://prod.api.market/api/v1/magicapi/chat-completions-ai-api-meta-meta-llama-3-70b-instruct/predictions/{id}" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY"
```

Response when the prediction has finished (200 OK):
```json
{
  "id": "dckb68vnssrne0d1c6sswtzs3c",
  "status": "succeeded",
  "output": ["Let", "'s", " break", " this", " problem", " down", " step", " by", " step", ".\n\n", "Step", " ", "1", ":", " Sarah", " already", " has", " ", "7", " ll", "amas", ".\n\n", "Step", " ", "2", ":", " Her", " friend", " gives", " her", " ", "3", " trucks", " of", " ll", "amas", ".", " We", " need", " to", " find", " out", " how", " many", " ll", "amas", " are", " in", " these", " ", "3", " trucks", ".\n\n", "Step", " ", "3", ":", " Each", " truck", " has", " ", "5", " ll", "amas", ",", " so", " we", " multiply", " the", " number", " of", " trucks", " (", "3", ")", " by", " the", " number", " of", " ll", "amas", " per", " truck", " (", "5", "):\n\n", "3", " trucks", " x", " ", "5", " ll", "amas", "/tr", "uck", " =", " ", "3", " x", " ", "5", " =", " ", "15", " ll", "amas", "\n\n", "Step", " ", "4", ":", " Sarah", " already", " had", " ", "7", " ll", "amas", ",", " and", " now", " she", " gets", " ", "15", " more", " ll", "amas", " from", " the", " trucks", ".", " To", " find", " the", " total", " number", " of", " ll", "amas", ",", " we", " add", " the", " two", " numbers", " together", ":\n\n", "7", " ll", "amas", " (", "she", " already", " had", ")", " +", " ", "15", " ll", "amas", " (", "from", " the", " trucks", ")", " =", " ", "7", " +", " ", "15", " =", " ", "22", " ll", "amas", "\n\n", "Therefore", ",", " Sarah", " has", " ", "22", " ll", "amas", " in", " total", "."],
  "error": null,
  "created_at": "2026-10-22T03:41:53.682Z",
  "started_at": "2026-10-22T03:41:54.182Z",
  "completed_at": "2026-10-22T03:41:57.221Z"
}
```

`output` is a list of text chunks in the order the model produced them. Join them with no separator to get the full response.

### Status Codes
- **201 Created**: prediction created (POST `/predictions`). This is the normal success response.
- **200 OK**: prediction details returned (GET `/predictions/{id}`).
- **400 Bad Request**: invalid input, for example a missing `prompt`, a `temperature` above 5 or both `max_tokens` and `max_new_tokens` set.
- **401 Unauthorized**: missing or invalid API key.
- **404 Not Found**: no prediction with that ID.
- **429 Too Many Requests**: rate limit exceeded.
- **500 Internal Server Error** or **503 Service Unavailable**: temporary server problem; retry later.

## Rate Limits
All plans allow 10 requests per second. Requests above that return `429 Too Many Requests`; back off before retrying and honour any `Retry-After` header. Polling a prediction with GET is free but still counts toward the per-second limit, so poll every 1 to 5 seconds rather than in a tight loop.

## Error Handling
- Check the HTTP status code of every request and read the response body for the error message.
- A prediction can also end with `status` set to `failed`; the `error` field then explains why. Failed predictions are not billed.
- Retry 429, 500 and 503 responses with exponential backoff. Fix the input before retrying a 400.

## Implementation Guide
### Example Code

#### JavaScript
```javascript
const apiKey = "YOUR_API_MARKET_KEY";
const baseUrl = "https://prod.api.market/api/v1/magicapi/chat-completions-ai-api-meta-meta-llama-3-70b-instruct";

async function createAndWaitForPrediction(input) {
  const response = await fetch(`${baseUrl}/predictions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-api-market-key": apiKey
    },
    body: JSON.stringify({ version: "meta/meta-llama-3-70b-instruct", input })
  });

  if (response.status !== 201) {
    throw new Error(`API error: ${response.status} ${await response.text()}`);
  }

  const prediction = await response.json();
  return pollForResults(prediction.id);
}

const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function pollForResults(predictionId) {
  const maxTimeout = 600000;
  const pollInterval = 1000;
  const startTime = Date.now();

  while (Date.now() - startTime < maxTimeout) {
    const response = await fetch(`${baseUrl}/predictions/${predictionId}`, {
      headers: { "x-api-market-key": apiKey }
    });

    if (!response.ok) {
      throw new Error(`API error: ${response.status} ${await response.text()}`);
    }

    const prediction = await response.json();

    if (prediction.status === "succeeded") {
      return prediction.output;
    }

    if (prediction.status === "failed" || prediction.status === "canceled") {
      throw new Error(`Prediction ${prediction.status}: ${prediction.error || "Unknown error"}`);
    }

    await delay(pollInterval);
  }

  throw new Error("Prediction timed out after 10 minutes");
}

createAndWaitForPrediction({
  "top_p": 0.9,
  "prompt": "Work through this problem step by step:\n\nQ: Sarah has 7 llamas. Her friend gives her 3 more trucks of llamas. Each truck has 5 llamas. How many llamas does Sarah have in total?",
  "max_tokens": 512,
  "min_tokens": 0,
  "temperature": 0.6,
  "prompt_template": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
  "presence_penalty": 1.15,
  "frequency_penalty": 0.2
}).then((output) => console.log(output.join("")));
```

#### Python
```python
import time
import requests

API_KEY = "YOUR_API_MARKET_KEY"
BASE_URL = "https://prod.api.market/api/v1/magicapi/chat-completions-ai-api-meta-meta-llama-3-70b-instruct"

def create_and_wait_for_prediction(input_data):
    headers = {"Content-Type": "application/json", "x-api-market-key": API_KEY}
    response = requests.post(
        f"{BASE_URL}/predictions",
        headers=headers,
        json={"version": "meta/meta-llama-3-70b-instruct", "input": input_data},
    )

    if response.status_code != 201:
        raise Exception(f"API error: {response.status_code} {response.text}")

    return poll_for_results(response.json()["id"])

def poll_for_results(prediction_id):
    max_timeout = 600
    poll_interval = 1
    headers = {"x-api-market-key": API_KEY}
    start_time = time.time()

    while time.time() - start_time < max_timeout:
        response = requests.get(f"{BASE_URL}/predictions/{prediction_id}", headers=headers)

        if response.status_code != 200:
            raise Exception(f"API error: {response.status_code} {response.text}")

        prediction = response.json()

        if prediction["status"] == "succeeded":
            return prediction["output"]

        if prediction["status"] in ("failed", "canceled"):
            raise Exception(f"Prediction {prediction['status']}: {prediction.get('error') or 'Unknown error'}")

        time.sleep(poll_interval)

    raise Exception("Prediction timed out after 10 minutes")

output = create_and_wait_for_prediction({
    "top_p": 0.9,
    "prompt": "Work through this problem step by step:\n\nQ: Sarah has 7 llamas. Her friend gives her 3 more trucks of llamas. Each truck has 5 llamas. How many llamas does Sarah have in total?",
    "max_tokens": 512,
    "min_tokens": 0,
    "temperature": 0.6,
    "prompt_template": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
    "presence_penalty": 1.15,
    "frequency_penalty": 0.2
})
print("".join(output))
```

## FAQ
- **How do I get the full answer as one string?**
  - `output` is a list of text chunks. Join them in order with no separator: `output.join("")` in JavaScript or `"".join(output)` in Python.
- **Why is my `system_prompt` ignored?**
  - The system prompt is inserted through the `{system_prompt}` placeholder in `prompt_template`. If you send your own template without that placeholder, as the example does, `system_prompt` has no effect.
- **How do I make answers repeatable?**
  - Set `seed`, and use a low `temperature` (0 is deterministic).
- **Should I use `max_tokens` or `max_new_tokens`?**
  - Use `max_tokens` and `min_tokens`. `max_new_tokens` and `min_new_tokens` are older names kept for backwards compatibility; do not set both forms.
- **How is a short request billed?**
  - Input and output tokens are weighted and summed, then rounded up to whole API units of 1,000 weighted tokens. The example request (68 input and 190 output tokens) used 1 unit, $0.0035.
- **What if my prediction fails?**
  - Failed predictions are not billed. Read the `error` field, adjust the input and try again.

## Contact Us
If you have any questions, reach out to us through the chat on the bottom right of the page. You can also email our support team at hello@magicapi.com.


Run Meta Llama 3 70B Instruct for chat answers, summaries and step-by-step reasoning with configurable sampling. Pay as you go, no monthly fee.