Name: Llama 3 8B Instruct API: Open Source Chat LLM by Meta
Brand: magicapi
SKU: api-magicapi-chatbot-language-model-meta-meta-llama-3-8b-instruct
Availability: InStock
Author: API Market

Shashank Agarwal

# 💬 Chat LLM API (Meta Llama 3 8B Instruct)

## Overview
Meta Llama 3 8B Instruct is an 8 billion parameter language model from Meta, fine-tuned for chat completions.

Send a `prompt` and an optional `system_prompt`. The model places both into `prompt_template` (by default the Llama 3 chat format) and returns generated text. Length, sampling, stop sequences and the seed are all adjustable per request.

## Features
- **System prompt**: `system_prompt` (default "You are a helpful assistant") is prepended to the prompt to guide the model's behavior.
- **Length controls**: `max_tokens` (default 512, minimum 1), `min_tokens` (minimum -1; -1 disables it) and `length_penalty` (0 to 5; below 1 favors shorter outputs, above 1 longer ones).
- **Sampling controls**: `temperature` (0 to 5, default 0.7), `top_p` (0 to 1, default 0.95) and `top_k` (default 0, minimum -1).
- **Repetition control**: `presence_penalty` (default 0) makes repeated tokens less likely as it increases.
- **Stop sequences**: a comma-separated list in `stop_sequences` (default `<|end_of_text|>,<|eot_id|>`).
- **Custom prompt template**: `prompt_template` can be any string that contains `{prompt}`.
- **Reproducible runs**: set `seed`; leave it blank for a random seed.

## Use Cases
1. **Chat assistants**: short conversational answers steered by a system prompt.
2. **Lightweight text tasks**: summaries, rewrites, classification prompts and extraction prompts where a small model is enough.
3. **Prototyping prompts**: test prompt templates and stop sequences cheaply before moving to a larger model.
4. **Batch text generation**: product descriptions, FAQ answers and similar short texts.

## Visual Examples
### Example Input
Prompt: "Johnny has 8 billion parameters. His friend Tommy has 70 billion parameters. What does this mean when it comes to speed?"

System prompt: "You are a helpful assistant". Settings: `max_new_tokens` 512, `temperature` 0.7, `top_p` 0.95, `length_penalty` 1, `presence_penalty` 0, `stop_sequences` `<|end_of_text|>,<|eot_id|>`, and the Llama 3 chat `prompt_template` shown in the request below.

### Example Output
The chunks of this run joined into one answer:

````markdown
When we talk about "parameters" in the context of machine learning models, we're referring to the number of independent variables that a model uses to make predictions or take actions. In other words, it's the number of "knobs" or "settings" that a model uses to learn and adapt to new data.

In terms of speed, having more parameters doesn't necessarily mean that a model is faster or slower. However, it can have an impact on the model's performance and efficiency in certain ways:

1. **Computational complexity**: Models with more parameters require more computational resources to train and evaluate, which can increase training times and slow down the model's performance. This is because the model needs to process more data and perform more calculations to update its parameters.
2. **Overfitting**: Models with many parameters are more prone to overfitting, which occurs when a model becomes too specialized to the training data and fails to generalize well to new, unseen data. Overfitting can lead to slower performance and decreased accuracy.
3. **Model size**: Larger models with more parameters require more memory and storage space, which can impact the model's deployment and usage. For example, a model with 70 billion parameters like Tommy's might be more challenging to deploy on a mobile device or a resource-constrained server.
4. **Training time**: Models with more parameters often require more training data and longer training times to converge. This can be a challenge, especially for large-scale models like Tommy's, which may require significant computational resources and data storage.

In summary, while having more parameters doesn't directly affect a model's speed, it can impact its performance, efficiency, and deployability.
````

## Input Parameters
| Field | Type | Required | Default | Allowed values | Description |
| --- | --- | --- | --- | --- | --- |
| `prompt` | string | Yes | none |  | Prompt to send to the model |
| `system_prompt` | string | No | `You are a helpful assistant` |  | System prompt to send to the model. It is prepended to the prompt and helps guide system behavior. |
| `max_tokens` | integer | No | `512` | minimum 1 | Maximum number of tokens to generate. A word is generally 2-3 tokens. |
| `min_tokens` | integer | No | none | minimum -1 | Minimum number of tokens to generate. To disable, set to -1. A word is generally 2-3 tokens. |
| `temperature` | number | No | `0.7` | 0 to 5 | Adjusts randomness of outputs: greater than 1 is random and 0 is deterministic; 0.75 is a good starting value |
| `top_p` | number | No | `0.95` | 0 to 1 | When decoding text, samples from the top p percentage of most likely tokens; lower to ignore less likely tokens |
| `top_k` | integer | No | `0` | minimum -1 | When decoding text, samples from the top k most likely tokens; lower to ignore less likely tokens |
| `stop_sequences` | string | No | `<\|end_of_text\|>,<\|eot_id\|>` |  | A comma-separated list of sequences to stop generation at. For example, `<end>,<stop>` stops generation at the first instance of `<end>` or `<stop>`. |
| `length_penalty` | number | No | `1` | 0 to 5 | Controls how long the outputs are. Below 1 the model tends to generate shorter outputs; above 1, longer outputs. |
| `presence_penalty` | number | No | `0` |  | Penalizes repeated tokens regardless of the number of appearances. As the value increases, the model is less likely to repeat tokens. |
| `seed` | integer | No | none |  | Random seed. Leave blank to randomize the seed. |
| `prompt_template` | string | No | Llama 3 chat format (see below) |  | Template for formatting the prompt. Can be any string, but must contain the substring `{prompt}`. |
| `log_performance_metrics` | boolean | No | `false` |  | Whether to log performance metrics for the request |
| `max_new_tokens` | integer | No | none | minimum 1 | Older name for `max_tokens`, kept for backwards compatibility. Use `max_tokens` instead; do not send both. |
| `min_new_tokens` | integer | No | none | minimum -1 | Older name for `min_tokens`, kept for backwards compatibility. Use `min_tokens` instead; do not send both. |

Default `prompt_template`:
```
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
```

## Pricing
- **Pay As You Go (PAYG) pricing**: you only pay for what you use.
- **Price**: $0.0032 per API unit.
- **1 API unit = 10,000 weighted tokens**; an output token counts 1, an input token counts 0.2.
- That is **$0.064 per million input tokens** and **$0.32 per million output tokens**.
- Each request is rounded up to a whole API unit, so the smallest charge per request is $0.0032.
- **Example cost**: the request above used 46 input tokens and 342 output tokens. 46 x 0.2 + 342 = 351.2 weighted tokens, rounded up to 1 API unit = **$0.0032**.
- **Failed or cancelled predictions are not billed.**
- **GET requests to check prediction status** are free and don't count toward API unit usage.
- **No monthly subscription fee.**

## API Documentation

### Base URL
All API requests go to: `https://prod.api.market/api/v1/magicapi/chatbot-language-model-meta-meta-llama-3-8b-instruct`

### Authentication
Send your api.market API key in the `x-api-market-key` header:
```
x-api-market-key: YOUR_API_MARKET_KEY
```

### Asynchronous Processing
**IMPORTANT**: This API is asynchronous. The flow is:
1. Send a POST request to `/predictions`. A successful request returns **201 Created** with a prediction `id` and a `status` of `starting` or `processing`.
2. Poll GET `/predictions/{id}` with that ID.
3. Keep polling while `status` is `starting` or `processing`.
4. Stop when `status` is `succeeded` (read `output`), `failed` or `canceled` (read `error`).

A polling interval of 1 second and an overall timeout of 600 seconds (10 minutes) work well.

### Endpoints

#### Create a Prediction
```bash
curl -X POST "https://prod.api.market/api/v1/magicapi/chatbot-language-model-meta-meta-llama-3-8b-instruct/predictions" \
  -H "Content-Type: application/json" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY" \
  -d '{
    "version": "meta/meta-llama-3-8b-instruct",
    "input": {
      "top_p": 0.95,
      "prompt": "Johnny has 8 billion parameters. His friend Tommy has 70 billion parameters. What does this mean when it comes to speed?",
      "temperature": 0.7,
      "system_prompt": "You are a helpful assistant",
      "length_penalty": 1,
      "max_new_tokens": 512,
      "stop_sequences": "<|end_of_text|>,<|eot_id|>",
      "prompt_template": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
      "presence_penalty": 0
    }
  }'
```

Response (201 Created):
```json
{
  "id": "f6vm7p472drna0d1c6srp7mdgm",
  "version": "meta/meta-llama-3-8b-instruct",
  "status": "starting",
  "created_at": "2026-10-22T03:41:58.102Z"
}
```

#### Check Prediction Status
```bash
curl -X GET "https://prod.api.market/api/v1/magicapi/chatbot-language-model-meta-meta-llama-3-8b-instruct/predictions/{id}" \
  -H "x-api-market-key: YOUR_API_MARKET_KEY"
```

Response when the prediction has finished (200 OK):
```json
{
  "id": "f6vm7p472drna0d1c6srp7mdgm",
  "status": "succeeded",
  "output": [
    "When we talk about \"parameters\" in the context of machine learning models, we're referring to the number of independent variables that a model uses to make predictions or take actions. In other words, it's the number of \"knobs\" or \"settings\" that a model uses to learn and adapt to new data.\n\nIn terms of speed, having more parameters doesn't necessarily mean that a model is faster or slower. However, it can have an impact on the model's performance and efficiency in certain ways:\n\n1. **Computational complexity**: Models with more parameters require more computational resources to train and evaluate, which can increase training times and slow down the model's performance. This is because the model needs to process more data and perform more calculations to update its parameters.\n2. **Overfitting**: Models with many parameters are more prone to overfitting, which occurs when a model becomes too specialized to the training data and fails to generalize well to new, unseen data. Overfitting can lead to slower performance and decreased accuracy.\n3. **Model size**: Larger models with more parameters require more memory and storage space, which can impact the model's deployment and usage. For example, a model with 70 billion parameters like Tommy's might be more challenging to deploy on a mobile device or a resource-constrained server.\n4. **Training time**: Models with more parameters often require more training data and longer training times to converge. This can be a challenge, especially for large-scale models like Tommy's, which may require significant computational resources and data storage.\n\nIn summary, while having more parameters doesn't directly affect a model's speed, it can impact its performance, efficiency, and deployability."
  ],
  "error": null,
  "created_at": "2026-10-22T03:41:58.102Z",
  "started_at": "2026-10-22T03:41:58.108Z",
  "completed_at": "2026-10-22T03:42:02.640Z"
}
```

`output` is an array of text chunks. Join them in order to get the full answer. The example above shows the chunks from this run already joined into one string.

### Status Codes
- **201 Created**: prediction created (POST `/predictions`). This is the normal success response.
- **200 OK**: prediction details returned (GET `/predictions/{id}`).
- **400 Bad Request**: invalid input, for example a missing `prompt` or a `temperature` above 5.
- **401 Unauthorized**: missing or invalid API key.
- **404 Not Found**: no prediction with that ID.
- **429 Too Many Requests**: rate limit exceeded.
- **500 Internal Server Error** or **503 Service Unavailable**: temporary server problem; retry later.

## Rate Limits
All plans allow 10 requests per second. Requests above that return `429 Too Many Requests`; back off before retrying and honour any `Retry-After` header. Polling a prediction with GET is free but still counts toward the per-second limit, so poll every 1 to 5 seconds rather than in a tight loop.

## Error Handling
- Check the HTTP status code of every request and read the response body for the error message.
- A prediction can also end with `status` set to `failed`; the `error` field then explains why. Failed predictions are not billed.
- Retry 429, 500 and 503 responses with exponential backoff. Fix the input before retrying a 400.

## Implementation Guide
### Example Code

#### JavaScript
```javascript
const apiKey = "YOUR_API_MARKET_KEY";
const baseUrl = "https://prod.api.market/api/v1/magicapi/chatbot-language-model-meta-meta-llama-3-8b-instruct";

async function createAndWaitForPrediction(input) {
  const response = await fetch(`${baseUrl}/predictions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-api-market-key": apiKey
    },
    body: JSON.stringify({ version: "meta/meta-llama-3-8b-instruct", input })
  });

  if (response.status !== 201) {
    throw new Error(`API error: ${response.status} ${await response.text()}`);
  }

  const prediction = await response.json();
  return pollForResults(prediction.id);
}

const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function pollForResults(predictionId) {
  const maxTimeout = 600000;
  const pollInterval = 1000;
  const startTime = Date.now();

  while (Date.now() - startTime < maxTimeout) {
    const response = await fetch(`${baseUrl}/predictions/${predictionId}`, {
      headers: { "x-api-market-key": apiKey }
    });

    if (!response.ok) {
      throw new Error(`API error: ${response.status} ${await response.text()}`);
    }

    const prediction = await response.json();

    if (prediction.status === "succeeded") {
      return prediction.output.join("");
    }

    if (prediction.status === "failed" || prediction.status === "canceled") {
      throw new Error(`Prediction ${prediction.status}: ${prediction.error || "Unknown error"}`);
    }

    await delay(pollInterval);
  }

  throw new Error("Prediction timed out after 10 minutes");
}

createAndWaitForPrediction({
    "top_p": 0.95,
    "prompt": "Johnny has 8 billion parameters. His friend Tommy has 70 billion parameters. What does this mean when it comes to speed?",
    "temperature": 0.7,
    "system_prompt": "You are a helpful assistant",
    "length_penalty": 1,
    "max_new_tokens": 512,
    "stop_sequences": "<|end_of_text|>,<|eot_id|>",
    "prompt_template": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
    "presence_penalty": 0
  }).then((output) => console.log(output));
```

#### Python
```python
import time
import requests

API_KEY = "YOUR_API_MARKET_KEY"
BASE_URL = "https://prod.api.market/api/v1/magicapi/chatbot-language-model-meta-meta-llama-3-8b-instruct"

def create_and_wait_for_prediction(input_data):
    headers = {"Content-Type": "application/json", "x-api-market-key": API_KEY}
    response = requests.post(
        f"{BASE_URL}/predictions",
        headers=headers,
        json={"version": "meta/meta-llama-3-8b-instruct", "input": input_data},
    )

    if response.status_code != 201:
        raise Exception(f"API error: {response.status_code} {response.text}")

    return poll_for_results(response.json()["id"])

def poll_for_results(prediction_id):
    max_timeout = 600
    poll_interval = 1
    headers = {"x-api-market-key": API_KEY}
    start_time = time.time()

    while time.time() - start_time < max_timeout:
        response = requests.get(f"{BASE_URL}/predictions/{prediction_id}", headers=headers)

        if response.status_code != 200:
            raise Exception(f"API error: {response.status_code} {response.text}")

        prediction = response.json()

        if prediction["status"] == "succeeded":
            return "".join(prediction["output"])

        if prediction["status"] in ("failed", "canceled"):
            raise Exception(f"Prediction {prediction['status']}: {prediction.get('error') or 'Unknown error'}")

        time.sleep(poll_interval)

    raise Exception("Prediction timed out after 10 minutes")

print(create_and_wait_for_prediction({
    "top_p": 0.95,
    "prompt": "Johnny has 8 billion parameters. His friend Tommy has 70 billion parameters. What does this mean when it comes to speed?",
    "temperature": 0.7,
    "system_prompt": "You are a helpful assistant",
    "length_penalty": 1,
    "max_new_tokens": 512,
    "stop_sequences": "<|end_of_text|>,<|eot_id|>",
    "prompt_template": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
    "presence_penalty": 0
}))
```

## FAQ
- **Should I use `max_tokens` or `max_new_tokens`?**
  - Use `max_tokens`. `max_new_tokens` is the older name, kept for backwards compatibility; do not send both. The same applies to `min_tokens` and `min_new_tokens`.
- **Do I need to change `prompt_template`?**
  - No. The default wraps `system_prompt` and `prompt` in the Llama 3 chat format. A custom template must contain `{prompt}`.
- **How do I get repeatable answers?**
  - Set `seed`, and lower `temperature` (0 is deterministic).
- **Why does `output` come back as a list?**
  - The answer is returned as an array of text chunks. Join them in order.
- **How is a request billed?**
  - By weighted tokens: input tokens count 0.2 and output tokens count 1, at $0.0032 per 10,000 weighted tokens, rounded up to a whole unit per request.
- **Is the 201 response from POST an error?**
  - No. 201 Created means the prediction was accepted. Use the returned `id` to poll GET `/predictions/{id}` until `status` is `succeeded`, `failed` or `canceled`.
- **What if my prediction fails?**
  - Failed predictions are not billed. Read the `error` field, adjust the input and try again.

## Contact Us
If you have any questions, reach out to us through the chat on the bottom right of the page. You can also email our support team at hello@magicapi.com.


Run Meta Llama 3 8B Instruct, an 8 billion parameter model fine-tuned for chat, with system prompts and sampling controls. Pay per token, no monthly fee.