Name: Llama 3.1 8B API: OpenAI Compatible LLM Inference
Brand: arbit-ai-api
SKU: api-arbit-ai-api-llama
Availability: InStock
Author: API Market

lockebrown2

# Llama 3.1 8B API

## Overview

The Llama 3.1 8B API gives you hosted access to Meta's Llama 3.1 8B Instruct model through endpoints that follow the OpenAI API format. It has 3 endpoints. `POST /v1/chat/completions` takes a list of chat messages with roles (`system`, `user`, `assistant`) and returns the model's reply. `POST /v1/completions` takes a plain text prompt and returns the text the model writes after it. `GET /v1/models` lists the model you can call, `llama-3.1-8b-instruct`, which is the value you pass in the `model` field of the other two endpoints. The model runs on vLLM, and the responses name the served weights as `hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4`, a 4-bit AWQ quantized build of Llama 3.1 8B Instruct.

It is built for developers who want a small, fast open source language model without running their own GPU server, and for teams that already use the OpenAI request format and want to point existing code at a Llama model. Chat responses return a `choices` array where each item has a `message` with the `role` and `content` of the reply and a `finish_reason`, and text completions return the generated `text` in each choice. Every response includes a `usage` object with `prompt_tokens`, `completion_tokens` and `total_tokens`, plus the `id`, `created` time and the served `model` name.

## Features

*   **3 endpoints**: Chat completions, text completions and a model list.
*   **OpenAI request format**: Send the same JSON body you would send to an OpenAI style `chat/completions` or `completions` endpoint, with fields such as `model`, `messages`, `prompt`, `max_tokens` and `temperature`.
*   **Llama 3.1 8B Instruct**: An open weights instruction tuned model from Meta, served as a 4-bit AWQ quantized build on vLLM.
*   **System prompts**: Chat requests accept a `system` message to set the tone or rules for the reply.
*   **Token usage in every response**: The `usage` object reports prompt, completion and total tokens for each call.
*   **Fast replies**: In our tests short chat and completion requests returned in about 1.2 to 1.8 seconds.
*   **Rate limit**: 10 requests per second on both the PAYG and Pro plans.

## Use Cases

*   **Chatbots and assistants**: Answer customer or user questions in a chat interface with a system prompt that sets the assistant's behaviour.
*   **Text generation**: Write short descriptions, summaries, replies or rewrites of existing text.
*   **Classification and extraction**: Ask the model to label text, pull out names or fields, or answer yes or no questions about a document.
*   **Drop-in OpenAI replacement**: Point code that already uses the OpenAI request format at an open source model for lower cost tasks.
*   **Prompt prototyping**: Try prompts on a small Llama model before moving them to production.
*   **Text completion**: Continue a sentence or template with the plain `completions` endpoint.

## API Documentation

### Base URL

All requests go through the API.market gateway at this base URL:

```
https://prod.api.market/api/v1/arbit-ai-api/llama
```

### Authentication

Every request needs your API.market key in the `x-api-market-key` header:

```
x-api-market-key: YOUR_API_KEY
```

Replace `YOUR_API_KEY` with your actual API key.

### Endpoints

#### 1. Chat completions

Generates the model's reply to a conversation. Send the conversation as a `messages` array, oldest message first, and the reply comes back in `choices[0].message.content`. Send the body as JSON with the `Content-Type: application/json` header; a body that is not JSON returns `400` with `{"detail":"Body must be JSON"}`.

**Method:** `POST`

**Endpoint:** `/v1/chat/completions`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `model` | body | The model to use. Use `llama-3.1-8b-instruct`. | string | Yes | - | - |
| `messages` | body | The conversation, as a list of objects with `role` (`system`, `user` or `assistant`) and `content`. | array | Yes | - | - |
| `max_tokens` | body | The most tokens the model may write in its reply. | integer | No | - | - |
| `temperature` | body | Sampling temperature. Lower values give more focused, repeatable answers, higher values more varied ones. | number | No | - | - |

**Example Request:**

```bash
curl -X POST "https://prod.api.market/api/v1/arbit-ai-api/llama/v1/chat/completions" \
  -H "x-api-market-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b-instruct",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Give two uses of a hash map, as a short list."}
    ],
    "max_tokens": 80,
    "temperature": 0.2
  }'
```

**Example Response (200 OK):**

```json
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null,
      "message": {
        "annotations": null,
        "audio": null,
        "content": "Here are two uses of a hash map:\n\n1. **Caching**: Hash maps can be used to store frequently accessed data, allowing for quick lookups and reducing the need for database queries or complex computations.\n2. **Data aggregation**: Hash maps can be used to group and aggregate data based on a common key, making it easier to analyze and process large datasets.",
        "function_call": null,
        "reasoning": null,
        "refusal": null,
        "role": "assistant"
      },
      "routed_experts": null,
      "stop_reason": null,
      "token_ids": null
    }
  ],
  "created": 1792564816,
  "ec_transfer_params": null,
  "id": "chatcmpl-993e97d2ed460b6a",
  "kv_transfer_params": null,
  "metrics": null,
  "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
  "object": "chat.completion",
  "prompt_logprobs": null,
  "prompt_text": null,
  "prompt_token_ids": null,
  "service_tier": null,
  "system_fingerprint": "vllm-0.31.0-d7200917",
  "usage": {
    "completion_tokens": 75,
    "completion_tokens_details": null,
    "prompt_tokens": 54,
    "prompt_tokens_details": null,
    "total_tokens": 129
  }
}
```

#### 2. Text completions

Continues a plain text prompt. The generated text comes back in `choices[0].text`. A `finish_reason` of `length` means the model stopped because it reached `max_tokens`, and `stop` means it finished on its own. Send the body as JSON with the `Content-Type: application/json` header.

**Method:** `POST`

**Endpoint:** `/v1/completions`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| `model` | body | The model to use. Use `llama-3.1-8b-instruct`. | string | Yes | - | - |
| `prompt` | body | The text the model should continue. | string | Yes | - | - |
| `max_tokens` | body | The most tokens the model may write. | integer | No | - | - |
| `temperature` | body | Sampling temperature. Lower values give more focused output, higher values more varied output. | number | No | - | - |

**Example Request:**

```bash
curl -X POST "https://prod.api.market/api/v1/arbit-ai-api/llama/v1/completions" \
  -H "x-api-market-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b-instruct",
    "prompt": "The capital of France is",
    "max_tokens": 10
  }'
```

**Example Response (200 OK):**

```json
{
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "logprobs": null,
      "prompt_logprobs": null,
      "prompt_token_ids": null,
      "routed_experts": null,
      "stop_reason": null,
      "text": " a city of grandeur and beauty. From the",
      "token_ids": null
    }
  ],
  "created": 1792564819,
  "ec_transfer_params": null,
  "id": "cmpl-9c65a6fa5981ae52",
  "kv_transfer_params": null,
  "model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
  "object": "text_completion",
  "service_tier": null,
  "system_fingerprint": "vllm-0.31.0-d7200917",
  "usage": {
    "completion_tokens": 10,
    "completion_tokens_details": null,
    "prompt_tokens": 6,
    "prompt_tokens_details": null,
    "total_tokens": 16
  }
}
```

#### 3. List models

Returns the models you can call. Use the `id` from this list as the `model` value in the chat and text completion requests.

**Method:** `GET`

**Endpoint:** `/v1/models`

**Parameters:**

| Name | In | Description | Type | Required | Default | Max Value |
|---|---|---|---|---|---|---|
| - | - | This endpoint takes no parameters. | - | - | - | - |

**Example Request:**

```bash
curl -X GET "https://prod.api.market/api/v1/arbit-ai-api/llama/v1/models" \
  -H "x-api-market-key: YOUR_API_KEY"
```

**Example Response (200 OK):**

```json
{
  "object": "list",
  "data": [
    {
      "id": "llama-3.1-8b-instruct",
      "object": "model",
      "created": 0,
      "owned_by": "self"
    }
  ]
}
```

## Status Codes

| Code | Status | Description |
|---|---|---|
| 200 | OK | The request worked. Chat replies are in `choices[0].message.content` and text completions in `choices[0].text`. |
| 400 | Bad Request | The request body could not be used, for example it is not valid JSON (`{"detail":"Body must be JSON"}`) or a required field such as `messages` or `prompt` is missing. |
| 401 | Unauthorized | The `x-api-market-key` value is invalid or expired. |
| 403 | Forbidden | Your subscription does not include this endpoint. |
| 404 | Not Found | The path does not exist. Check that you are calling `/v1/chat/completions`, `/v1/completions` or `/v1/models` under the base URL. |
| 429 | Too Many Requests | You went over your plan's limit of 10 requests per second, or your plan's quota. |
| 500 | Internal Server Error | The model server failed to answer. Retry after a short delay. |

## Rate Limits

Both the PAYG and Pro plans allow 10 requests per second. On PAYG you pay for each request by the size of its input text: every started 1,000 characters of your `prompt` or message `content` counts as one unit, with a minimum of one unit per request. The Pro plan includes 10,000 calls a month, and calls after that are billed per call. Requests above 10 per second return `429 Too Many Requests`. Back off before retrying and honour any `Retry-After` header. Longer replies take longer to generate, so set your client timeout with your `max_tokens` value in mind.

## FAQ

**Which model name should I send?**

Send `"model": "llama-3.1-8b-instruct"`. This is the only model returned by `GET /v1/models`. The response's `model` field shows the served weights, `hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4`.

**Is this the full precision Llama 3.1 8B model?**

No. The served model is a 4-bit AWQ quantized build of Llama 3.1 8B Instruct. Quantized models use less memory and answer faster, and their output can differ slightly from the full precision model.

**Can I use my existing OpenAI client code?**

The endpoints accept OpenAI style request bodies and return OpenAI style responses. Set the base URL to `https://prod.api.market/api/v1/arbit-ai-api/llama/v1` and send your API.market key in the `x-api-market-key` header instead of an `Authorization` header.

**What is the difference between chat completions and text completions?**

Chat completions take a list of role tagged messages and are best for assistants and instructions. Text completions take a single prompt and simply continue it, which suits templates and sentence completion.

**How do I control the length of the answer?**

Set `max_tokens`. When the reply is cut off at that limit, `finish_reason` is `length`. When the model finishes on its own, it is `stop`.

**How is usage billed on the PAYG plan?**

On PAYG each request is billed by the size of its input text, one unit for every started 1,000 characters of the `prompt` or message `content` you send. A short prompt counts as one unit, and a 2,500 character conversation counts as three. The `usage` object in the response reports tokens for your own tracking.

## Contact Us

If you have any questions or need assistance, you can reach out to us through the embedded chat on the bottom right corner of the page or send an email to hello@magicapi.com.


Llama 3.1 8B API with OpenAI compatible chat completions and text completions. Hosted Llama 3.1 8B Instruct inference, pay as you go or monthly.

Llama 3.1 8B API: OpenAI Compatible LLM Inference