llama-server. Point any OpenAI SDK or compatible client at your deployment’s base URL with /v1 appended.
Endpoints
All paths are relative to your deployment base URL:
Request
Body
Body Attributes
The model to use. A deployment serves a single model, so this can be
"local" or the model id you deployed — either way the request is routed to your deployment’s model.Array of message objects, each with a
role (system, user, or assistant) and content. Used by /v1/chat/completions.Sampling temperature. Lower values (
0.1–0.3) produce focused, deterministic output; higher values increase creativity. Range: 0.0–2.0.Nucleus sampling — only consider tokens with cumulative probability above this threshold. Range:
0.0–1.0. Use either temperature or top_p.Penalty applied to repeated tokens. Values
> 1 discourage repetition (a llama.cpp sampling option).One or more sequences where generation stops. The stop sequence is not included in the output.
When
true, responses are streamed as Server-Sent Events (text/event-stream). See Streaming.Response
Response Fields
The generated choices. Each item contains an
index, a message (with role and content), and a finish_reason.Streaming
Set"stream": true to receive Server-Sent Events (SSE) as tokens are generated:
chat.completion.chunk object, terminated by data: [DONE]:
SDK Examples
This endpoint is a drop-in replacement for the OpenAI API. Just change thebase_url and api_key:
Other OpenAI routes
/v1/embeddings is only available when the deployed model supports embeddings. If it does not, the endpoint returns an error from llama-server.
