Inference chat

What it is

A standard chat UI that streams responses token-by-token from the selected backend. The backend dropdown (top-right of the tab) picks between:

• Local — the in-app supervised llama-server
• Connected hosts — any remote host you're SSH-connected to via the Hosts tab
• Registered remote servers — any OpenAI-compatible endpoint registered via the Remote servers card

The tab also exposes the sampling parameters most teams want to control: temperature, top-p, top-k, max tokens, frequency penalty, presence penalty, repetition penalty, stop sequences, seed.

When you'd use it

• Spot-check the model after a fresh serve, calibration, or fine-tune
• Tune sampling for your workload before locking it into your application config
• Compare backends by sending the same prompt to local + remote and reading the responses side-by-side
• Debug a hung production endpoint by sending a known-good prompt and watching whether tokens flow at all

It's NOT a production chat UI. There's no user authentication, session history, or moderation; sessions are in-memory and lost when you close the app.

Sampling controls — what they do

• Temperature (0-2): how random sampling is. 0 = always pick the most likely token (deterministic). 0.7 = balanced creativity. >1.0 = wild.
• Top-p (0-1): only sample from tokens whose cumulative probability mass is at most this. 0.9 is a common default; effectively caps the long tail.
• Top-k: only sample from the k most likely tokens. Set to 40 for a tighter constraint; 0 disables.
• Frequency penalty (-2 to 2): discourage repeating the same tokens. Higher = more variety.
• Presence penalty (-2 to 2): discourage repeating ANY token that's appeared. Use sparingly; can push the model to ramble.
• Repetition penalty (1.0-1.3): llama.cpp-specific. Multiplies the logits of recently-seen tokens. 1.1 is a safe default.
• Stop sequences: strings that, when generated, cut off the response immediately. Comma-separated.
• Seed: integer. Same seed + same params + same model = exact reproducibility.

System prompt

The system-prompt field at the top of the chat is prepended to every message in the conversation. Use it to set the model's role ("You are a helpful coding assistant..."), format constraints ("Respond in JSON"), or behavior rules.

A long system prompt is exactly the case the prompt cache was built for — turn that on if you're sending many requests with the same system prompt.

Common pitfalls

• Sampling drifts during the conversation: each message uses the current sampling settings, but past messages were generated with whatever was set then. To reproduce a result, clear the chat and re-run with the same settings.
• Comparing across backends, getting different outputs: even at temperature 0, different inference engines tokenize and sample slightly differently. Use seed + temperature 0 for the closest apples-to-apples; expect minor variance.
• Forgetting the seed: if you're debugging a "the model said something weird," set seed to a specific value so the next person to repro can hit the same path.