The assistant's language model runs on the studio's own GPU server on its local network, not on an outside AI service. The model is Qwen3-30B-A3B-Instruct-2507, served by llama.cpp across two Intel Arc graphics cards, behind a LiteLLM gateway. No question, page or answer leaves the studio's hardware. What the assistant does with the model is in The assistant.
The model and the hardware
- Model
Qwen3-30B-A3B-Instruct-2507: a mixture-of-experts model with about 30 billion parameters, of which about 3 billion work on each token, which is why it is fast for its size.
- Quantisation
UD-Q4_K_XL: weights stored at about 4 bits, so the whole model fits in the two cards' memory.
- GPUs
An Intel Arc B580 and an Intel Arc Pro B50, with the model split across both.
- Server
llama.cpp with its Vulkan back end, on the studio's own machine on the local network.
- Parallel requests
4, each with its own 16,384-token context.
- Embeddings
bge-m3, a second model on the same gateway, which turns a text into 1,024 numbers that place it by meaning; texts that mean the same thing land close together.
- Gateway
LiteLLM, which gives the assistant app an OpenAI-compatible address and its own key.
The path from the panel to the GPU
The assistant app calls POST {AI_GATEWAY_URL}/chat/completions with its gateway key, the model alias blockout-chat, streaming on, and thinking switched off, and POST {AI_GATEWAY_URL}/embeddings with blockout-embed for the handbook search. The gateway's address and key are held by the assistant app alone; the cluster's nodes are the only machines the server's firewall lets through to it.
How fast it is
- Writingtokens a second
About 33, measured. A token is a piece of a word, so this is roughly a short paragraph every few seconds.
- Reading a prompttokens a second
About 520, measured, for prompt text the server has not seen before.
- Cached prompt
The fixed part of the assistant's prompt is the same for every question, so the server keeps it cached and a question pays only for its page, history and results.
A typical answer reads about 3,500 tokens of new results (about 7 seconds at 520 a second), writes a short plan, then writes the words, at most 450 tokens (under 15 seconds at 33 a second). Every answer is bounded at 90 seconds.
How many questions it can answer at once
The server takes four requests at once. The assistant app takes one answer per person and three across the whole assistant, leaving the fourth for the handbook's embeddings. A fourth question while three are running is refused as busy, with a time to try again, rather than queued. The gateway allows the assistant's key 600 requests a minute, so the four GPU slots are the real limit. Each answer makes two model calls in turn, the plan and then the words, which share their first messages so the server reuses its cache between them.
Embedding the handbook
When the assistant app starts it embeds every section of this handbook it has not embedded before, in the background, 32 at a time: about 1,300 sections take a little over a minute. Each vector is kept in Redis against a fingerprint of the section's text, so a restart embeds nothing again and only a section that changed is embedded anew. Until they are ready, and whenever a question's embedding takes longer than 2.5 seconds, the search uses words alone; nobody waits for it.
Why the model is never trusted with free choices
Even a good model can write a page that does not exist or a number it did not read. So the plan is forced into JSON by a schema the server turns into a grammar (the reply can only be valid JSON of that shape), the pages and page parts it may name are fixed lists, the figures an answer quotes are computed by the assistant app from the API's answers, and every sentence of the words is checked as it streams. See How it keeps to the facts.
Privacy: no question leaves the studio
- Where it runs
On the studio's own server on its local network. No outside AI service is called, and nothing is sent to a model provider.
- What is logged
One line per answer: how long it took, which reads ran and token counts. Never the question, the plan, the answer or what was read.
- What is kept
Nothing. The panel keeps a conversation only in the open tab's memory.
When the model cannot be reached
If the gateway or the server is down, the API's status check says the assistant is not connected, the chat says so, and any chart it shows is labelled a sample. Questions are not queued for later. The rest of the panel is unaffected: nothing else depends on the model.
Configuration for developers
The assistant app reads AI_GATEWAY_URL (the gateway's base address, ending in /v1),
AI_GATEWAY_API_KEY (its virtual key on the gateway, limited to the blockout-chat and
blockout-embed models), AI_MODEL (blockout-chat by default) and AI_EMBED_MODEL
(blockout-embed by default). Without the URL or the key it reports itself not ready and the
panel shows it as not connected. Tests never call a model: they use the app's scripted model
(AI_PROVIDER=fake, refused outside tests). An evaluation of 46 questions against the real model
runs on demand from a machine the gateway admits (apps/ai/test/eval/eval.ts).
Related: The assistant, Architecture, Security and privacy.
Updated