← All articles · Dashboard
Python runtime
The optional Python sidecar Blueprint uses for the heavy bits — LoRA training, LLMLingua compression, vLLM, TensorRT-LLM. Opt-in feature by feature.
3 min read
What it is
llama.cpp does inference. But other parts of the LLM stack — fine-tuning a LoRA, compressing prompts with LLMLingua, running vLLM or TensorRT-LLM — are Python-side ecosystems. Bundling all of PyTorch + transformers + every accelerator wheel into the desktop installer would push the download past 2 GB even for users who don't need them.
Solution: an opt-in Python sidecar. Blueprint ships uv (~5 MB,
astral.sh's Rust-based Python toolchain) and on first use of a
Python-side feature, runs uv python install + uv pip install <packages> into a venv under ~/.blueprint/python/.
The user picks which features to install from the Python runtime card on the Dashboard. Nothing happens until they opt in.
The feature catalog
| Feature | Marginal size | Needs GPU? | Provides |
|---|---|---|---|
| Python core | 230 MB | no | uv-managed Python 3.11 + minimal libs (fastapi, uvicorn, requests) |
| LLMLingua | 250 MB | no | prompt compression sidecar |
| PyTorch + CUDA 12.x | 2,500 MB | yes (NVIDIA) | the GPU compute stack everything below depends on |
| LoRA training | 800 MB | yes | transformers, peft, accelerate, bitsandbytes, trl, datasets |
| vLLM engine | 1,200 MB | yes | PagedAttention-based serving |
| TensorRT-LLM engine | 2,000 MB | yes | NVIDIA's compiled-engine path |
Sizes are MARGINAL — what each feature adds on top of what's already installed. Pick "LoRA training" and the card automatically queues "Python core" + "PyTorch + CUDA 12.x" as dependencies.
How to install
- Dashboard → Python runtime card.
- Pick the feature. The card shows the total disk impact (the feature itself + any dependencies it pulls in).
- Click Install. Blueprint downloads uv if not yet present,
runs
uv python install, thenuv pip install <packages>in the managed venv. Logs stream into the install panel. - The card flips to Installed when done. The corresponding feature card elsewhere on the Dashboard (TrainCard, Compress card, the vLLM engine option) becomes active.
Total install time depends on download speed and CPU — PyTorch + CUDA is the slowest step (~5-15 min on most boxes).
How to uninstall
Same card → Uninstall. Removes the packages from the venv but
keeps the feature available to reinstall later. To wipe the entire
Python install, the Reset Blueprint data action in the Maintain tab
nukes ~/.blueprint/python/ along with everything else.
Common pitfalls
- Trying to use vLLM / TensorRT-LLM without a CUDA GPU: these features require an NVIDIA card with a working CUDA driver. Apple Silicon and AMD aren't supported by the upstream Python packages yet. The card warns; the install still proceeds; runtime fails later.
- uv's first-time Python download failing: uv pulls Python from
astral's CDN. Corporate proxies sometimes block. Workaround: set
https_proxyenv vars before launching Blueprint. - Disk full halfway through: PyTorch + CUDA is 2.5 GB. If
~/.blueprint/is on a small partition, the install fails partway with no clean rollback. Free space before installing.
Need help with this in production?
Inspire AI Lab runs LLM optimization engagements end-to-end using Blueprint. If you'd rather hand the work to us instead of running it yourself, book a 30-minute review.