Picking the right open model for your workload
There are eight open models worth considering in 2026. Picking the right one for your workload is mostly a function of three constraints: hardware, latency, and domain. Here's the decision matrix we use on engagements.
The open-model landscape stabilized around eight model families that are worth real engagement effort: Llama-3.x, Qwen 2.5, DeepSeek V3 / R1, Mistral Large, Phi-4, Gemma 3, Command R+, and Mixtral. Everything else is either too small to be useful, too rough to ship, or a fine-tune of one of these.
The question isn't "which model is best." It's "which model fits your hardware, your latency budget, and your domain." Here's the matrix we use to triage on the first call of an engagement.
The three constraints
Hardware. What GPU(s) does the deployment target have? This sets a ceiling on parameter count and minimum quant level. A 4090 maxes out at 13B at Q4 or 32B at Q3. An A100 80GB handles 70B comfortably.
Latency. What's the acceptable TTFT and tokens-per-second? Real-time chat needs less than 500ms TTFT and more than 30 tokens/sec. Batch processing tolerates 5 seconds and 5 tokens/sec.
Domain. What's the task? Coding, general chat, function-calling, long-context summarization, multi-lingual, math, vision? Different model families excel at different things.
The matrix below cuts on these three.
The eight families, briefly
Llama-3.3 (8B, 70B) — Meta's flagship. Best general-capability open model in the 70B class. Good function calling. Excellent long-context (128K). Liberal license (Llama Community License). Default pick if no other constraint says otherwise.
Qwen 2.5 (0.5B, 1.5B, 7B, 14B, 32B, 72B) — Alibaba's family. The Coder variants (Qwen 2.5 Coder 7B/14B/32B) are the best open coding models we've benchmarked. Strong multilingual. Apache 2.0 license. Default for code-heavy workloads.
DeepSeek V3 / R1 — V3 is a 671B MoE with 37B active parameters; very strong general capability. R1 adds reasoning chains and beats GPT-4o-mini on math + code. Needs serious GPU (multi-H100). MIT-licensed. Pick when you need reasoning depth and have budget.
Mistral Large (123B) — Strong general-purpose, very good at French + European languages, decent function calling. Mistral AI Research License (commercial use needs separate agreement) — read carefully.
Phi-4 (14B) — Microsoft's compact model. Punches above its weight on reasoning and math. MIT-licensed. Best 14B-class option for math-heavy domains.
Gemma 3 (1B, 4B, 12B, 27B) — Google's family. The 27B variant is competitive with 70B-class models on some benchmarks. Wide vision support. Permissive license.
Command R+ (104B) — Cohere's family, very strong at RAG and tool use. Non-commercial license — commercial requires Cohere agreement. Pick when RAG quality matters and licensing isn't a blocker.
Mixtral 8x22B (141B total, 39B active) — Mistral's earlier MoE. Still excellent at general capability per active-param. Apache 2.0. Good throughput for the quality.
Decision matrix
| If your constraint is... | Pick |
|---|---|
| 4090 (24GB), real-time chat, general | Qwen 2.5 7B Instruct Q4_K_M |
| 4090, real-time, code-heavy | Qwen 2.5 Coder 14B Q4_K_M |
| 4090, batch reasoning | Phi-4 14B Q4_K_M (or 32B if you can wait) |
| A6000 / A100 40GB, real-time chat | Qwen 2.5 32B Instruct Q5_K_M, OR Llama-3.3 8B for very high concurrency |
| A100 80GB / H100, real-time | Llama-3.3 70B Instruct Q4_K_M |
| H100, reasoning quality matters | DeepSeek R1 (multi-H100 setup) |
| CPU only (no GPU) | Qwen 2.5 7B Instruct Q4_K_M or Llama-3.3 8B Q4_K_M |
| Mac M-series (32GB+) | Qwen 2.5 14B or Llama-3.3 8B in MLX or llama.cpp |
| Long context (>32K) | Llama-3.3 70B (128K native) or Qwen 2.5 (128K) |
| Multilingual (non-English-dominant) | Qwen 2.5 (strong CJK), Mistral Large (strong EU), DeepSeek (broad) |
| Function calling / agentic | Llama-3.3 70B or Qwen 2.5 32B (Coder variant for code calls) |
| Vision-language | Qwen 2.5-VL, Llama-3.2-Vision, Gemma 3 (4B/12B/27B) |
| Tight license (Apache only) | Qwen 2.5 or Mixtral |
| Compact for edge | Qwen 2.5 0.5B/1.5B/3B, Gemma 3 1B/4B, Phi-4-mini |
Domain-specific notes
Coding. Qwen 2.5 Coder, in order of preference: 32B > 14B > 7B. The 32B is genuinely competitive with Claude on most coding benchmarks. Deepseek-Coder-V3 is the absolute top if you have the GPU.
Math / reasoning. DeepSeek R1 > Phi-4 > Qwen 2.5. R1 is on a different tier for math-heavy workloads but needs multi-GPU.
Function calling / tool use. Llama-3.3 70B is solid. Qwen 2.5 32B is close. Function-calling is highly sensitive to fine-tuning details, so a domain-specific LoRA on top of either is the right move for production.
RAG over long docs. Command R+ if license allows; Llama-3.3 70B with 128K context otherwise. Both handle multi-document QA cleanly.
Customer support / chat. Llama-3.3 8B Instruct is the workhorse. Fast, good general capability, fine-tunes cleanly with LoRA. Most of our customer-support engagements end up here.
Multilingual non-English. Qwen 2.5 for Chinese, Japanese, Korean. Mistral Large for European languages. Llama-3.3 is acceptable for major languages but not class-leading.
What we don't recommend
Models below 3B for production. They sometimes work for narrow tasks (text classification, intent detection), but quality drops sharply and the inference savings rarely justify the regression.
Older models (Llama-2, Mistral-7B-v0.1). The newer generation is meaningfully better at the same parameter count. There's no reason to ship anything older than ~6 months for new production deployments.
Quants below Q3 without imatrix calibration. Q2 quants are noticeably degraded. Custom imatrix recovers some of the loss but the floor is still rough. We ship Q4 or higher on production unless the client is genuinely VRAM-constrained.
Models without a clear license. "Free for research" doesn't help if your engagement is producing a commercial deployment. Always confirm license terms before recommending.
The shortcut
If you're early in an engagement and haven't measured anything yet, the safe default for most general-purpose chat workloads is:
- Single-tenant / internal tool: Llama-3.3 8B Instruct Q4_K_M on whatever GPU is available.
- Multi-tenant / customer-facing: Llama-3.3 70B Instruct Q4_K_M on an A100 80GB or H100 with vLLM.
You can always swap to something more specific (Qwen Coder for code, DeepSeek R1 for reasoning) after you've measured what the workload actually wants. Starting with a known-good baseline saves time vs. agonizing over model selection on day one.