A practical guide to LLM variants on Hugging Face

1. The post-training axis: base → instruct → chat → reasoning

Every modern LLM goes through stages. Each stage produces a checkpoint that may be released separately.

Base (a.k.a. pretrained, foundation)

The raw output of pretraining on trillions of tokens of text. It only knows how to continue text — give it "The capital of France is" and it completes "Paris." Give it "Write me a poem" and it might continue with "...about my cat, said the user, and then the assistant replied..." because it has no notion of being an assistant.

Internally: a transformer trained with next-token prediction (cross-entropy loss) on a massive corpus. No notion of conversation, no refusals, no instruction-following.

When to pick it: fine-tuning your own model, research, text completion tasks where you don't want assistant behavior, FIM for code completion if the base was trained with FIM objectives.

Repo signals: gemma-3-27b, Llama-3.1-8B, Qwen2.5-7B (no -Instruct, no -Chat).

Instruct / SFT (supervised fine-tuned)

Base model + further training on (instruction, response) pairs. Now it follows instructions. This is done via SFT: same next-token loss, but only on curated instruction-following data, often with a chat template (special tokens like <|user|>, <|assistant|>).

The model learns: "when I see this template structure, I should produce a helpful response in the assistant role."

When to pick it: the default for almost everything. Coding, writing, Q&A, agents.

Repo signals: -Instruct, -it (Gemma's convention), -SFT.

Chat / RLHF / DPO

Instruct model + a preference-tuning stage. Humans (or another LLM) rank pairs of responses; the model is trained to prefer the ranked-better ones. This makes outputs more helpful, more honest, less toxic, and aligned to a "house style."

Three flavors of how this is done — you'll see these terms in model cards:

• RLHF (PPO) — train a reward model from preferences, then RL the LLM against it. Powerful, finicky, expensive. What ChatGPT was originally trained with.
• DPO — Direct Preference Optimization. Skip the reward model; optimize directly on preference pairs with a clever loss. Cheaper, more stable, very common in open weights.
• GRPO / RLVR — used heavily in reasoning models (next section). RL against a verifiable reward (did the math answer match? did the code pass tests?) instead of a learned reward model.

When to pick it: when available, you almost always want this over a pure SFT/instruct model. The differences are real for chat quality, but not always for raw task accuracy.

Repo signals: -Chat, -DPO, -RLHF, often just rolled into -Instruct without a separate label.

Reasoning models

A reasoning model is one that's been specifically trained to spend more compute at inference time by producing long internal chains of thought before answering. The training recipe:

1. Start from a strong instruct model.
2. Generate lots of problems with verifiable answers (math, code, logic puzzles).
3. Let the model produce long CoT attempts; reward only the ones that get the right answer (RLVR — RL with Verifiable Rewards, often via GRPO).
4. The model learns to "think out loud" — backtrack, self-correct, plan — because that behavior empirically leads to correct answers.

What you see at inference: the model emits a <think>...</think> block (or similar) with potentially thousands of tokens of reasoning, then a final answer. This is test-time compute — trading latency and tokens for accuracy.

When to pick reasoning:

• Math, hard code, logic, planning, multi-step analysis
• You can afford the latency and token cost (it can 10–50× output length)

When NOT to:

• Simple instruction following, summarization, chat, RAG over short docs
• Latency-sensitive UX (autocomplete, voice)
• Tight token budgets

Repo signals: R1, -Thinking, -Reasoning, o1-style, QwQ. Some models are hybrid (Qwen3, Claude) — same weights, reasoning toggled by a flag or system prompt.

---

2. The architecture axis: dense vs. MoE

Dense

Every parameter participates in every forward pass. A dense 70B model uses 70B params worth of compute per token. Simple, well-understood, predictable.

Mixture of Experts (MoE)

The feedforward layers (the "MLP" blocks between attention layers) are replaced with N experts + a router. For each token, the router picks the top-K experts (typically K=2 out of 8, or K=8 out of 128) and only those run. This is a Mixture of Experts.

Total params = all experts combined → determines RAM/disk.
Active params = what actually runs per token → determines speed and FLOPs.

So 26B-A4B means 26B total, 4B active. You pay 26B in memory, 4B in compute.

Why it works: different experts specialize (loosely — code, language X, math, etc.). The model has the knowledge capacity of a big model with the speed of a small one.

Tradeoffs:

• Big RAM footprint for modest speed → great if you have RAM but a slow GPU/CPU
• Routing imbalance can cause some experts to be underused
• Worse than equivalent-active dense at very small active sizes
• Not all inference engines handle MoE quantization equally well

Examples: Mixtral 8x7B, Mixtral 8x22B, DeepSeek-V3 (671B / 37B active), Qwen3-235B-A22B, Gemma 3n with the elastic MatFormer trick.

Hybrid / SSM / Mamba variants

A small but growing class replaces some attention layers with state-space models (Mamba, Mamba-2) or hybrid attention/SSM stacks (Jamba, Zamba). They scale better with long context (linear instead of quadratic) and can be faster, but ecosystem support (quantization, fine-tuning) lags.

Repo signals: Mamba, Jamba, RWKV, Zamba.

---

3. The precision axis: quantization

The model is trained in bfloat16 or float16 (16 bits per weight). To run it cheaper, weights are compressed to fewer bits per weight. Quantization may also affect activations and the KV cache.

What "n-bit" actually means

A 4-bit quant doesn't store every weight as one of 16 values. It groups weights (e.g., blocks of 32 or 128), stores a scale and possibly a zero-point per group at higher precision, then stores each weight as a 4-bit index into that group's range. Effective bits-per-weight is usually n + overhead — a "4-bit" GGUF is often 4.5–5 bpw real.

The major quantization formats

Format	Made for	Notes
GGUF (llama.cpp)	CPU + any GPU, cross-platform	The dominant format for local inference. Uses k-quants (Q4_K_M, Q5_K_S) and i-quants (IQ3_XS, IQ2_M) which are smarter than naive round-to-nearest. Runs in llama.cpp, Ollama, LM Studio, Jan, etc.
MLX	Apple Silicon	Native Apple framework; fastest path on M-series Macs.
AWQ	NVIDIA GPUs	Activation-aware: protects the weights that matter most based on activation magnitudes. Popular for vLLM serving. 4-bit is the standard.
GPTQ	NVIDIA GPUs	Older than AWQ, uses second-order info (approximate Hessian) to minimize quant error layer by layer. Still common.
EXL2 / EXL3	NVIDIA GPUs	Variable bitrate per layer; aggressive and fast. Power-user format.
bitsandbytes (NF4 / FP4)	NVIDIA GPUs	On-the-fly 4-bit and 8-bit, used heavily for QLoRA fine-tuning. Not the fastest at inference but ubiquitous.
AQLM, QuIP#, HQQ	Research / specialty	Extreme low-bit (2-bit) with clever codebooks. Niche but improving.

How to read GGUF quant suffixes

• Q8_0 — 8-bit, near-lossless. Big.
• Q6_K — 6-bit k-quant. Excellent quality, moderate size. Usually the sweet spot.
• Q5_K_M, Q5_K_S — 5-bit medium / small. Strong quality, smaller.
• Q4_K_M — 4-bit medium. The standard "I want it to fit and still be smart" pick.
• Q4_K_S — 4-bit small. A bit more degraded.
• Q3_K_*, IQ3_* — 3-bit. Visible degradation, usable for very large models.
• IQ2_*, Q2_K — 2-bit. Only worth it for very large models you couldn't otherwise run.

Rule of thumb on quality loss: going from fp16 → 8-bit is essentially free. 8 → 6 is nearly free. 6 → 4 is small but real. 4 → 3 is noticeable. 3 → 2 is large and only worth it on big models. The bigger the model, the better it tolerates aggressive quantization.

What quantization does internally

Each layer's matmul y = Wx becomes "dequantize W on the fly to compute Wx." The dequantization is fused with the matmul kernel so it's cheap. Activations stay in higher precision (often fp16); weights are the compressed thing. KV cache can be separately quantized (often 8-bit or 4-bit) to fit longer context.

---

4. The derivation axis: distills, merges, fine-tunes

Distillation

A smaller "student" model is trained to imitate a larger "teacher." Two flavors of distillation:

• Logit distillation — student matches the teacher's full output distribution (KL divergence). Requires access to the teacher's logits. Stronger but requires the teacher to be runnable.
• Data distillation — generate a big dataset of (prompt, teacher's response) pairs and SFT the student on it. This is what most "distilled" open weights actually are.

The DeepSeek-R1-Distill-* models (Qwen-7B, Llama-70B, etc.) are existing base models SFT'd on R1's reasoning traces. They're not R1 — they're smaller models that picked up some of R1's reasoning style.

Repo signals: -Distill, Distilled-, names like DeepSeek-R1-Distill-Qwen-32B.

Merges

Take two or more existing models with the same architecture and combine their weights — linear interpolation, SLERP, TIES, DARE, model soups. No training. Free if you have a GPU.

Quality is unpredictable: sometimes better than either parent, often worse, occasionally produces a model that benchmarks well but breaks subtly. The community on HF is full of these.

Repo signals: names with multiple model names smashed together, mention of mergekit, slerp, ties, dare.

Fine-tunes / continued pretraining

Someone took a base or instruct model and trained further on:

• A specific domain (medical, legal, code in language X)
• A persona / style (uncensored, roleplay, "dolphin," "nous")
• A new language
• A task (function calling, JSON output)

Quality varies wildly. Reputable fine-tuners: NousResearch, Cognitive Computations (Eric Hartford / "dolphin"), HuggingFaceH4, AllenAI, Teknium.

LoRA / QLoRA adapters

A small (MB-scale) addendum to a base model, not a full model. You load the base + adapter at runtime. A LoRA is much cheaper to ship and combine than full fine-tunes. Many "fine-tunes" on HF are actually adapters published alongside the base model name they require.

---

5. Context length variants

Models are pretrained at some native context length (e.g., 8K). Longer-context variants are produced by:

• RoPE scaling / NTK / YaRN — rescale the rotary position embeddings so the model generalizes to positions it didn't see in training. Cheap, sometimes lossy.
• Continued pretraining at long context — actually train on long sequences. Expensive but better.
• Position interpolation — linear interpolation of position indices.

You'll see 128k, 1M, -long in names. Watch out: long-context numbers on the box are often ambitious. Real performance often degrades well before the advertised limit ("lost in the middle"). Benchmarks like RULER and NIAH are more honest than the marketing number.

---

6. Multimodal variants

• VL / Vision — the model accepts images. A vision encoder (often SigLIP or a ViT) projects images into the LLM's token space. Examples: Qwen2.5-VL, Llama-3.2-Vision, Gemma 3 (natively multimodal).
• Audio / Speech — accepts audio input, sometimes produces speech output. Whisper-style encoders bolted on.
• Omni — text + image + audio + sometimes video, in and out. Qwen2.5-Omni, GPT-4o-style.
• Embedding models — not generative; produce vectors for retrieval. Different repo (-embedding, bge, e5, gte).
• Reranker models — score (query, document) pairs for retrieval.

---

7. Format-specific repos

The same model is republished in many formats by different teams:

• Original — by the model's creator, in safetensors at fp16/bf16.
• GGUF — TheBloke (historical), bartowski, lmstudio-community, mradermacher.
• MLX — mlx-community, lmstudio-community.
• AWQ / GPTQ — TheBloke (historical), Qwen team, neuralmagic.
• EXL2 — turboderp, LoneStriker.

Always check the original model card for capabilities, then pick a quant repo for your runtime.

---

8. Reading a HF repo name end-to-end

lmstudio-community/gemma-4-26B-A4B-it-MLX-6bit

• lmstudio-community/ — quantizer / packager
• gemma-4 — model family + generation
• 26B-A4B — MoE: 26B total, 4B active
• it — instruction-tuned
• MLX — format (Apple Silicon)
• 6bit — quantization level

bartowski/Qwen3-30B-A3B-Thinking-2507-GGUF (hypothetical):

• bartowski/ — quantizer
• Qwen3 — family
• 30B-A3B — MoE
• Thinking — reasoning-enabled
• 2507 — release date / version (Jul 2025)
• GGUF — format

unsloth/DeepSeek-R1-Distill-Qwen-14B-Q4_K_M-GGUF:

• unsloth/ — quantizer
• DeepSeek-R1-Distill — distilled from R1's reasoning traces
• Qwen-14B — student is Qwen 14B
• Q4_K_M — 4-bit medium k-quant
• GGUF — format

---

9. Decision framework

Start with the task, then narrow:

Step 1 — Task class

• Chat, writing, general Q&A, RAG → Instruct/Chat dense ~7–32B at Q4–Q6
• Hard code, math, multi-step reasoning → Reasoning model if you can spare the latency
• Code autocomplete / FIM → Small base or code-specialized model (~1–7B), high quant
• Agents, tool use → Instruct model with explicit tool-use training (Qwen, Llama, Hermes); reasoning helps but blows token budget
• Vision tasks → VL variant
• Embeddings / retrieval → dedicated embedding model, not a chat LLM

Step 2 — Hardware envelope

Estimate RAM needed: total_params × bits_per_weight / 8 plus ~2–8 GB for KV cache and overhead. On an M1 Max 64GB you can comfortably run anything up to ~50 GB on disk; above that, KV cache for long context starts hurting.

Step 3 — Architecture

• Lots of RAM, modest compute (CPU, Apple Silicon, single consumer GPU) → MoE wins
• GPU-rich, RAM-tight → Dense wins
• Need long context cheaply → consider Mamba/hybrid if ecosystem allows

Step 4 — Quant

• ≥30B params → Q4_K_M or 4-bit AWQ is usually fine
• 7–13B → prefer Q5_K_M or Q6_K; the smaller the model, the more you feel quant loss
• <7B → Q6_K or Q8_0 if you have the headroom

Step 5 — Variant

• Prefer the original publisher's instruct model unless you have a specific reason
• Avoid random merges unless someone you trust benchmarks them
• For reasoning, prefer the original (R1, QwQ, Qwen3-Thinking) over distilled versions when you can fit them

---

10. How characteristics combine — concrete examples

Use case	Pick	Why
Local chat assistant on M1 Max 64GB	Qwen3-30B-A3B-Instruct MLX 6bit, or gemma-3-27b-it MLX 6bit	Fast (low active params or moderate dense), high-quality quant, fits with room
Hard math/code at the desk	QwQ-32B or Qwen3-32B-Thinking GGUF Q5_K_M	Reasoning, dense for predictability
Code FIM in Zed	Qwen2.5-Coder-7B base, Q6 or fp16	Small, FIM-trained, low latency
RAG over your own docs	Mid-size instruct (Qwen3-14B or Llama-3.3-8B) + a reranker + an embedding model	Three different models, each specialized
Agentic tool-calling	Qwen3-14B-Instruct or Llama-3.3-70B-Instruct	Strong tool-use training; skip reasoning unless task needs it
Vision Q&A on screenshots	Qwen2.5-VL-7B or Gemma-3-12B-it (natively multimodal)	VL variant required
Squeezing the biggest model possible	DeepSeek-V3 or Qwen3-235B-A22B at IQ3 / Q3_K_S GGUF	MoE + aggressive quant; 235B with 22B active is tractable on a beefy box

---

11. What to ignore

• Leaderboard rank as your only signal — many models are gamed
• "Uncensored" branding as a quality signal — usually orthogonal to capability
• Bigger always = better — a well-tuned 14B beats a sloppy 70B for most things
• New release excitement — wait a week, read the actual benchmarks people run on their own data

---

Glossary

Hover any underlined term in the text above for a quick definition. Click to jump here for the full entry.

---

This document was produced by Claude based on David Morel’s instructions and questions. Treat it as a starting point, not a canonical reference — model landscape moves fast and specific details may be outdated.