Frontier Architecture Decoding Notes (7): Qwen3.8 Dual Checkpoint — Dense 27B vs Sparse Flash-Next (360GB weights only to buy 6B activation)

0. Conclusion First

These two checkpoints are not size variants of the same model — they are two weight sets with clearly different architectures and weight-organization schemes:

• Qwen3.8-Flash-Next = Qwen4ExpForConditionalGeneration sparse MoE. Main model is 125B by card wording, ~6B active per token; plus ~51B N-gram embedding and MTP weights. Repo has 131 BF16 safetensors shards, ~360 GB (335.276 GiB).
• Qwen3.8-27B = Qwen3_5ForConditionalGeneration dense model. Repo has 18 BF16 safetensors shards, ~55.6 GB (51.747 GiB).

Flash-Next’s weight files are 6.48× those of 27B, yet it does not compute all weights per token — most of the storage comes from 512 experts and N-gram lookup tables. “Flash” mainly denotes a compute/access-efficiency design, not a smaller download. Both share many tokenizer / vision-preprocess / generation configs, but config / index / weights all differ — shards are not interchangeable.

Model-name correction update (2026-08-28): fa6 (Gated DeltaNet) in this series, based on a then-current snapshot, judged that “qwen3.8-27B does not exist; Qwen3.8 is MoE like 2.4T-A95B.” A direct repo scan on 2026-08-28 confirms: Qwen3.8-27B exists as a real dense checkpoint (55.6GB BF16, Qwen3_5ForConditionalGeneration). fa6’s judgment held for its snapshot; this repo now exposes the 27B dense checkpoint, so we correct it here. Repo revision short IDs: Flash-Next 2741eec1, 27B 1098534a.

1. Architecture Overview: Same Vocab/Context, Opposite Skeletons

Both share a 248,320 vocab (padded) and 262,144 native context, but build capacity in opposite ways: Flash-Next uses a narrow backbone + experts / N-gram lookup, while 27B uses a wide backbone + per-layer dense FFN.

2. Precision and Capacity Basis

Both repos are BF16, not FP8. config declares text_config.dtype = bfloat16 (mamba_ssm_dtype: float32 is SSM-compute only, not FP32 weights). Header scan confirms: Flash-Next has 1,655 BF16 tensors + 3 I64 metadata; 27B has 1,199 BF16 tensors. Neither has quantization_config. Separate quantized repos: Flash-Next-FP8 ≈ 185.5 GB, 27B-FP8 ≈ 30.9 GB.

The capacity gap comes almost entirely from the weights themselves (non-weight files are both ~23 MB, nearly identical).

3. Architecture Differences (from config.json)

4. Weight Organization in the Index

Flash-Next packs experts into large tensors: layers.0.mlp.experts.gate_up_proj shape [512,1280,2560], first dim = 512 experts, then distributed across 131 shards. Each token routes only 10 routed + 1 shared, yet the checkpoint must store all 512 experts — compute approaches the few active experts, but disk / reachable storage approaches all of them.

The N-gram table is an extra capacity axis: the index holds 128 ple_embedding.ngram_embedding.shard_*.weight, shape [2,500,012, 160], totaling 51.2B params / 102.4 GB. The official note stresses these are fetched mainly via local n-gram lookup, not the per-token regular matmul budget; one design goal is easier Host-Memory placement with async prefetch. But you cannot skip these shards on download, and whether offload is efficient still depends on the inference framework.

27B uses plain dense FFN: gate_proj/up_proj/down_proj shapes [17408,5120]/[5120,17408], one set per layer; capacity from 64 layers + 17,408 mid-dim, no hundreds of expert copies.

5. Weight Storage Breakdown (by tensor name)

6. Which Files Are Shared / Not Interchangeable

Identical SHA256 (reusable): chat_template.jinja, configuration.json, generation_config.json, merges.txt, preprocessor_config.json, tokenizer.json, tokenizer_config.json, video_preprocessor_config.json, vocab.json — tokenizer, vocab, image/video preprocess, and generation config are highly consistent.

Must be treated as model-specific (not interchangeable): config.json, model.safetensors.index.json, all model-*.safetensors, README, LICENSE, .gitattributes. Both repos use the generic model-00001-of-... naming, but shard counts, index mappings, and internal tensor names differ completely — do not pair one repo’s shard #1 with another repo’s index. Note: identical vision-preprocess files ≠ identical vision weights (LM hidden dim 2,560 vs 5,120, different connector).

7. Implications for Inference Deployment

Static weights only (excluding runtime / activation / KV cache / framework overhead / fragmentation): 27B ≈ 51.75 GiB (near single-card / few-card); Flash-Next ≈ 335.28 GiB (usually needs multi-card, multi-node, quantization, or Host-Memory offload). Actual device memory cannot equal “6B activation” — expert weights and N-gram tables must still be reachable somewhere.

• Shard count (131/18) ≠ GPU count / TP degree: runtime partitioning is decided by the inference framework.
• Framework compatibility: the workspace vllm-v0.21.0 has qwen3_5 / qwen3_5_mtp paths (close to 27B), but no qwen4_exp / Flash-Next implementation was found — Flash-Next must be tested against the model card’s latest vLLM/SGLang recipe; do not assume direct loading.
• Long context: both native 262,144 tokens, YaRN-extensible to 1M; YaRN changes position encoding only, not static weight size, but greatly increases KV cache and runtime memory.

8. Selection Guidance

Next: how Flash-Next’s 512 experts + N-gram lookup are partitioned and offloaded in inference frameworks (EP / Host-Memory prefetch / quantization), and how it fundamentally differs from the “attention-modification” routes of DeepSeek V4 and MiniMax M3.