4-bit on two cards, tested against 16-bit.

The 16-bit Qwen3.8 Flash Next checkpoint is 360 GB. Two Radeon AI PRO R9700 cards offer 64 GB of VRAM between them, so the original model is far out of reach. Our new 4-bit release runs on those two cards, with results close to the 16-bit reference on math, Python coding and knowledge tasks.¹

The practical result: 189.8 tokens/s for weighted single-request generation, 420.7 tokens/s across eight concurrent requests, and an optional RAM prefix cache for long coding conversations. It is regular vLLM with the Paiton plugin and our own kernels. The benchmarked setup needs a 256 GB RAM host, because less-used experts and large tables live in system memory.²

This follows our 3-bit Flash Next release. Here the priority is retaining more of the original model's quality.

Accuracy against the 16-bit model

We ran both models on the same questions, with the same prompts, per-question sampling seeds and thinking-on chat template. The 4-bit model used the shipped 4bit serving mode; the 16-bit reference used the original checkpoint.¹

Task accuracy; higher is better. GSM8K and HumanEval use their full test sets. MMLU-Pro uses a fixed 1,400-question subset, 100 questions in each of 14 categories. HumanEval is pass@1. Both models have a 16,384-token response limit.

GSM8K differs by 0.1 percentage point; HumanEval is tied. In a second seed set, the 16-bit model itself changed by 0.0 points on GSM8K and 1.2 points on HumanEval. The paired tests for the two quantized comparisons both give p = 1.00.¹

MMLU-Pro is one point lower for the 4-bit model, with a paired 95% interval of −2.3 to +0.3 points and p = 0.16. That does not establish statistical equivalence. More 4-bit replies reached the response limit: 104 versus 79. On the 1,271 questions where neither model reached it, scores are 91.1% for 4-bit and 91.0% for 16-bit. That subset helps describe the gap, but does not prove a longer limit would remove it.¹

Beyond task scores

Task scores can hide differences in the model's token predictions. We also measured perplexity, long-document negative log-likelihood and full-vocabulary KL divergence against BF16, the 16-bit reference.¹

Perplexity increase relative to BF16; lower is better. The two context lengths are evaluated separately. KL is a different measure: the average difference between next-token probability distributions on 170,884 text and assistant positions, with each model free to choose its experts. Zero KL would mean identical distributions.

The unseen text contains 22 documents, 262,144 tokens in total, never used to build the quant. Perplexity uses raw text without a chat template, scoring the second half of each context chunk.¹

The MXFP4 row is our own measurement on the same experts, not another serving engine. The 4-bit served configuration has 41% lower KL and about 8% fewer bits per expert weight, including scales. The trunk and cache also differ, so this does not isolate the weight format's effect or mean 41% better task accuracy.¹

Across 44 held-out documents of 8K to 32K tokens, negative log-likelihood is within 0.16% of BF16. A needle-retrieval test at 131,072 tokens returns 40 out of 40, with all 419 answer tokens matching the reference.¹

These distribution and retrieval checks emulate the default served arithmetic: INT6 trunk, int8 prompt activations and FP8 KV cache. The task benchmarks above run through the server. The separate 200K serving mode uses an 8-bit trunk and was not accuracy-tested separately.¹

Useful speed on two workstation GPUs

The published performance measurements use BetterBench 0.6.0, standard profile, through the exact release image and launcher, on two R9700 cards in a PCIe 4.0 x16 host with 251 GiB of RAM.²

Generated tokens per second; higher is better. The single-request number is a weighted decode score. The eight-request number is combined server throughput from a separate concurrency workload, not a rate per user or an eightfold scaling comparison.

All 48 of 48 requests completed at concurrency eight. BetterBench measures eight decode categories, but the weighted score combines six: code 30%, reasoning 20%, prose and JSON 15% each, file editing and summarization 10% each. Chat and math are measured without contributing to that score.²³

Long prompts

Prefill is reading the input before the model starts answering. The long-context mode processes the nominal 64K prompt depth at 6,991 input tokens/s. A separate cold-cache check with exactly 64,000 prompt tokens measures 7,162 tokens/s.²

Input tokens processed per second; higher is better. BetterBench prompt-depth settings are nominal, not exact token counts. These rows use 4bit-200k, with a 200,000-token window and speculative decoding off. They are separate from the default decode measurements.

Both use an FP8 E4M3 KV cache. Cache capacity is shared across requests; it is not the maximum window of a single conversation. Each window must fit the prompt, chat formatting and generated answer together.¹⁴

Faster repeated conversations with the RAM prefix cache

Coding agents resend their conversation on each turn. The optional 4bit-cache mode reuses unchanged prefixes in 2,048-token blocks and keeps evicted prefix blocks in an 8 GiB pinned RAM tier.²

Time to first token in separate repeated-prompt cache checks; lower is better. A cache hit requires a reusable prefix still present in the cache. These are not the cold-cache BetterBench prefill measurements.

The tested RAM-tier hits give identical output to a cold start. The tradeoff is about 22% less multi-user throughput in the cache-mode validation, 321.4 versus 411.3 aggregate tokens/s. That validation baseline is separate from the 420.7 BetterBench headline. New prompts also start 0.1 to 0.2 seconds later.²⁴

The full RAM tier needs about 108 GiB of free GTT, including its extra pinned memory. With less available, the launcher warns and shrinks or disables the RAM tier, keeping VRAM prefix caching. Use 4bit-cache explicitly; the ordinary 4bit and 4bit-200k modes reject --prefix-caching.²⁵

What 4-bit means here

Flash Next is a mixture-of-experts model. In each layer, the 410 most-used experts use INT4, calibrated with GPTQ in a rotated basis. The 102 least-used experts use the 3-bit codebook format from our earlier release. Together, the routed experts average 3.926 bits per weight, including scales. That average belongs to the mixed expert set, not to the 3-bit format alone.¹

The default serving mode uses a 6-bit trunk, while other parts retain 8-bit or higher precision. The MTP layer supplies speculative drafts, which the target model verifies. The model is not uniformly four bits.¹

Getting here took roughly 250 GPU-hours on an AMD Instinct MI355X system for quantization, tuning and accuracy testing, plus about 100 hours on the two R9700 cards testing the served model. Those estimates cover our 3-bit and 4-bit Flash Next releases together.

The full 4-bit expert set does not fit next to the runtime and cache on two 32 GB cards. About 20.8 GiB of experts per card stay in VRAM; the GPUs read the rest from pinned system memory when needed. The launcher pins roughly 96 GiB in total, about 55 GiB for tables and 41 GiB for expert copies. Host memory bandwidth and PCIe links therefore matter.²

The benchmarked allocation uses GPU-mapped GTT memory, with about 98 GiB free GTT needed at baseline. That is why we specify a 256 GB host. With less GTT, the launcher can fall back to ROCm user-pointer memory, warning about lower speed and possible pauses during host-memory compaction. It does not change kernel memory limits.²

Run the release

Use Linux x86-64, two Radeon AI PRO R9700 cards, a ROCm 10 host driver, Docker with access to /dev/kfd and /dev/dri, and Python 3.10 or newer with huggingface_hub installed. Allow 133 GB for weights, plus the roughly 41 GB runtime image and working space.⁶

The commands pin the launcher and download the complete model revision first. This avoids a first-start issue in the pinned launcher, which checks for the drafter before its automatic download. Use a fresh checkout if you already have local plugin changes.⁵

git clone https://github.com/Eliovp-BV/paiton-vllm-plugin.git
cd paiton-vllm-plugin
git checkout a985fa598d9a3c7536720304830a0e427f3af1f2

python3 -m venv .venv
. .venv/bin/activate
python3 -m pip install huggingface_hub
python3 - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="EliovpAI/Qwen3.8-Flash-Next-W4A8-Paiton-RDNA4",
    revision="9ec8e306503d5f9c9e8fd9c0d77de8fa63428888",
    local_dir="flashnext-w4a8",
)
PY
(cd flashnext-w4a8 && sha256sum -c SHA256SUMS)

python3 models/Qwen3.8-Flash-Next/launch-flashnext.py \
  --mode 4bit --weights "$PWD/flashnext-w4a8" --dry-run
bash models/Qwen3.8-Flash-Next/run-flashnext-4bit.sh \
  --weights "$PWD/flashnext-w4a8" --host 127.0.0.1

The dry run reports RAM, GTT limits and GPU-link checks. The launcher pulls the runtime by digest. After the weight download, first start takes around five to six minutes; wait for launcher: READY on port …. The API is http://127.0.0.1:18982/v1, with model name Qwen3.8-Flash-Next.⁶⁴

The default server binds all network interfaces without an API key. The example restricts it to localhost. For trusted remote clients, protect access with a firewall or set PAITON_API_KEY in the environment. Prefer the environment variable to --api-key, which exposes the key in the process list.⁶

Stop the running server before choosing another mode:

# 200K window, speculative decoding off
bash models/Qwen3.8-Flash-Next/run-flashnext-4bit-200k.sh \
  --weights "$PWD/flashnext-w4a8" --host 127.0.0.1

# Repeated coding conversations, opt-in RAM prefix cache
bash models/Qwen3.8-Flash-Next/run-flashnext-4bit-cache.sh \
  --weights "$PWD/flashnext-w4a8" --host 127.0.0.1

The 4-bit setup guide documents the options. These weights require the Paiton runtime and plugin; stock model loaders do not support the format.¹

Also in this release

The 3-bit Flash Next launcher gets a steadier startup on busy hosts, GPU-link and host-memory checks, and explicit 2,048-token prefix-cache blocks. The default all-interface binding and optional API key apply to both releases. The 3-bit weights and runtime image are unchanged.⁷

Qwen3.8 27B on one R9700 also gets a launcher update. Its 3-bit --mode long now holds 316,325 tokens of shared cache, and startup memory peaks are lower across the 3-bit modes. The MXFP4 configurations are unchanged.⁸

Working toward 128 GB hosts

Not everyone has 256 GB of RAM. We are working on a 4-bit configuration that keeps the large n-gram table in ordinary, unpinned memory, with the goal of preserving bit-identical output. The target is 128 GB machines. This is work in progress, not a released or qualified configuration.

Credits and licensing

Qwen3.8 Flash Next is by the Qwen team; vLLM supplies the serving framework and BetterBench the performance suite. The model weights remain under Qwen Community License 1.0, with commercial-use conditions. Review the upstream license before deployment.

Read the 4-bit model card and accuracy details, explore Paiton, or contact us about a local AMD AI workload.