3-bit accuracy. Faster conversation restarts.

Illustration: AI-generated editorial artwork, not a hardware photograph or benchmark capture.

Better answers, less waiting between turns. Our 24 October Paiton release brings new 3-bit Qwen3.8 Flash Next weights close to the 16-bit reference on three task benchmarks, plus native prefix caching with multi-token prediction (MTP) for long conversations on two Radeon AI PRO R9700 cards. A separate accuracy update also improves our 27B model on one card.¹²

The latest 3-bit cache mode resumes a repeated 200K-token prompt in 0.68 seconds, versus 25.81 seconds cold, while keeping speculative decoding on. These are cache-hit measurements, not the wait for a new document.³

Accuracy: close task scores at 3-bit

The new weights improve the calibration behind our original 3-bit Flash Next release, without changing its format, size or kernels. Here they are alongside the 4-bit release and original 16-bit model.¹

Accuracy; higher is better. Text tasks ran through the released server configurations, not the native throughput setup below. AI2D emulates served arithmetic and is shown separately. The 3-bit results are from 24 October; 4-bit and 16-bit references are from 12 October.

None of the three text-task differences from 16-bit is statistically significant in the paired tests. That does not establish equivalence: HumanEval is 1.8 percentage points lower, with a paired 95% interval of −5.0 to +1.3 points. The 16-bit reference itself moved by up to 1.2 points between two seed sets.¹⁴

Vision remains behind the reference. The AI2D evaluation does not establish native-worker image-input support: the released 3-bit modes are text-only.¹⁵

How close are the token predictions?

KL measures differences in next-token probability distributions; zero means identical distributions. Negative log-likelihood measures how well predictions fit observed text. These percentages are relative NLL changes, not perplexity changes. Distribution checks emulate served arithmetic; the separate retrieval checks use the served model. Neither replaces broad task evaluation.¹⁴

What changed in the weights

The recipe combines separate calibration statistics for each expert's output projection, a more varied calibration set of about four million tokens, and retained 8-bit shared layers around the 3-bit experts. Only the rounding changes; the packed format and runtime kernels stay the same.¹

Prefix caching with MTP still on

Coding agents resend their conversation on every turn. Prefix caching reuses the unchanged part; MTP drafts several next tokens for the target model to verify. The native worker now combines both. It runs the model step without PyTorch on the GPU, while vLLM still provides the API and scheduling.⁵

Time to first token in separate repeated-prompt checks; lower is better. A hit needs an unchanged prefix still available in GPU or RAM cache. These are not cold-cache prefill-throughput benchmarks.

Tested requests served alone produce identical cached and cold output; concurrent batching can change tokens on either path. An optional RAM tier retains evicted sessions and sizes itself to available memory. The 4-bit cache configuration generates at 206.9 tokens/s with that tier on.³

The newer 3bit-long-cache-mtp-native mode extends this to a 262,144-token window: 213.5 tokens/s with the RAM tier on, 250.0 off. Its cache and decode figures are separate validation measurements, not reruns of the main release comparison below. Prompt, formatting and answer must fit the window together.³

For smaller hosts, --mode decode --fn-worker native --prefix-caching supports 64 GB RAM, a 98,304-token window and about 101 tokens/s, without MTP. It is a different mode, not the configuration behind the 213.5 tokens/s result.⁵

One near-capacity test with 2.05 million idle-session tokens resumed cold despite expected RAM hits. Output stayed correct; the issue remains open. Also, the 3-bit model is more prone to early endings beyond roughly 28K context in tested sessions. The public guidance prefers 4-bit for long coding conversations.³¹

Generation speed on two workstation cards

The accuracy update changes the weight values, not the amount of data read or the kernels. The published native-worker throughput remains in the same performance range.³

Two R9700 cards, native worker, prefix caching off. Higher is better. Single-request weighted decode and aggregate concurrent throughput are different workloads, not a per-user scaling comparison.

These are opt-in native-worker results. Ordinary decode and 4bit commands still select the torch worker unless requested otherwise. Neither the cache modes nor their RAM tiers are included in this table.⁵³

A second accuracy update for Qwen3.8 27B

Our 3-bit 27B release runs on one R9700. This update keeps the 3-bit codes fixed and further trains their scale factors toward the 16-bit model, using 4.2 million new tokens separate from earlier calibration and reported benchmarks.²

Served KL falls 3.0%, with a 95% interval of 0.4% to 6.4%. Long-document predictions improve across 8K–32K buckets; the 131K check finds no worsening, with gains from 64K onward. Format, size and kernel cost are unchanged; measured throughput can vary with speculative acceptance. New task checks show no significant change, but are weight-only evaluations, not fresh served-model task runs.²

How to read the measurements

Flash Next text tasks use thinking on, temperature 1.0, top-p 0.95, top-k 20 and a 16,384-token answer limit. Prompts and per-question seeds match; capped answers count as wrong. GSM8K and HumanEval use their full test sets; MMLU-Pro uses 100 questions from each of 14 subjects. The original 16-bit weights ran twice on a larger AMD server. Comparisons use paired confidence intervals and exact McNemar tests.¹⁴

Performance uses BetterBench 0.6.0, standard profile, one run per configuration, and LocalMaxxing v0.1.54, retaining the median single-user-throughput run of three after warm-up. These are our measurements, not independent reproductions.³

Try the release

Use Linux x86-64, two R9700 cards, a ROCm 10 host driver, Docker GPU access and Python 3.10+. Fast 3-bit native and 3-bit MTP caching need roughly 128 GB+ RAM and 60 GiB free GTT; 4-bit native needs about a 98 GiB GTT limit. Preflight checks the actual host. Allow 116 GB for 3-bit weights or 133 GB for 4-bit, plus the roughly 56 GB native image.⁵

Use a fresh checkout if you have local plugin changes. The pinned launcher selects its runtime and model revisions, downloads the updated 3-bit weights into a new folder and leaves the old download intact.⁶

git clone https://github.com/Eliovp-BV/paiton-vllm-plugin.git
cd paiton-vllm-plugin
git checkout 4882e36e455c1e657175db910367c992837c72a5

python3 -m venv .venv
. .venv/bin/activate
python3 -m pip install huggingface_hub

python3 models/Qwen3.8-Flash-Next/launch-flashnext.py \
  --mode 3bit-long-cache-mtp-native --host 127.0.0.1 --dry-run
python3 models/Qwen3.8-Flash-Next/launch-flashnext.py \
  --mode 3bit-long-cache-mtp-native --host 127.0.0.1

For 4-bit MTP caching, use bash models/Qwen3.8-Flash-Next/run-flashnext-4bit.sh with --fn-worker native --prefix-caching --host 127.0.0.1 instead. Wait for launcher: READY; the API is http://127.0.0.1:18982/v1, model Qwen3.8-Flash-Next.⁵

For 27B, follow the pinned setup and upgrade guide. Download its new weight revision explicitly; updating the launcher alone does not update those weights.

The server's default binding exposes all interfaces without an API key. Keep --host 127.0.0.1 for local use. For remote clients, protect access with a firewall and PAITON_API_KEY; avoid placing a key directly in command arguments. Stop the running server before selecting another mode.⁵

Native modes reject structured outputs, strict/required tool calls and other API options; ordinary automatic tool calling works. Check the compatibility notes before connecting an agent.⁵

Credits and licensing

Qwen supplies the models, vLLM the serving framework, and BetterBench and LocalMaxxing the benchmark tools. Flash Next weights use Qwen Community License 1.0, with commercial-use conditions; the 27B weights use Apache 2.0. Review the model licenses and calibration attributions before deployment.¹²

Explore Paiton for the runtime, or sovereign AI for a locally controlled deployment. For the underlying calibration, modes and reproducibility details, follow the pinned sources below.