3-bit weights. 20% faster decode. 1 × Radeon AI PRO R9700.

A local assistant is more useful when it writes faster, starts processing a long document sooner and has room for more people to work at once. Our new Paiton release brings those improvements to Qwen3.8 27B on 1 × Radeon AI PRO R9700 (32 GB), without adding another GPU.

Against our previous 24 September MXFP4 release, weighted generation speed rises from 153.8 to 184.4 tokens per second: 19.9% faster decode. Eight concurrent requests produce 492.1 tokens per second in aggregate, up from 425.3. Smaller model weights also leave room for 43.5% more reported cache-token capacity in the 65K serving profile.¹

The change is our own rotated 3-bit weights for the large decoder projections, with 4-bit activation math for selected layers during prompt processing. Other tensors retain their existing formats. The tradeoff is real: the MMLU-Pro general-knowledge subset drops by about three percentage points. Math and code differences are not distinguishable from noise in these tests, and all 80 bounded long-context retrieval checks pass. If knowledge accuracy matters most, the MXFP4 option remains one switch away.

What changed for everyday use

The previous release already combined MXFP4 weights, an FP8 cache, DFlash2 speculative decoding and Paiton's native AMD execution inside vLLM. This update keeps that serving foundation and reduces the amount of model data the GPU needs to read. The result is faster output, faster prompt processing and additional cache space on the same card.

Those benefits are different measurements. Decode is generation after the first token. Aggregate throughput is output shared across simultaneous requests. Prefill is processing the prompt before generation begins. A higher token rate in one phase should not be presented as an identical reduction in total request time.

The headline 19.9% compares this release with the 24 September MXFP4 image, not the older Radiance comparison in our earlier Qwen3.8 article. That article's original 57% result remains a separate benchmark with its own configuration.

Faster generation, across different workloads

BetterBench 0.6.0, quick preset. Generation after the first token, in tokens/s; higher is better. Values are means of two complete runs. The weighted result rises 19.9% against the previous release. Workload gains vary, so this is not a promise that every prompt generates 20% faster. Published benchmark report.

The middle column separates the smaller runtime improvements available with MXFP4 from the larger gain of the new weight format. Weighted decode on this round's MXFP4 path is about 1.5% higher than the previous release.

More output when requests overlap

Aggregate generated tokens/s, with 48 requests per concurrency level; higher is better. These are combined server rates, not the speed each user receives. One R9700, 65,536-token configured context, up to eight scheduled sequences.

For a shared local assistant, this means the GPU can deliver more output while requests overlap. The exact improvement depends on the prompt lengths, output lengths and available cache.

Prompt processing improves too

Input tokens processed per second; higher is better. The depth labels are nominal BetterBench settings, not exact prompt lengths. The largest nominal 64K workload has a median actual prompt length of 47,016.5 tokens. Measured time to first token is shown separately below.

The tested prefill rates improve by 5.0–12.7%. Client time to first token falls by 4.8–10.8%, comparing the mean of the two runs' median TTFTs. A throughput increase and a latency reduction are different percentages; scheduling and other overheads can also affect the wait.²

More room for long conversations

Model memory falls from 19.18 to 15.89 GiB. The launcher reallocates the freed memory to the conversation cache, increasing reported cache-token slots from 174,634 to 250,578. Cache capacity is an estimate from the serving configuration, not a count of fully measured conversations.

The last row is a fractional capacity estimate, not a claim that 3.8 requests can run. Prompt text, chat/tool formatting and generated output all consume context. Eight scheduled sequences also does not mean eight full-length 65K conversations fit simultaneously.¹

We tested the practical effect separately with four prompts of about 61,400 tokens, each requesting 512 output tokens. Actual prompt lengths range from 61,390 to 61,426 tokens. The original cache budget fits only two of these requests at a time. Giving W3A4 the freed memory as cache lets all four decode together:

Four long requests on one R9700. Complete batch wall time is lower-is-better. This control uses the 26 September MXFP4 build, not the 24 September image. Expanded-cache W3A4 changes the resource budget and uses the shipped calibration; the same-cache W3A4 check used an earlier calibration.

The new weights make the two-request decode faster even without extra cache. Reallocating memory then reduces the wait for the whole four-request batch. The 199 tok/s figure is the aggregate decode rate while all four are active, not throughput across the complete batch. Whole-device peak usage remains nearly the same: 31.39 GiB with the default W3A4 cache versus 31.37 GiB for the MXFP4 control, leaving little headroom on this card. This is one long-context workload, not a universal concurrency guarantee.²

What we changed, and why

First, make the existing runtime predictable

Before changing weights, we investigated server starts that alternated between roughly 28 and 36 ms per speculative step despite an unchanged configuration. Pinning the runtime to one hardware compute queue with GPU_MAX_HW_QUEUES=1 kept those starts in the faster mode. The image includes that setting. All comparison arms already used it, so its gain is not included in the headline improvement.

We also combined the GatedDeltaNet speculative verification operation into one native kernel. That operation measured 23–28% faster, while the combined runtime fixes improve end-to-end weighted MXFP4 decode by about 1.5%. An individual kernel gain and the overall generation gain are not the same thing.

Read fewer bytes when generating

Decode on this card is largely limited by memory bandwidth. The previous kernels were already close to the card's available bandwidth, so reducing weight traffic offered more room than scheduling changes alone.

The large linear projections move from approximately 4.25 bits per weight with MXFP4 to 3.125 bits with grouped INT3. This affects 24.3 billion decoder-projection weights, not every tensor in the 27B model, and is roughly a quarter less data for those weights. Native Paiton kernels use the packed values directly. Decode keeps 8-bit FP8 activations; in the serving comparison, median single-stream forward time falls from about 28.3–28.4 ms to 22.4–22.5 ms.¹²

Use different precision for prompt processing

Prompt processing has a different bottleneck. Simply using 3-bit weights with the old 8-bit activation path made it slower in our checks. The release therefore uses 4-bit activation math only in selected prefill projections, while generation keeps 8-bit activations.

A block-wise Hadamard rotation spreads large activation outliers across small groups, making lower-precision prompt processing more useful. The weights are calibrated in that rotated basis, and the runtime applies the corresponding activation transform. Grouped scaling retains more local precision than one scale for an entire token. This is the public method behind the W3A4 name, not a new model architecture.¹

Calibrate our own weights and check the data licenses

These are our own GPTQ-calibrated weights, rather than a required third-party 3-bit checkpoint. Calibration uses 292,864 tokens across math, code, science, web text, multilingual material, agentic traces and long documents. The layer-by-layer process takes about 13 minutes on a single AMD Instinct MI355X.

Before release, we replaced calibration sources carrying noncommercial or share-alike terms with permissively licensed sources. The shipped weights use that final calibration. The calibration report lists dataset revisions and licenses; the third-party notices preserve required attributions.

Keep startup and integrity checks explicit

The runtime loads the rotated replacements alongside the pinned base checkpoint and checks their files by SHA-256. No separate third-party 3-bit model is needed. We also fixed a warm-up issue where uninitialized dummy inputs could trigger a non-finite-value flag before the first real request: the flag is cleared after warm-up, while strict checks remain enabled for served requests.

Startup is about 230 seconds, compared with about 210 seconds previously. The warm serving gains do not remove this initial loading time.

Quality: the benefit comes with a choice

Served-model accuracy with greedy decoding, thinking disabled and identical questions. The paired 95% confidence intervals include zero for GSM8K and HumanEval, but not for the MMLU-Pro subset. The 80 needle tests are a bounded retrieval check, not broad long-context validation.

Math and code stay close in the evaluated sets; the intervals do not prove identical quality. Knowledge-heavy multiple choice shows a measurable reduction of about three points. DFlash2 acceptance also remains close, with task-level changes from −0.4% to +2.8%.¹

For knowledge-heavy work, choose --weights mxfp4. It uses the higher-precision weight path and still receives this round's MXFP4 runtime improvements. W3A4 is useful when faster generation and more conversation cache are worth that measured accuracy tradeoff. Outputs can differ; we do not promise identical answers.

Multilingual, tool-calling and longer-context quality were not measured in this evaluation. The released W3A4 package is text-only and R9700-specific, and these performance and quality results cover the 65K profile. Other launcher profile combinations have not been measured; the separate 200K image continues to use MXFP4. These replacement weights require the Paiton runtime; stock vLLM, Transformers and llama.cpp cannot load them on their own.¹³

How to read these results

The performance comparison uses one R9700 at its 300 W cap, vLLM 0.29, ROCm 10, a 65,536-token context, up to eight sequences, DFlash2 and thinking disabled. Automatic prefix caching and n-gram co-drafting are off. Each image ran in two fresh server processes, interleaved, and the tables show the mean of the two runs. BetterBench used version 0.6.0 and its quick preset. The reference image is qwen38-rocm10-vllm029-65k-20260924-r3.

The public performance runs used the earlier v3 calibration. The accuracy figures and expanded-cache four-request run use the final, permissively calibrated weights that are shipped. Their tensor format and runtime are identical, but these are not all tests of one identical set of weight bytes. Sampled text and speculative acceptance can differ: W3A4 changed seven of twelve greedy control outputs relative to MXFP4. These are serving-throughput measurements, not identical-output timing. Percentage changes come from unrounded measurements, so recalculating from the rounded tables can differ slightly.¹²

BetterBench's weighted decode emphasizes code (30%), reasoning (20%), prose and JSON (15% each), then file editing and summarization (10% each). Chat and math are shown separately and have no weight in that headline. Each run scores five requests per decode category, 48 per concurrency level and eight per nominal prefill depth, after fixed warmups. File-edit results vary most between the two W3A4 runs: 179.0 and 211.1 tok/s. The full report and source data retain those measurement details.

The 300 W figure is the configured GPU power cap. We did not measure whole-system energy or cost per token, and faster generation alone is not an electricity-savings measurement.

Try it on your R9700

Use Linux x86-64, Python 3, Docker, the Hugging Face CLI and AMD GPU device access. The commands below pin the public release checkout and the target, drafter and rotated-weight revisions together. If you already have the plugin repository, use a separate checkout rather than replacing local work. The release setup guide documents existing downloads and other profiles.³

git clone https://github.com/Eliovp-BV/paiton-vllm-plugin.git
cd paiton-vllm-plugin
git checkout a44044105287da3d040652ea8bcd3927a94a1bf2

export PAITON_TARGET_DIR="$PWD/model-cache/qwen38-nvfp4"
export PAITON_DRAFT_DIR="$PWD/model-cache/qwen38-dflash2"
export PAITON_W3ROT_DIR="$PWD/model-cache/qwen38-w3rot-int3"
export PAITON_CACHE_DIR="$PWD/runtime-cache/qwen38-rocm10-65k-w3a4"
mkdir -p "$PAITON_TARGET_DIR" "$PAITON_DRAFT_DIR" "$PAITON_W3ROT_DIR" "$PAITON_CACHE_DIR"

hf download unsloth/Qwen3.8-27B-NVFP4 \
  --revision f0b7c9e722f5565102fff8481c99e4d86ae099c7 --local-dir "$PAITON_TARGET_DIR"
hf download tcclaviger/Qwen3.8-27B-DFlash2-FP8 \
  --revision ee0cb26a8279b7910cc28d82a8a3e15e4728d56f --local-dir "$PAITON_DRAFT_DIR"
hf download EliovpAI/Qwen3.8-27B-W3Rot-INT3-Paiton-RDNA4 \
  --revision 278486debe64e21e5e9d45ac8d02798d72fbdf83 --local-dir "$PAITON_W3ROT_DIR"
(cd "$PAITON_W3ROT_DIR" && sha256sum -c SHA256SUMS)

bash models/Qwen3.8-MXFP4-DFlash2/run-rocm10-65k.sh

The released W3A4 image is ghcr.io/eliovp/paiton-vllm-plugin:qwen38-rocm10-vllm029-65k-20260926-w3a4-r1. Its immutable digest is:

sha256:c4134aba665f6dd3b89354a43be2b5b814f7078db456351647a3f1b106a0da49

When PAITON_W3ROT_DIR is set, the model-specific launcher selects the rotated 3-bit weights. After stopping that server, select MXFP4 with:

bash models/Qwen3.8-MXFP4-DFlash2/run-rocm10-65k.sh --weights mxfp4

Once ready, the server exposes the OpenAI-compatible API at http://127.0.0.1:18982/v1 using the model name Qwen3.8. In another terminal, send a streaming request:

curl --fail http://127.0.0.1:18982/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.8","messages":[{"role":"user","content":"Write a short Python function that removes duplicates while keeping their original order."}],"temperature":0,"max_tokens":256,"stream":true,"chat_template_kwargs":{"enable_thinking":false}}'

These commands run the model-specific W3A4 Docker profile with DFlash2. The shorter native paiton serve qwen38-nvfp4 preset is a different, non-speculative configuration and does not load these 3-bit weights.³

What is next, and what is not shipped

A 4-bit conversation cache passed our bounded accuracy gate, but currently reduces attention traffic without adding usable cache capacity. Its engineering gain was about 4–5% with two 61K requests. Doubling capacity in the same memory is still work in progress, not a benefit of this release. A longer speculative block is another investigation, not a published setting here.

Credits and licensing

Qwen3.8 27B is by the Qwen team. The pinned target checkpoint is provided by Unsloth, and the DFlash2 drafter by tcclaviger, building on z-lab's DFlash2 work. The released rotated weights are an Apache-2.0 quantized derivative; the model card records Apache-2.0 terms for those upstream checkpoints. The public method builds on GPTQ and GSQ. vLLM provides the serving foundation, and BetterBench the generation workload suite. Radiance and StillDeadcode/libr4d are credited for adapted kernel techniques in the public runtime guide.¹³

The public adapter, model weights and packaged native runtime have distinct terms; the weights' Apache-2.0 license is not a blanket license for proprietary compiler or implementation code. Calibration datasets also retain their own attribution requirements and, where applicable, Common Crawl terms. Review the published notices before deployment.

Explore Paiton, read the previous Qwen3.8 release story, or contact us to discuss a local AMD AI workload.