A local assistant is more useful when it writes faster, starts processing a long document sooner and has room for more people to work at once. Our Paiton release brings those improvements to Qwen3.8 27B on 1 × Radeon AI PRO R9700 (32 GB), without adding another GPU.
In the original 26 September benchmark against our 24 September MXFP4 release, weighted generation speed rose from 153.8 to 184.4 tokens per second: 19.9% faster decode. Eight concurrent requests produced 492.1 tokens per second in aggregate, up from 425.3. The smaller weights initially left room for 43.5% more reported cache-token capacity with the FP8 cache in the 65K serving profile. The current image adds a 4-bit cache, increasing that reported capacity further.12
The change is our own rotated 3-bit weights for the large decoder projections, with 4-bit activation math for selected layers during prompt processing. Other tensors retain their existing formats. The tradeoff is real: the MMLU-Pro general-knowledge subset drops by about three percentage points against MXFP4 in the original test. Math and code differences are not distinguishable from noise in those tests, and all 80 bounded long-context retrieval checks pass. If knowledge accuracy matters most, the MXFP4 option remains one switch away.
Update, 30 September: one image, more ways to serve
The current release guide describes a unified image, updated to r2 on 29 September. The 65K text mode still supports up to eight scheduled requests. With W3A4 weights, its new 4-bit attention cache is on by default; --kv-cache fp8 restores the original benchmark's cache choice. The same image can instead serve one 200K-token conversation with prefix caching, or accept images in 65K mode with --vision. Those modes have different memory and performance characteristics, and vision cannot be combined with the tested long-context mode.2
The 26 September image behind the charts below could stop under some concurrent loads. The 28 September r1 image fixed that engine stop, but it and earlier images still had a separate DFlash2 draft-head initialization fault that could make speculative acceptance collapse on some hosts. Use the 29 September r2 image for new installations. Its launcher pins the image by digest; the setup below pins the repository and weights as well. The original 26 September measurements remain historical measurements, not a new benchmark of every r2 mode.2
What changed for everyday use
The 26 September release combined the existing MXFP4, FP8 cache, DFlash2 speculative decoding and Paiton's native AMD execution inside vLLM with smaller optional W3A4 weights. The result was faster output, faster prompt processing and additional cache space on the same card. The later image adds cache compression and mode choices to that foundation.
Those benefits are different measurements. Decode is generation after the first token. Aggregate throughput is output shared across simultaneous requests. Prefill is processing the prompt before generation begins. A higher token rate in one phase should not be presented as an identical reduction in total request time.
The headline 19.9% compares this release with the 24 September MXFP4 image, not the older Radiance comparison in our earlier Qwen3.8 article. That article's original 57% result remains a separate benchmark with its own configuration.
Faster generation, across different workloads
BetterBench 0.6.0, quick preset. Generation after the first token, in tokens/s; higher is better. Values are means of two complete runs. The weighted result rises 19.9% against the previous release. Workload gains vary, so this is not a promise that every prompt generates 20% faster. Published benchmark report.
| Workload | Previous MXFP4 release | This round, MXFP4 | W3A4 release | Change vs previous |
|---|---|---|---|---|
| Chat | 121.2 tok/s | 122.9 tok/s | 135.8 tok/s | +12.0% |
| Code | 179.9 tok/s | 182.7 tok/s | 226.0 tok/s | +25.6% |
| File edit | 179.9 tok/s | 182.7 tok/s | 195.0 tok/s | +8.5% |
| JSON | 217.5 tok/s | 220.8 tok/s | 269.0 tok/s | +23.7% |
| Math | 183.8 tok/s | 186.5 tok/s | 228.9 tok/s | +24.5% |
| Prose | 78.5 tok/s | 79.6 tok/s | 94.7 tok/s | +20.7% |
| Reasoning | 117.8 tok/s | 119.4 tok/s | 133.5 tok/s | +13.3% |
| Summarization | 138.4 tok/s | 140.5 tok/s | 158.4 tok/s | +14.4% |
| Weighted | 153.8 tok/s | 156.1 tok/s | 184.4 tok/s | +19.9% |
The middle column separates the smaller runtime improvements available with MXFP4 from the larger gain of the new weight format. Weighted decode on this round's MXFP4 path is about 1.5% higher than the previous release.
More output when requests overlap
Aggregate generated tokens/s, with 48 requests per concurrency level; higher is better. These are combined server rates, not the speed each user receives. One R9700, 65,536-token configured context, up to eight scheduled sequences.
| Concurrent requests | Previous MXFP4 release | W3A4 release | Change |
|---|---|---|---|
| 1 | 122.0 tok/s | 148.8 tok/s | +22.0% |
| 2 | 204.2 tok/s | 249.3 tok/s | +22.1% |
| 4 | 308.2 tok/s | 368.3 tok/s | +19.5% |
| 8 | 425.3 tok/s | 492.1 tok/s | +15.7% |
For a shared local assistant, this means the GPU can deliver more output while requests overlap. The exact improvement depends on the prompt lengths, output lengths and available cache.
Prompt processing improves too
Input tokens processed per second; higher is better. The depth labels are nominal BetterBench settings, not exact prompt lengths. The largest nominal 64K workload has a median actual prompt length of 47,016.5 tokens. Measured time to first token is shown separately below.
| Nominal depth | Median actual prompt tokens | Previous MXFP4 release | W3A4 release | Change |
|---|---|---|---|---|
| 2K | 1,516.5 | 3,689 tok/s | 4,156 tok/s | +12.7% |
| 8K | 5,894.5 | 3,834 tok/s | 4,165 tok/s | +8.6% |
| 16K | 11,802 | 3,871 tok/s | 4,103 tok/s | +6.0% |
| 32K | 23,549.5 | 3,751 tok/s | 3,958 tok/s | +5.5% |
| 64K | 47,016.5 | 3,455 tok/s | 3,629 tok/s | +5.0% |
The tested prefill rates improve by 5.0–12.7%. Client time to first token falls by 4.8–10.8%, comparing the mean of the two runs' median TTFTs. A throughput increase and a latency reduction are different percentages; scheduling and other overheads can also affect the wait.3
| Nominal depth | Previous TTFT, mean of run medians | W3A4 TTFT, mean of run medians | Less waiting |
|---|---|---|---|
| 2K | 407.453 ms | 363.395 ms | 10.8% |
| 8K | 1,548.047 ms | 1,416.748 ms | 8.5% |
| 16K | 3,054.554 ms | 2,877.005 ms | 5.8% |
| 32K | 6,270.297 ms | 5,943.109 ms | 5.2% |
| 64K | 13,617.798 ms | 12,962.238 ms | 4.8% |
More room for long conversations
Original 26 September FP8-cache comparison. Model memory falls from 19.18 to 15.89 GiB. The launcher reallocates the freed memory to the conversation cache, increasing reported cache-token slots from 174,634 to 250,578. The current 65K W3A4 default uses a 4-bit cache instead; see below. Cache capacity is an estimate from the serving configuration, not a count of fully measured conversations.
| Resource | Previous MXFP4 release | W3A4 release |
|---|---|---|
| Model memory | 19.18 GiB | 15.89 GiB |
| Reported cache-token capacity | 174,634 | 250,578 |
| Equivalent capacity at 65,536 tokens per sequence | About 2.7 | About 3.8 |
The last row is a fractional capacity estimate, not a claim that 3.8 requests can run. Prompt text, chat/tool formatting and generated output all consume context. Eight scheduled sequences also does not mean eight full-length 65K conversations fit simultaneously.1
We tested the practical effect separately with four prompts of about 61,400 tokens, each requesting 512 output tokens. Actual prompt lengths range from 61,390 to 61,426 tokens. The original cache budget fits only two of these requests at a time. Giving W3A4 the freed memory as cache lets all four decode together:
Four long requests on one R9700. Complete batch wall time is lower-is-better. This control uses the 26 September MXFP4 build, not the 24 September image. Expanded-cache W3A4 changes the resource budget and uses the shipped calibration; the same-cache W3A4 check used an earlier calibration.
| Long-context measurement | MXFP4 | W3A4, same cache budget | W3A4, expanded cache |
|---|---|---|---|
| Decode while all active requests run | 69 tok/s, 2 active | 117 tok/s, 2 active | 199 tok/s, 4 active |
| All four decoding together | No, two at a time | No, two at a time | Yes |
| Wall time, 4 × 61K prompts + 512 tokens each | 105.6 s | 92.9 s | 86.2 s |
The new weights make the two-request decode faster even without extra cache. Reallocating memory then reduces the wait for the whole four-request batch. The 199 tok/s figure is the aggregate decode rate while all four are active, not throughput across the complete batch. In this original FP8-cache test, whole-device peak usage was 31.39 GiB with W3A4 versus 31.37 GiB for the MXFP4 control, leaving little headroom on this card. This is one long-context workload, not a universal concurrency guarantee.3
The current 65K mode: a 4-bit cache for more overlapping work
The newer image stores the full-attention keys and values in 4 bits when W3A4 weights and the default 65K mode are selected. The 4-bit cache is not the default for MXFP4, the one-request 200K mode, or the desktop profile. It makes more of the same card available to vLLM's scheduler; it does not double the model's per-request context limit. In vLLM's startup log, reported cache capacity is 393,216 tokens, versus 250,578 with W3A4 plus FP8. After accounting for other per-request state, the eight-request attention-token budget rises from 211,136 to 358,336, or 1.70×.2
| Separate 65K cache test, one R9700 | 26 Sep image, W3A4 + FP8 | 28 Sep image, W3A4 + 4-bit |
|---|---|---|
| Four roughly 61K-token requests, combined decode while all run | 188 tok/s | 219 tok/s |
| Six roughly 61K-token requests | Not all admitted together | All six, 248 tok/s combined |
| Eight roughly 32K-token requests | Not all admitted together | All eight, 339 tok/s combined |
These rates are aggregate decode while all requests are active, not per-user rates or whole-batch throughput. The four-request FP8 result is 188 tok/s in this paired test, versus 199 tok/s in the earlier test above; compare arms within each test, not across runs. These cache measurements predate the r2 draft-head fix. The 4-bit cache did not measurably improve single-request decode or time to first token. Paired checks against the FP8 cache found no observed loss on GSM8K, HumanEval, the MMLU-Pro subset or 61K needle retrieval, but those bounded checks do not prove identical quality. A 24-session synthetic coding-agent test also found differences within noise.2
A separate 200K conversation mode, plus optional image input
The current unified image can serve one conversation up to 200,000 total tokens with an 8 GiB FP8 cache and prefix caching. The release guide reports 94 seconds to first token for a fresh 198,989-token prompt with W3A4, 102 seconds for MXFP4 in the unified image, and 115 seconds for the earlier standalone 200K image; it does not identify these as a fresh r2 rerun. A follow-up reusing the unchanged prefix took 1.2 seconds; at that length W3A4 decode ran at 73–78 tokens/s. The team also tested a 215K-token prompt with a 220K setting, but not the model's full 262K window. The planted-fact and tool-call checks passed; they are not a broad long-conversation quality evaluation. Cache hits require an unchanged prefix still resident in memory.2
For image input, --vision loads the checkpoint's vision encoder in 65K mode only. It was checked on code screenshots, a failing test, a form and a chart, including eight simultaneous image requests. The encoder uses memory otherwise available to cache: the W3A4 + 4-bit setup reports 278,050 cache tokens with vision versus 393,216 without it. Video and vision with the prefix-cached long-context mode are not qualified.2
What we changed, and why
First, make the existing runtime predictable
Before changing weights, we investigated server starts that alternated between roughly 28 and 36 ms per speculative step despite an unchanged configuration. Pinning the runtime to one hardware compute queue with GPU_MAX_HW_QUEUES=1 kept those starts in the faster mode. The image includes that setting. All comparison arms already used it, so its gain is not included in the headline improvement.
We also combined the GatedDeltaNet speculative verification operation into one native kernel. That operation measured 23–28% faster, while the combined runtime fixes improve end-to-end weighted MXFP4 decode by about 1.5%. An individual kernel gain and the overall generation gain are not the same thing.
Read fewer bytes when generating
Decode on this card is largely limited by memory bandwidth. The previous kernels were already close to the card's available bandwidth, so reducing weight traffic offered more room than scheduling changes alone.
The large linear projections move from approximately 4.25 bits per weight with MXFP4 to 3.125 bits with grouped INT3. This affects 24.3 billion decoder-projection weights, not every tensor in the 27B model, and is roughly a quarter less data for those weights. Native Paiton kernels use the packed values directly. Decode keeps 8-bit FP8 activations; in the serving comparison, median single-stream forward time falls from about 28.3–28.4 ms to 22.4–22.5 ms.13
Use different precision for prompt processing
Prompt processing has a different bottleneck. Simply using 3-bit weights with the old 8-bit activation path made it slower in our checks. The release therefore uses 4-bit activation math only in selected prefill projections, while generation keeps 8-bit activations.
A block-wise Hadamard rotation spreads large activation outliers across small groups, making lower-precision prompt processing more useful. The weights are calibrated in that rotated basis, and the runtime applies the corresponding activation transform. Grouped scaling retains more local precision than one scale for an entire token. This is the public method behind the W3A4 name, not a new model architecture.1
Calibrate our own weights and check the data licenses
These are our own GPTQ-calibrated weights, rather than a required third-party 3-bit checkpoint. Calibration uses 292,864 tokens across math, code, science, web text, multilingual material, agentic traces and long documents. The layer-by-layer process takes about 13 minutes on a single AMD Instinct MI355X.
Before release, we replaced calibration sources carrying noncommercial or share-alike terms with permissively licensed sources. The shipped weights use that final calibration. The calibration report lists dataset revisions and licenses; the third-party notices preserve required attributions.
Keep startup and integrity checks explicit
The runtime loads the rotated replacements alongside the pinned base checkpoint and checks their files by SHA-256. No separate third-party 3-bit model is needed. We also fixed a warm-up issue where uninitialized dummy inputs could trigger a non-finite-value flag before the first real request: the flag is cleared after warm-up, while strict checks remain enabled for served requests.
With a warm runtime cache, the 26 September image started in about 230 seconds, compared with about 210 seconds previously. The current image may take about seven minutes on its first start with an empty runtime cache. Warm serving gains do not remove that initial preparation time.2
Quality: the benefit comes with a choice
Served-model accuracy with greedy decoding, thinking disabled and identical questions. The paired 95% confidence intervals include zero for GSM8K and HumanEval, but not for the MMLU-Pro subset. The 80 needle tests are a bounded retrieval check, not broad long-context validation.
| Benchmark | MXFP4 | W3A4 | Difference and paired 95% confidence interval |
|---|---|---|---|
| GSM8K, 5-shot, 1,319 questions | 95.68% | 95.30% | −0.38 pts −1.44, +0.68 |
| HumanEval pass@1, 164 tasks | 95.12% | 93.90% | −1.22 pts −4.99, +2.56 |
| MMLU-Pro subset, 0-shot, 14 × 100 questions | 62.57% | 59.71% | −2.86 pts −4.81, −0.90 |
| Needle retrieval at 61,440 tokens, 80 checks | 100% | 100% | No observed difference |
Math and code stay close in the evaluated sets; the intervals do not prove identical quality. Knowledge-heavy multiple choice shows a measurable reduction of about three points. DFlash2 acceptance also remains close, with task-level changes from −0.4% to +2.8%.1
For knowledge-heavy work, choose --weights mxfp4. It uses the higher-precision weight path and still receives this round's MXFP4 runtime improvements. W3A4 is useful when faster generation and more conversation cache are worth that measured accuracy tradeoff. Outputs can differ; we do not promise identical answers.
Multilingual, tool-calling and longer-context quality were not measured in the original W3A4-versus-MXFP4 evaluation. The original throughput and paired accuracy results cover the 65K text profile, not every mode of the current image. The newer release has separate bounded 200K conversation, tool-call and 65K vision checks above; these do not establish broad quality parity. W3A4 remains R9700-specific and requires the Paiton runtime; stock vLLM, Transformers and llama.cpp cannot load the replacement weights on their own.12
How to read these results
The performance comparison uses one R9700 at its 300 W cap, vLLM 0.29, ROCm 10, a 65,536-token context, up to eight sequences, DFlash2 and thinking disabled. Automatic prefix caching and n-gram co-drafting are off. Each image ran in two fresh server processes, interleaved, and the tables show the mean of the two runs. BetterBench used version 0.6.0 and its quick preset. The reference image is qwen38-rocm10-vllm029-65k-20260924-r3.
The original public performance runs used the earlier v3 calibration. The original accuracy figures and expanded-cache four-request run use the final, permissively calibrated weights. Their tensor format and runtime are identical, but these are not all tests of one identical set of weight bytes. Sampled text and speculative acceptance can differ: W3A4 changed seven of twelve greedy control outputs relative to MXFP4. These are serving-throughput measurements, not identical-output timing. Percentage changes come from unrounded measurements, so recalculating from the rounded tables can differ slightly.13
BetterBench's weighted decode emphasizes code (30%), reasoning (20%), prose and JSON (15% each), then file editing and summarization (10% each). Chat and math are shown separately and have no weight in that headline. Each run scores five requests per decode category, 48 per concurrency level and eight per nominal prefill depth, after fixed warmups. File-edit results vary most between the two W3A4 runs: 179.0 and 211.1 tok/s. The full report and source data retain those measurement details.
The 300 W figure is the configured GPU power cap. We did not measure whole-system energy or cost per token, and faster generation alone is not an electricity-savings measurement. The 4-bit cache, 200K and vision results above are separate checks, not rows in the original BetterBench tables. A comparison in the release guide found the 28 September image's 65K prefill within ±0.3% and aggregate throughput within −2.4% to 0.0% of the 26 September image; it does not turn the original 492.1 tok/s into a fresh r2 measurement.2
Try it on your R9700
Use Linux x86-64, Python 3, Docker, the Hugging Face CLI and AMD GPU device access. Allow about 75 GB of disk for the image and three model packages. The commands below pin the current public launcher checkout and the target, drafter and rotated-weight revisions together. If you already have the plugin repository, use a separate checkout rather than replacing local work. The release setup guide documents existing downloads and other profiles.2
git clone https://github.com/Eliovp-BV/paiton-vllm-plugin.git
cd paiton-vllm-plugin
git checkout 86ff71c4a3ccaa5da410c2b94e1976fab1c6ab87
export PAITON_TARGET_DIR="$PWD/model-cache/qwen38-nvfp4"
export PAITON_DRAFT_DIR="$PWD/model-cache/qwen38-dflash2"
export PAITON_W3ROT_DIR="$PWD/model-cache/qwen38-w3rot-int3"
export PAITON_CACHE_DIR="$PWD/runtime-cache/qwen38-rocm10-20260928"
mkdir -p "$PAITON_TARGET_DIR" "$PAITON_DRAFT_DIR" "$PAITON_W3ROT_DIR" "$PAITON_CACHE_DIR"
hf download unsloth/Qwen3.8-27B-NVFP4 \
--revision f0b7c9e722f5565102fff8481c99e4d86ae099c7 --local-dir "$PAITON_TARGET_DIR"
hf download tcclaviger/Qwen3.8-27B-DFlash2-FP8 \
--revision ee0cb26a8279b7910cc28d82a8a3e15e4728d56f --local-dir "$PAITON_DRAFT_DIR"
hf download EliovpAI/Qwen3.8-27B-W3Rot-INT3-Paiton-RDNA4 \
--revision d74ae7d5f5f1b4b45dd12fb1271e3664283a2ec1 --local-dir "$PAITON_W3ROT_DIR"
(cd "$PAITON_W3ROT_DIR" && sha256sum -c SHA256SUMS)
bash models/Qwen3.8-MXFP4-DFlash2/run-rocm10.sh
The current unified image, selected by that launcher, is ghcr.io/eliovp/paiton-vllm-plugin:qwen38-rocm10-vllm029-20260929-r2. Its immutable digest is:
sha256:1195f31329966b3dc4e8e2d17327d339827b3d2b09165f9969b053a6fc2db045
When PAITON_W3ROT_DIR is set, the launcher selects the rotated 3-bit weights and, in 65K mode, the 4-bit cache. Without that variable, it serves MXFP4. Stop the first server before choosing another mode or selecting MXFP4 with:
bash models/Qwen3.8-MXFP4-DFlash2/run-rocm10.sh --weights mxfp4
For one long conversation, start the same launcher instead with --context 200000; for image input in 65K mode, use --vision. Only one mode should run on the GPU at a time. To recreate the original FP8-cache profile with W3A4, add --kv-cache fp8 to the 65K command. Once ready, the server exposes the OpenAI-compatible API at http://127.0.0.1:18982/v1 using the model name Qwen3.8. In another terminal, send a streaming request:
curl --fail http://127.0.0.1:18982/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"Qwen3.8","messages":[{"role":"user","content":"Write a short Python function that removes duplicates while keeping their original order."}],"temperature":0,"max_tokens":256,"stream":true,"chat_template_kwargs":{"enable_thinking":false}}'
These commands run the model-specific W3A4 Docker profile with DFlash2. The shorter native paiton serve qwen38-nvfp4 preset is a different, non-speculative configuration and does not load these 3-bit weights.2
What is next
The 4-bit cache now adds usable capacity in the qualified 65K W3A4 mode; it does not change the one-request 200K profile or establish a full 262K window. The model's full window and an adaptive longer speculative block remain investigations, not published settings in this release.2
Credits and licensing
Qwen3.8 27B is by the Qwen team. The pinned target checkpoint is provided by Unsloth, and the DFlash2 drafter by tcclaviger, building on z-lab's DFlash2 work. The released rotated weights are an Apache-2.0 quantized derivative; the model card records Apache-2.0 terms for those upstream checkpoints. The public method builds on GPTQ and GSQ. vLLM provides the serving foundation, and BetterBench the generation workload suite. Radiance and StillDeadcode/libr4d are credited for adapted kernel techniques in the public runtime guide.12
The public adapter, model weights and packaged native runtime have distinct terms; the weights' Apache-2.0 license is not a blanket license for proprietary compiler or implementation code. Calibration datasets also retain their own attribution requirements and, where applicable, Common Crawl terms. Review the published notices before deployment.
Explore Paiton, read the previous Qwen3.8 release story, follow the community discussion and field reports, or contact us to discuss a local AMD AI workload.