Qwen3.8 Flash Next on an AMD MI350P: twice the speed

On one CDNA4-based AMD Instinct MI350P, Paiton generates text at 273.1 tokens per second, compared with 122.1 for our tested stock vLLM + AITER setup. That is 2.24× the speed for a single request. With 64 requests running together, total output reaches 2,508.6 tokens/s.

There is more room for conversation history, too: about 2.2 million tokens of GPU KV-cache capacity, 22% more than the tested vLLM setup. That capacity is shared across requests, not a 2.2-million-token context window for one conversation.

The practical benefit is less waiting while text appears and more output from the same GPU. The headline comparison measures generation from an empty context, not the complete time from prompt to answer. It compares our A5 weights running on Paiton's native CDNA4 runtime against AMD Quark MXFP4 on vLLM, so it is a full-stack comparison rather than an engine-only test.

Our 4-bit quantization also stays close to the original model: 93.78% next-token agreement with BF16 in the separate weight evaluation. That measures agreement between predictions, not the percentage of correct answers. The weights and quantization recipe are available under the upstream Qwen Community License 1.0.

These are results from Paiton, our AI optimization framework, using its native CDNA4 runtime. The quantization targets the CDNA4 family, including MI350P, MI350X and MI355X, with a compatible runtime and sufficient GPU and host memory. The measurements below are from one MI350P only. The A5 format currently requires the proprietary Paiton runtime, which is not distributed with the weights; stock vLLM, Transformers and llama.cpp cannot load it directly.

What Paiton delivers on one MI350P

With the A5 weights and Paiton's native CDNA4 runtime, one 144 GB AMD Instinct MI350P at a 450 W power limit generates 273.1 tokens/s for a single request in LocalMaxxing. That is 2.24× our tested stock vLLM + AITER setup using AMD's MXFP4 checkpoint. At 64 concurrent requests, aggregate decode reaches 2,508.6 tokens/s, with 2.19 million KV-cache tokens available on the GPU.

These are our measurements, not independent certification. They compare complete stacks: Paiton's A5 weights and native runtime against AMD Quark MXFP4 weights on vLLM or SGLang. The weights and execution paths differ, so this is not an engine-only comparison or the speed you get just by downloading our quant.

The separate serving test adds 1,024 input tokens followed by exactly 1,024 output tokens. With one request, Paiton starts answering in a median 98 milliseconds. With 16 requests in flight, that becomes 525 ms, compared with 1,782 ms for the tested stock vLLM setup. At 64, Paiton's median is 861 ms, against 2,811 ms.

1K input / 1K output serving sweep. Per-request decode is the reciprocal of median time per output token after the first; it is not total throughput divided by concurrency. Sample p99 describes these recorded requests, not a production guarantee. Chart data.

As load rises, each answer arrives more slowly, but the card does more work in total. Per-request speed affects how quickly an assistant replies; aggregate output determines how many requests can share the hardware.

These are fixed-length requests using source-text windows, not a trial with 64 people chatting. Real traffic brings variable prompts, pauses, tool calls and arrival rates. The table is a measured starting point for capacity planning, not a substitute for testing that traffic.

How much of the original model does A5 retain?

We checked the quantized weights against the original BF16 checkpoint, emulating the arithmetic used during serving. The published model card reports the full method and results.

Held-out weight evaluation against BF16. Full-vocabulary KL is lower-is-closer; top-1 measures next-token agreement, not task accuracy. The INT4-g128 comparison uses our RDNA4 recipe in the same evaluation tool. This is separate from the 130-position engine check later in the article.

These scores cover text and assistant positions from a held-out 294,912-token evaluation set. Top-1 agreement means both models select the same next token at that position. It does not mean 93.78% of answers are correct. KL compares their full next-token probability distributions; zero means identical. Two correct BF16 implementations score about 0.008 in this method.

At longer contexts, negative log-likelihood stays within +0.2% of BF16 across the 8K to 32K document buckets. Needle tests reproduce the reference answers for 8 of 8 prompts at 131K and 4 of 4 at 258K. A continuation after a 254K-token user turn is 1.9% higher in negative log-likelihood than BF16.

These are weight-quality and retrieval checks, not proof of production serving at those lengths. Math, coding, knowledge, multilingual and tool-calling task quality were not benchmarked for this release.

What is Paiton?

Paiton is ElioVP's framework for making AI models run more efficiently on AMD GPUs. It brings together model compilation, hardware-specific kernels, runtime optimization and integration with serving software such as vLLM. The native runtime used in this benchmark is one part of that framework, not the whole product.

AMD described this approach in Introduction to Paiton by ElioVP, published in October 2025. Our Paiton product page covers the current offering, including language models, image generation and video.

What we changed for this CDNA4 release

The release combines our quantization, native CDNA4 kernels and a vLLM server.

First, our own A5 quantization of Qwen3.8-Flash-Next. The routed experts, which hold most of the model's weights, use native FP4 codes with one scale per 128 weights. The main non-expert weights use 8-bit formats. The recipe uses roughly four million calibration tokens and is documented in the public model card.

Second, Paiton's native CDNA4 runtime. It executes the forward pass and sampling with kernels designed for this hardware, including fused paths for decode, prefill and expert computation. Asynchronous scheduling overlaps host preparation with GPU execution.

Third, the regular vLLM serving interface. The Paiton plugin connects the runtime to vLLM's OpenAI-compatible server. Existing clients can keep using that API.

Speculative decoding helps at low concurrency. A fine-tuned MTP drafter proposes three tokens, and the main model checks them together. Above four running requests, this release turns drafting off because its verification overhead outweighs the saving. That tradeoff becomes visible in the results below.

“One GPU” does not mean every tensor lives in GPU memory. Approximately 48 GiB of n-gram embeddings live in host RAM, with the required rows gathered during execution. This, alongside the smaller expert weights, leaves more GPU memory for serving state.

Faster decode across the tested range

LocalMaxxing's lmx speed test measures generation from an empty context, separately from prompt processing. Paiton leads every tested concurrency level in this benchmark, from one request to 64.

Aggregate output tokens/s; higher is better. One MI350P at a configured 450 W limit. Paiton uses A5; the other stacks use AMD Quark MXFP4. Results and provenance.

SGLang uses version 0.5.22 with AITER; vLLM uses 0.32.0 with AITER. The SGLang MTP profiles did not accommodate a configuration for 32 or more running requests in our setup, so those measurements use a separately labelled non-MTP profile. These are tested configurations, not universal limits for either engine.

At one request, the advantage over the fastest alternative is 2.24×. At 16 it is 23%; at 64, 15%. The gain narrows as batching makes the baseline more efficient, but remains present across this matrix.

Reading a long prompt sooner

Prefill is the work before generation: reading the input and building the state needed to answer. Here, Paiton is almost twice as fast as the selected stock vLLM baseline. SGLang is much closer, especially on the longer prompt.

Single-prompt input tokens/s at nominal 8K and 64K depths; higher is better. Actual tokenized lengths are approximately 8,200 and 64,100 tokens. These are separate from decode throughput.

For the approximately 64K-token prompt, measured time to first token is 4.49 seconds with Paiton, 4.64 seconds with SGLang MTP16, and 8.88 seconds with vLLM. Against the best alternative, the prefill advantage is 13% at 8K and 3% at 64K. The larger, roughly 2× gain is specifically against the selected vLLM baseline.

About 2.2 million KV-cache tokens on one GPU

The KV cache is working memory for the tokens a model is processing. More capacity can accommodate longer requests or more requests, subject to the engine's other memory and scheduling requirements.

Runtime-reported GPU KV-cache capacity, not additional physical memory or measured simultaneous long-conversation capacity.

Paiton reports 22% more token capacity than vLLM and approximately 6× the default eight-slot SGLang MTP configuration. Increasing SGLang's running-request allowance reserves more per-request state, leaving less space for the KV cache.

Here, 32K means 32,768 tokens. The last column simply divides capacity by that number and rounds down. It does not establish that 66 conversations can run simultaneously: Paiton is configured for 64 running sequences, generated tokens need room, and other per-request state also matters. We did not benchmark 64 simultaneous 32K prompts.

All stacks were configured with a 262,144-token context limit. A configured limit is not evidence that every configuration can serve a request that large. In particular, SGLang's 16-slot MTP cache is smaller than that limit.

Load changes the result

The serving sweep adds input processing and longer outputs. We tested three shapes: 1K / 1K, 1K / 8K, and 8K / 1K input/output tokens, at one to 64 concurrent requests. Here K means 1,024 tokens.

Aggregate output versus per-request decode in the serving sweep. These are different measurements from LocalMaxxing. Full latency, throughput and power records are included in the downloads.

For one request producing an 8K-token answer, Paiton's median completion time is 22.6 seconds, versus 43.1 seconds for SGLang and 67.0 seconds for vLLM. At 64 concurrent requests in that shape, total output is 1,937 tok/s, against 1,737 tok/s for vLLM.

Long inputs put more pressure on scheduling. At 64 requests with 8K input / 1K output, Paiton's median first-token time rises to 9.6 seconds. That is lower than vLLM's 37.6 seconds, but it is still a substantial wait. A high aggregate score does not erase that latency.

Energy: more output per GPU watt

A power limit is not measured consumption. During the 1K / 1K sweep, Paiton's mean GPU reading rises from approximately 308 W at one request to 445 W at 64. Combining that reading with throughput gives the following efficiency result.

Output tokens/s per kW of measured GPU power; higher is better. Excludes CPU, system RAM, power-supply losses and cooling.

Paiton delivers 2.60× the output per unit of GPU energy of the vLLM baseline at one request and 20% more at 64. SGLang's 16-slot profile is 5.6% more efficient than Paiton at 16 requests. That exception matters when choosing a target operating load.

These are GPU-only figures from AMD SMI, not wall-plug efficiency or a whole-server cost calculation. Host energy is particularly relevant here because part of the model resides in system memory.

A separate accuracy check of the engines

The weight evaluation at the start of this article tests the quantization. We also checked the execution paths, comparing next-token distributions at 130 teacher-forced positions following 8,128-token prompts. This uses a different metric: the reference's top 1,024 tokens plus the remaining probability mass as one tail term.

Narrow next-token diagnostic, not answer accuracy. Lower KL is closer to BF16. Paiton and AMD MXFP4 run on MI350P; the BF16 control runs on MI355X. The harness paths differ.

Paiton is closer to the reference on this small sample. It does not mean that 93.1% of answers are correct, nor that one setup is five times more accurate on user tasks.

Paiton is measured through its engine test path with MTP off; the other rows use a vLLM API client. Short-prompt rows were excluded because the BF16 control showed inconsistencies there. The comparison also combines checkpoint and kernel differences. Published logs support these selected rows, but the bundle does not contain the full reference distributions or evaluation client needed to rerun this diagnostic independently.

From RDNA4 on Radeon to CDNA4 on Instinct

This work builds on our Radeon releases. With 3-bit Qwen3.8 27B on one Radeon AI PRO R9700, Paiton reached 492.1 output tokens/s across eight concurrent requests and 19.9% faster weighted decode than the preceding MXFP4 release. That article covers the settings and accuracy tradeoffs of the comparison.

Our later Flash Next update on two R9700s improved 3-bit accuracy and added prefix caching with MTP. Its optional native worker reached 258.7 tokens/s weighted single-request decode. In a separate test, a repeated 200K-token conversation started responding in 0.68 seconds on a cache hit, instead of 25.81 seconds cold. That requires the unchanged conversation prefix to remain cached.

The 27B W3Rot weights and Flash Next W3A8 weights recorded more than 2,200 combined Hugging Face downloads in the preceding 30 days, checked on 30 October 2026. That counts downloads, not unique users.

The aim is the same across these releases: get more from AMD hardware by optimizing the model and its execution together. The R9700 and MI350P figures come from separate tests with different models and settings; this is not a direct comparison between the cards.

A look at the benchmark output

Final frame of the original terminal capture, unchanged. Click to enlarge. This separate run uses 20-second cells, not the headline benchmark's 30-second cells. The source recording and timing logs remain in the accuracy and recording download.

Inspect the measurements

The evidence is split into manageable downloads, with raw matrices, per-request records, commands and checksums. The supplied SHA-256 manifest's 177 entries were checked successfully before publication. That verifies file integrity, not independent experimental validation.

Why these results are not on LocalMaxxing: we would like to publish the numbers there as well, but its GPU detection did not recognize the MI350P in this run, which prevented publication. The report's CPU_ONLY label is a detection limitation, not how the benchmark ran: the model ran on the MI350P. We are sharing the raw results here so readers can inspect them directly.

• Paiton benchmark evidence
• vLLM benchmark evidence
• SGLang benchmark evidence
• Accuracy logs and terminal recording evidence
• Methods, commands, serving client and checksum manifest
• Chart data and selected source paths

Download the stock baseline container

The public CDNA4 container packages stock vLLM 0.32 + AITER, configured to serve AMD's Quark MXFP4 checkpoint. Pull the published image by digest:

docker pull ghcr.io/eliovp/paiton-vllm-plugin@sha256:20b729e6c7b0f2de101fd27d080d91291a7d37c4002312fe9ee5aeb5ff131b19

This downloads the baseline container; it does not start a server or supply the proprietary engine. The methods download contains the original benchmark commands. This image is not a serving path for the A5 weights and will not reproduce the Paiton column.

Measurement settings

SGLang profiles. The serving sweep uses MTP with 16 slots for 1K / 1K and eight slots for the longer shapes, selected by peak throughput for each shape. At 32 and 64 requests, it uses the non-MTP configuration.

Common conditions. One MI350P, a configured 450 W GPU limit, fans at full speed, one benchmark at a time and the client on the server host.

LocalMaxxing. lmx 0.1.54 with llm-inference-bench 0.7.7, 30-second cells, empty-context decode capped at 8,192 output tokens, and nominal 8K/64K prefill. After warmup, the main configurations run three matrices; we select the whole run with median single-request decode. Comparator c32/c64 measurements are separate single runs, not three-run medians.

Serving sweep. A streaming client uses greedy generation, ignores EOS and gives each request a distinct source-text window. After two short-output warmup rounds, it starts up to max(8, 2 × concurrency) requests, stopping new starts after 100 seconds and allowing in-flight requests to finish. Cells contain 2 to 128 measured requests, all with zero errors and no short outputs. Small-sample p99 values are close to the observed maximum, not reliable population percentiles.

Selection detail. The displayed vLLM 1K / 1K c32 result uses the earlier recorded cell, 1,169.2 tok/s; a later repeat produced 1,064.0 tok/s. Both are in the download. Other c32/c64 sweep rows use the later batch. The source paths in the chart data make that choice explicit.

The MI350P results show what the Paiton framework delivers on this CDNA4 setup: its largest speed advantage is at low concurrency, with gains continuing under heavier load. Our next optimization task is to make speculative decoding pay off above four running requests.

Read the public weights and quantization recipe or learn more about Paiton.

Cover: AI-generated editorial illustration of an MI350P, not a product photograph or benchmark capture.

Running a token factory? Get more from your AMD GPUs.

Already serving models, or interested in starting a token factory? We're here to help you increase throughput, lower your cost per token and maximize the return on your AMD hardware.

Let's talk about your token factory.