Paiton: 21% More Qwen3.8 Throughput on Radeon AI PRO R9700

One GPU. The same public source checkpoint. The same benchmark requests. More output.

Paiton served AMD’s public Qwen3.8-27B-Quark-Qronos-INT4-W4A16 at 39.77 output tokens per second in our batch-one interactive benchmark on a single 32 GB AMD Radeon AI PRO R9700.

Our fastest qualified stock vLLM configuration on the same system reached 32.86 output tokens per second. That gives Paiton 21.0% more output per active inference hour from the same GPU.

The advantage widened to 54.3% in the coding workload and reached 26.8% with a 4,096-token input.

Throughput across three workloads

These are single-user serving tests with one request at a time, one active model sequence, temperature zero, thinking disabled and fixed random inputs.

Each workload was run twice from a fresh server. Every run used three warmups followed by 12 measured requests, and the table reports the mean of both runs. All six Paiton runs completed 12 of 12 requests with zero failures and returned the full requested output length.

Chat templating increased the actual prompt lengths to 268–270, 1,036–1,038 and 4,108–4,110 tokens respectively.

Latency: faster streaming, mixed first-token results

Output throughput is only part of the user experience, so we are publishing first-token and streaming latency as well.

Paiton reduced median time per output token in every workload:

• Interactive: 30 ms to 24 ms
• Coding: 39 ms to 25 ms
• Long context: 37 ms to 28 ms

Time to first token was workload-dependent. Median interactive TTFT increased from 258 ms to 290 ms. Coding improved slightly from 757 ms to 737 ms, while long-context TTFT fell from 3,337 ms to 2,984 ms.

The result is a clear streaming-throughput gain, not a claim that every latency metric improves in every scenario.

What the gain means for cost

For owned inference hardware, throughput determines how much output a fixed hour of GPU time can produce. At equal hourly system cost, a 21.0% throughput gain gives 21.0% more tokens for the same active runtime budget.

Because cost per token is the inverse of throughput, the corresponding modeled reduction in time-based cost per million output tokens is 17.4%.

The examples below amortize the GPU over 5,000 productive inference hours and assume an equal 450 W whole-system draw for both runtimes.

At the measured interactive rates, the same card produces roughly 124 million additional output tokens over 5,000 productive hours.

For the same modeled $100 ownership-and-electricity budget, Paiton produces about 42.6 million tokens instead of 35.2 million. In the European example, €100 produces about 30.8 million tokens instead of 25.4 million.

These figures are a transparent time-based ownership model, not a measured energy-efficiency claim. They exclude the host computer, idle time, cooling, maintenance, financing, tax and residual value. Substitute your own purchase price, electricity rate and productive lifetime as needed. The 17.4% relative reduction remains the same when both runtimes carry the same hourly cost.

A stock baseline worth comparing against

We did not compare Paiton with an eager or default installation and call the result an optimization win. We spent significant time qualifying the fastest stable stock configuration we could obtain on this system.

The stock baseline used compiled vLLM execution at optimization level 2, full-and-piecewise HIP graph capture, stock RDNA hybrid W4A16 kernels, vLLM’s Triton GDN implementation, the ROCm attention backend and the R9700’s supported high clock policy.

Both sides used the same GPU, ROCm host, model revision, tokenizer, pinned vLLM revision, request data, random seed, output lengths, concurrency and supported GPU clock policy. We retained the complete result files and averaged two fresh-server runs per workload rather than selecting one favorable terminal result.

The stock logs confirm compiled execution and graph capture. They load no Paiton model artifact or Paiton compute kernel.

What Paiton does

At a high level, Paiton builds a qualified, model- and hardware-specific execution path and integrates it with the serving runtime. The original AMD checkpoint remains unchanged on disk.

Paiton assembles target-specific runtime artifacts during model loading. The optimized path should therefore not be assumed to produce output that is bit-for-bit identical to stock W4 execution.

That is where the implementation detail stops. The model-specific kernel, fusion, scheduling and runtime design are closed-source Paiton IP. What we do publish is the part customers can validate: the supported target, benchmark method, throughput, latency, quality gates, package identity and operating scope.

Quality checks

We ran a deterministic 12-case suite covering instruction following, arithmetic, algebra, logic, factual recall, translation, structured JSON, Python, SQL, summarization, format constraints and modular reasoning.

All 12 cases passed. Repeated temperature-zero responses were byte-identical, and all recorded log probabilities were finite.

This is a practical quality gate for the stated local-chat scope. It is not a replacement for a full academic accuracy evaluation, and it should not be read as a claim of quality parity across every task or prompt distribution.

Run the local chatbot

The public image contains the Paiton runtime plugin and compiled runtime artifacts. It downloads the original 19.9 GB AMD checkpoint directly from Hugging Face into a persistent Docker volume; Paiton does not redistribute the model weights.

docker run -d \
  --name paiton-qwen38 \
  --device /dev/kfd \
  --device /dev/dri \
  --group-add video \
  --ipc=host \
  --network host \
  --mount type=volume,src=paiton-qwen38-cache,dst=/models/cache \
  ghcr.io/eliovp/paiton-vllm-plugin@sha256:c56baf54aca1ad229829c1de26e8792806608e65ee9d06ee210b79cd49f70bc9

The first start downloads the checkpoint. With the weights already cached, model assembly takes roughly 10 to 12 minutes on our R9700. Follow startup with:

docker logs -f paiton-qwen38

Once the log reports that the application is ready, open the included streaming chat:

docker exec -it paiton-qwen38 paiton-chat

Use /reset to clear the conversation and /quit to exit. Thinking is off by default for responsive local conversation; pass --thinking when required.

The same server exposes an OpenAI-compatible endpoint at http://127.0.0.1:8000/v1/chat/completions using model name qwen38.

Reproduce the interactive benchmark

After the server is ready:

docker exec paiton-qwen38 sh -lc '
MODEL_DIR="$(find /tmp -maxdepth 2 -type d \
  -path "/tmp/paiton-qwen38-reused-*/model" -print -quit)"
exec vllm bench serve \
  --backend openai-chat \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/chat/completions \
  --model qwen38 \
  --tokenizer "$MODEL_DIR" \
  --dataset-name random \
  --seed 42 \
  --num-warmups 3 \
  --num-prompts 12 \
  --random-input-len 256 \
  --random-output-len 256 \
  --random-range-ratio 0 \
  --random-prefix-len 0 \
  --request-rate inf \
  --max-concurrency 1 \
  --temperature 0 \
  --ignore-eos \
  --extra-body '\''{"chat_template_kwargs":{"enable_thinking":false}}'\'' \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,95,99 \
  --disable-tqdm
'

One run is useful as a local check, but consumer GPU power management can make a cold run slower. Run a short warmup first and confirm that the memory clock has reached its normal loaded state before recording a comparison.

Tested configuration

The package fails closed on an unsupported GPU identity, architecture, runtime contract, artifact checksum, model contract or serving configuration. Multiple simultaneous requests queue.

This article does not claim support for other GPUs, ROCm versions, tensor-parallel configurations, multimodal input or production continuous batching.

The practical result

On one Radeon AI PRO R9700, Paiton increased interactive Qwen3.8 output throughput from 32.86 to 39.77 tokens per second. That translates into 21.0% more output per active hour, a 17.4% reduction in modeled time-based cost per million output tokens, and roughly 124 million additional tokens over 5,000 productive hours at the measured interactive rate.

The coding and long-context results show that the gain is not limited to one prompt shape. The latency data also makes the boundary clear: streaming became faster across all three workloads, while first-token latency remained workload-dependent.

Optimize your inference workload with Paiton

This is a qualified Radeon AI PRO R9700 result for one model and one serving scope. Paiton’s broader commercial work also targets AMD Instinct CDNA accelerators for larger inference deployments, where throughput, fleet utilization and cost per generated token compound across the infrastructure.

We benchmark the real workload, identify the runtime bottleneck, build the qualified AMD path and measure the delivered result against an agreed baseline.

Learn more about Paiton.