Qwen3.8 GGUF in vLLM: Faster Responses on One Radeon

Original NEO CODER MAX weights. Paiton-compiled execution. Up to 6.4% lower warm request latency than our matched llama.cpp baseline on a Radeon AI PRO R9700.¹

Original GGUF weights. Actual vLLM serving. Paiton-compiled execution on one Radeon AI PRO R9700.²

DavidAU’s Qwen3.8 NEO CODER MAX has attracted plenty of attention. On September 14, 2026, its Hugging Face repository showed 875,703 downloads in the preceding month. That is a repository download count, not a count of unique users, but it is a useful indication of the interest around this fine-tune.³⁴

We wanted to answer a different question from another model review:

Can you keep the original GGUF fine-tune, serve it through vLLM, and still compete with llama.cpp on a single Radeon?

For this tested model and configuration, yes. Paiton runs the selected Q4_K_M checkpoint through native AMD execution inside vLLM, rather than routing requests to a separate llama.cpp server.²

The fine-tune is the author’s. The execution path is ours.

GGUF support is not the same as optimized GGUF execution

vLLM already supports GGUF. Its current documentation nevertheless describes that support as “highly experimental and under-optimized,” with potential incompatibilities with other features. Support now lives in the upstream vllm-gguf-plugin.⁵

That plugin already documents several model families, including related Qwen vision-language models. This is not an announcement that GGUF has become possible in vLLM for the first time.⁶

Our focus is narrower: make this particular GGUF fine-tune run efficiently on AMD hardware without making users switch to a different checkpoint or a different serving framework.

GGUF is a container for weights and metadata, not a requirement to use one inference engine.⁷ And llama.cpp already provides its own OpenAI-compatible server.⁸ An API wrapper alone would not be the achievement here.

The distinction is actual vLLM serving, backed by Paiton’s compiled native execution.

Keep the fine-tune. Change the execution path.

The selected checkpoint is the author’s original mixed Q4_K_M GGUF, including its higher-precision tensors and BF16 output head. We have not replaced it with base Qwen or converted it into a new AWQ checkpoint.⁹

Paiton provides the native language and vision execution. vLLM retains the model-loading integration, scheduling, sampling and streaming interface. The compiler remains proprietary; the public release contains the runtime artifacts needed to use it.²

This is an optimization inside an existing serving stack, not a requirement to adopt another inference server.

The public integration at a glance: keep the author’s GGUF, serve through vLLM, and execute with Paiton on the R9700. This is an integration overview, not a disclosure of compiler internals.²

Faster complete responses, not just a faster kernel

The comparison below measures complete streaming HTTP requests, with exactly 128 generated tokens. Both engines ran serially on the same idle R9700. Each workload had one warmup and five measured requests.¹⁰

Complete streaming requests, not kernel-only timings. Both engines generate exactly 128 tokens. The 4,096-token result is near-parity; these five-request samples do not establish statistical significance.¹⁰

These are warm-request measurements, not first-install or cold-start timings.¹⁰

The 4,096-token result is best described as near-parity. A 76-millisecond median difference in a small sample is not a basis for a sweeping performance claim.

Nor does lower complete-request latency mean every part of every request is faster: llama.cpp still wins some single-output, prefill-dominated tests.²

The useful result is that this GGUF can stay inside vLLM while delivering competitive latency. In these measured full-response cases, it was also lower.

What was held constant?

The comparison used the same pinned GGUF, source tokenizer, 8,192-token context, 2,048-token prefill chunks and BF16 KV cache. Both engines had one active sequence, MTP and prefix caching disabled, and greedy sampling with fixed token counts. llama.cpp was an unmodified HIP build, not a CPU fallback.¹⁰

The fixed-length prompts are synthetic timing workloads, not a coding-productivity benchmark. Shorter reasoning or earlier stopping is not being counted as faster execution.¹⁰

Image input works too

This release also accepts one PNG or JPEG image through the chat-completions interface. Its image encoder runs through the native Paiton path; image embeddings and generated text share the same context budget.¹¹

For a 1,024 × 1,024 image and 128 output tokens, median complete-request time was 6.279 seconds with Paiton versus 6.455 seconds with llama.cpp, about 2.7% lower. That includes image processing, language prefill, generation and serving overhead.²

Two qualified image sizes, each followed by 128 generated tokens. Timing includes image encoding, language prefill, generation and serving overhead. This measures image understanding, not image generation.¹⁰

This is single-image inference, not video or unrestricted multi-image support.¹¹

Same weights does not mean identical arithmetic

Keeping a GGUF checkpoint unchanged does not make two runtimes numerically identical. The qualified execution profile uses different activation arithmetic from the older FP32 reference path; the weight values remain unchanged.¹²

The published checks include 10 of 11 fixed text tasks passed, with the same failed task as llama.cpp, and five image fixtures passed. The final prefill optimization also matched 31,784,960 compared logits against the preceding qualified Paiton profile, not against every other engine.¹²¹³

Those are useful release checks, not proof of unchanged capability on every coding task, conversation or image. We keep the numerical comparisons and their reference profiles explicit rather than describing the release as universally bit-identical.

The deployment profile

The published v1.1.0 release is qualified for the following configuration:¹²¹¹

The model’s name includes MTP, but these results do not use speculative MTP decoding. Likewise, queued clients should not be confused with qualified multi-sequence GPU batching.

This release targets the 32 GB R9700. It is not a promise of support for a smaller GPU, another GGUF quantization, longer contexts or high-concurrency serving.¹

Run the published release

The qualified local serving profile. The API panel is illustrative, not an application screenshot or benchmark trace. Compiler access is not needed to use the published runtime.¹²

On a Linux system with an R9700, a working AMD driver and Docker GPU access, clone the public helper repository and launch the pinned v1.1.0 image:¹¹²

git clone --depth 1 https://github.com/Eliovp-BV/paiton-vllm-plugin.git
cd paiton-vllm-plugin

PAITON_NEO_IMAGE=ghcr.io/eliovp/paiton-vllm-plugin@sha256:534287969135f581744ae481b578599468b0bf7ac9a4051b0941500e4c18da4d \
  ./models/Qwen3.8-NEO-CODER-MAX/serve-docker.sh

The first start downloads approximately 19.43 GB of pinned language and projector files. Later starts reuse the cache and verify the hashes. No compiler checkout is required. An existing copy of the exact pinned GGUF can also be mounted and reused using the model guide’s instructions.¹

After the server is ready, send a request from another terminal:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen38-neo",
    "messages": [
      {
        "role": "user",
        "content": "Write a Python function that removes duplicate integers while preserving their original order."
      }
    ],
    "temperature": 0,
    "max_tokens": 256,
    "stream": true,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

The example explicitly disables thinking through the retained source template. It is an interactive example, not the fixed 128-output-token benchmark request.¹

More useful performance, without another serving stack

The point is not that every model needs a new engine. It is that a useful fine-tune should not need to lose its identity to fit an optimized deployment path.

For this release, the original GGUF and vLLM stay in place. Paiton changes the execution underneath them.²

Keep the fine-tune. Keep vLLM. Get more from the Radeon you already have.

Start with the public model guide and its benchmark report. For AMD inference workloads beyond this qualified profile, talk to us about Paiton.¹