A 35B MoE at 44.6 tok/s on one 32 GB GPU

One workstation GPU. A 256-token response completed in a median 5.12 seconds. No cloud inference API.

Paiton now runs the public Ornith 1.5 35B A3B MXFP4 checkpoint at an average of 44.63 output tokens per second on one AMD Radeon AI PRO R9700.

The fastest qualified stock vLLM configuration we obtained on the same machine reached 35.13 output tokens per second. That gives Paiton 27.03% more output throughput and reduces modeled time-based cost per generated token by 21.28%.

Our earlier Qwen3.8 benchmark tested three different prompt shapes. This release answers a different question: can a roughly 35B mixture-of-experts model, its speculative draft and a useful serving runtime fit on one 32 GB card, and still feel fast enough for local use?

Benchmark boundary: 256 requested input tokens, 256 output tokens, one active sequence, temperature zero and thinking disabled. Chat templating produced 268 to 270 actual prompt tokens. The headline measures output generation, not prompt-processing speed, long-context coding, tool use or concurrent serving.

A 35B model that actually fits

Ornith 1.5 35B A3B is a mixture-of-experts model. The A3B label means that roughly 3 billion parameters are activated for each token. It does not mean the whole model occupies the memory of a 3B model; the full expert pool still has to be stored.

The tested MXFP4 target is approximately 22.9 GB, while the public DFlash draft adds about 772 MB. Together, they fit inside the R9700's 32 GB of VRAM with the qualified runtime and a reserved KV cache.

The public runtime plugin is Apache-2.0-licensed. The target checkpoint is MIT-licensed, while the DFlash draft is Apache-2.0-licensed. The Paiton compiler itself is not distributed.

The upstream Ornith team targets coding and agentic workflows and publishes 262,144-token serving examples with tool use. That makes it a relevant local target rather than a synthetic capacity demo, but those broader capabilities are not qualified by this package. The published Paiton path is capped at 8,192 tokens, text only, and automatic tool choice is outside this release's tested scope.

That boundary matters. Buyers evaluating local AI usually need to know more than whether a checkpoint can be loaded: what fits, what starts reliably, which API is available and where the qualification ends.

Where the 27% comes from

Most of the uplift is already present before speculative decoding is enabled.

The Paiton target path delivers 23.55% more throughput than stock on its own. DFlash then moves the result from 43.405 to 44.628 tokens per second, adding another 2.82% over Paiton without speculation.

DFlash uses a lightweight block-diffusion draft model to propose several upcoming tokens in parallel. Ornith verifies those proposals, accepts the valid prefix and remains the authority for the generated output.

DFlash provides the final lift, but it is not the source of the full 27%. That is where the public implementation detail stops: Paiton's generated kernels, fusion choices, schedules and runtime mechanics remain proprietary.

It starts 20 ms later but finishes 2.66 seconds sooner

Generation speed is only half of the user experience. A local user also notices how long the model takes to begin and how long the complete answer takes to arrive.

For this workload, median time to first token increased from approximately 290 ms to 310 ms. Once generation started, the faster stream more than recovered that 20 ms delay:

• Median time per output token fell from 27.46 ms to 18.86 ms, a 31.3% reduction.
• Median end-to-end time fell from approximately 7.78 seconds to 5.12 seconds.
• The complete 256-token response finished about 2.66 seconds sooner.

This is the honest latency story: a marginally slower first token and a materially faster complete answer. With speculative decoding, TPOT is a streaming-cadence metric rather than a direct inverse of complete-request throughput because accepted tokens can arrive in bursts. Complete-request throughput and end-to-end time remain the primary measures here.

Workloads dominated by large prompt ingestion need separate prompt-processing measurements; this article does not infer those from decode throughput.

More output from the same active hour

At the measured rates, stock vLLM needs about 7.91 active hours to generate one million output tokens. Paiton with DFlash needs about 6.22 hours.

That is why a 27.03% throughput increase becomes a 21.28% cost reduction: cost per token is the inverse of throughput.

Using the same illustrative ownership model as the Qwen3.8 article, with 5,000 productive inference hours, equal 450 W whole-system draw, purchase prices of $1,299 or €1,749, and electricity at $0.17 or €0.2558 per kWh, the result is:

The same card therefore produces approximately 171 million additional output tokens over 5,000 productive hours in this model.

This is a time-and-ownership model, not a measured wall-power efficiency claim. It assumes equal system draw and excludes the host computer, idle time, cooling, maintenance, financing, tax and residual value.

Why the stock comparison is credible

We did not compare Paiton with a default or deliberately weak installation. The selected stock cell was the fastest stable result obtained after testing the available execution, graph, attention, recurrent and mixture-of-experts settings.

It used pinned vanilla ROCm vLLM, O2 compilation, graph capture at batch size one, Triton attention, Triton GDN decode, the unfused Triton MoE backend, DFlash disabled, and the same supported AUTO performance level with the COMPUTE power profile.

Both sides used the same GPU, checkpoint, tokenizer, pinned vLLM revision, request data, random seed, output length, concurrency and GPU profile. The headline gives stock its fastest qualified cell and compares it with the mean of two fresh-server Paiton runs.

The stock cell completed all 12 measured requests. The two Paiton runs completed all 24 and returned every requested output token. A deterministic natural-language request before and after the Paiton runs remained readable, stopped naturally, produced finite log probabilities and was byte-identical.

That last check is a serving sanity gate, not an academic accuracy evaluation, a coding benchmark or proof of quality parity with the upstream model. Test the quantized checkpoint on your own prompts and acceptance criteria before deployment.

The public Ornith benchmark record contains the publication workload, stock settings and reproduction command.

Run Ornith 1.5 locally

This is a checked Linux and Docker path for one Radeon AI PRO R9700 with ROCm 7.14. Start the server and open the included terminal chat with one command:

git clone --depth 1 https://github.com/Eliovp-BV/paiton-vllm-plugin.git \
  && cd paiton-vllm-plugin \
  && ./models/Ornith-1.5/serve-docker.sh --chat

The first start downloads the pinned public target and DFlash draft, then creates a ready-to-run copy in a persistent Docker volume. The same server exposes an OpenAI-compatible endpoint at:

http://127.0.0.1:8000/v1/chat/completions

Use model name ornith. The public package guide includes the direct Docker command, the DFlash-off option and the exact operating scope.

The image contains the runtime plugin and compiled Paiton artifact. It contains no model weights and no Paiton compiler source.

Tested configuration

The package fails closed outside its supported GPU, checkpoint and serving contract. Additional requests queue instead of entering an unqualified concurrent path.

From a workstation result to production economics

The Qwen3.8 release showed Paiton improving three different prompt shapes on the R9700. Ornith adds a distinct proof point: a 35B MoE target and speculative draft fit on one 32 GB card, serve through a familiar API and generate at 44.63 output tok/s.

For local users, that means a faster private model endpoint without a hosted per-token dependency. For operators running AMD Instinct CDNA infrastructure, the leverage is larger: every percentage point compounds across accelerators, models and billions of generated tokens.

Running an AMD token factory? Bring us the model, traffic shape and current baseline. We will show where throughput, latency and cost per generated token can move.