[{"data":1,"prerenderedAt":20002},["ShallowReactive",2],{"blog-post-\u002Fblog\u002Ffurther-optimizing-amd-powered-inference-with-paiton":3,"blog-posts-sidebar":709},{"id":4,"title":5,"body":6,"categories":687,"date":697,"description":698,"extension":699,"image":700,"meta":701,"navigation":702,"originalUrl":703,"path":704,"seo":705,"slug":706,"stem":707,"__hash__":708},"blog\u002Fblog\u002Ffurther-optimizing-amd-powered-inference-with-paiton.md","Further Optimizing AMD-Powered Inference with Paiton",{"type":7,"value":8,"toc":672},"minimark",[9,17,29,63,70,73,79,85,105,107,113,156,170,172,178,277,280,287,294,297,521,527,547,550,556,563,568,574,598,600,606,626,636,638,644,647,653,659,661,667],[10,11,13],"h3",{"id":12},"executive-summary",[14,15,16],"strong",{},"Executive Summary",[18,19,20,21,24,25,28],"p",{},"If you’ve followed our journey so far, you’ll know that Paiton is laser-focused on AMD-centric inference optimization. Our latest work takes ",[14,22,23],{},"DeepSeek R1 Distill Llama 8B"," to the next level, delivering 10–15% higher throughput, improved time-to-first-token (TTFT), and more stable performance at ",[14,26,27],{},"lower"," batch sizes, an area that previously needed a boost.",[30,31,32,47,57],"ul",{},[33,34,35,38,39,42,43,46],"li",{},[14,36,37],{},"Core Gain",": Up to ",[14,40,41],{},"10–15%"," improvement in Requests\u002Fs and token throughput vs. stock vLLM at ",[14,44,45],{},"batch size 32",".",[33,48,49,52,53,56],{},[14,50,51],{},"Lower TTFT",": ~",[14,54,55],{},"12.95%"," faster mean TTFT, critical for user-facing scenarios like chatbots.",[33,58,59,62],{},[14,60,61],{},"New Benchmark Tool",": A custom approach for orchestrating and measuring performance across multiple AMD MI300 GPUs, ensuring real-world reliability.",[18,64,65,66,69],{},"In short, ",[14,67,68],{},"Paiton"," further cements its ability to exploit AMD hardware’s raw power, bridging the performance gap in a landscape that’s sometimes overshadowed by NVIDIA solutions.",[71,72],"hr",{},[10,74,76],{"id":75},"deepseek-r1-distill-llama-8b-optimization-the-next-chapter-with-paiton-amd",[14,77,78],{},"DeepSeek R1 Distill Llama 8B Optimization: The Next Chapter with Paiton & AMD",[10,80,82],{"id":81},"why-revisit-deepseek-r1-distill-llama-8b",[14,83,84],{},"Why Revisit DeepSeek R1 Distill Llama 8B?",[30,86,87,93,99],{},[33,88,89,92],{},[14,90,91],{},"Targeted Gains",": Our initial optimizations were strong at higher batch sizes, but lower batch sizes showed room for improvement.",[33,94,95,98],{},[14,96,97],{},"Broader Real-World Impact",": Many users run medium or small batches in production (e.g., 16–128), so we wanted to lift performance across the board.",[33,100,101,104],{},[14,102,103],{},"Incremental Advances",": Building on the foundation we shared in our earlier A First Look at Paiton in Action blog post, we’re now honing those kernels for an even sharper competitive edge.",[71,106],{},[10,108,110],{"id":109},"how-we-pushed-efficiency-even-further",[14,111,112],{},"How We Pushed Efficiency Even Further",[114,115,116,133,146],"ol",{},[33,117,118,121],{},[14,119,120],{},"GEMM Optimizations",[30,122,123,130],{},[33,124,125,126,129],{},"We ",[14,127,128],{},"streamlined matrix multiplications"," (GEMM) to reduce overhead, crucial for transformers.",[33,131,132],{},"Coalesced memory accesses, minimized kernel-launch latencies, and leveraged AMD’s HBM more effectively.",[33,134,135,138],{},[14,136,137],{},"Refined Kernel Execution",[30,139,140,143],{},[33,141,142],{},"By analyzing concurrency “hotspots,” we merged or reordered certain GPU kernels, ensuring minimal idle cycles.",[33,144,145],{},"Focused especially on batch size 32, a common real-world concurrency setting for LLM inference.",[33,147,148,151],{},[14,149,150],{},"Lower Batch Size Focus",[30,152,153],{},[33,154,155],{},"We identified inefficiencies at small batch sizes; new code paths handle them more gracefully, improving TTFT and overall throughput.",[18,157,158,161,162,169],{},[14,159,160],{},"Pro Tip",": For an overview of the broader improvements we’ve carried out, see our ",[163,164,168],"a",{"href":165,"rel":166},"https:\u002F\u002Feliovp.com\u002Fblog\u002Fai-model-optimization-with-paiton",[167],"nofollow","AI Model Optimization with Paiton"," blog post.",[71,171],{},[10,173,175],{"id":174},"benchmarking-setup-methodology",[14,176,177],{},"Benchmarking Setup & Methodology",[114,179,180,199,253],{},[33,181,182,185],{},[14,183,184],{},"Hardware Environment",[30,186,187,193],{},[33,188,189,192],{},[14,190,191],{},"AMD MI300X"," GPUs in our research data center.",[33,194,195,198],{},[14,196,197],{},"Port"," and concurrency controlled via vLLM’s online mode to mimic real serving scenarios.",[33,200,201,204],{},[14,202,203],{},"Serving & Benchmark Tools",[30,205,206,227,232],{},[33,207,208,211,212],{},[14,209,210],{},"vLLM (stock)"," with:\n",[30,213,214,217,220],{},[33,215,216],{},"CUDA Graph turned on (except we disable Triton for Flash Attention).",[33,218,219],{},"num_sched_prompts increased for concurrency.",[33,221,222,226],{},[223,224,225],"code",{},"VLLM_USE_TRITON_FLASH_ATTN=0"," for improved stability\u002Fperf reasons.",[33,228,229,231],{},[14,230,68],{}," injected into vLLM for direct apples-to-apples comparison.",[33,233,234,237,238],{},[14,235,236],{},"Our New Benchmarking Tool",":\n",[30,239,240,247,250],{},[33,241,242,243,246],{},"Built on ",[14,244,245],{},"dstack"," for orchestrating multi-developer usage.",[33,248,249],{},"Checks GPU availability, preventing resource conflicts.",[33,251,252],{},"Outputs a standardized table of metrics (Requests\u002Fs, TTFT, E2E Latency, etc.).",[33,254,255,237,258],{},[14,256,257],{},"Primary Metric",[30,259,260,271],{},[33,261,262,263,266,267,270],{},"We emphasize ",[14,264,265],{},"Requests\u002Fs"," and ",[14,268,269],{},"Output Tokens\u002Fs",", key metrics for users running real-time or near-real-time inference.",[33,272,273,276],{},[14,274,275],{},"Time-to-First-Token"," (TTFT) is also critical for user experience.",[18,278,279],{},"Here, we’ll showcase a video preview of our benchmark tool in action. The frontend isn’t exactly a masterpiece.. because, well, it was made by our backend engineer.. but it gets the job done!",[18,281,282],{},[283,284],"img",{"alt":285,"src":286},"","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fbenchmarking-setup-methodology.jpg",[288,289,291],"h2",{"id":290},"key-results-stock-vllm-vs-paiton",[14,292,293],{},"Key Results: Stock vLLM vs. Paiton",[18,295,296],{},"Below is a condensed version of our benchmark table at batch size = 32, a prime environment for many real-world deployments:",[298,299,300,315],"table",{},[301,302,303],"thead",{},[304,305,306,309,311,313],"tr",{},[307,308],"th",{},[307,310],{},[307,312],{},[307,314],{},[316,317,318,342,355,369,383,397,410,423,437,451,465,479,493,507],"tbody",{},[304,319,320,326,332,337],{},[321,322,323],"td",{},[14,324,325],{},"Metric",[321,327,328,331],{},[14,329,330],{},"Paiton-vLLM","DeepSeek-R1-Distill-Llama-8B",[321,333,334,331],{},[14,335,336],{},"Stock-vLLM",[321,338,339],{},[14,340,341],{},"% Improvement",[304,343,344,347,350,352],{},[321,345,346],{},"Successful Requests",[321,348,349],{},"32",[321,351,349],{},[321,353,354],{},"0.00%",[304,356,357,360,363,366],{},[321,358,359],{},"Duration (s)",[321,361,362],{},"4.91",[321,364,365],{},"5.43",[321,367,368],{},"9.58%",[304,370,371,374,377,380],{},[321,372,373],{},"Request Throughput (req\u002Fs)",[321,375,376],{},"6.51",[321,378,379],{},"5.89",[321,381,382],{},"10.53%",[304,384,385,388,391,394],{},[321,386,387],{},"Output Token Throughput (tok\u002Fs)",[321,389,390],{},"1342.99",[321,392,393],{},"1214.81",[321,395,396],{},"10.55%",[304,398,399,402,405,408],{},[321,400,401],{},"Total Token Throughput (tok\u002Fs)",[321,403,404],{},"2655.85",[321,406,407],{},"2402.36",[321,409,396],{},[304,411,412,415,418,421],{},[321,413,414],{},"Mean TTFT (ms)",[321,416,417],{},"164.74",[321,419,420],{},"189.25",[321,422,55],{},[304,424,425,428,431,434],{},[321,426,427],{},"p99 TTFT (ms)",[321,429,430],{},"234.75",[321,432,433],{},"247.38",[321,435,436],{},"5.11%",[304,438,439,442,445,448],{},[321,440,441],{},"Mean TPOT (ms)",[321,443,444],{},"8.48",[321,446,447],{},"9.88",[321,449,450],{},"14.17%",[304,452,453,456,459,462],{},[321,454,455],{},"p99 TPOT",[321,457,458],{},"30.16",[321,460,461],{},"30.81",[321,463,464],{},"2.11%",[304,466,467,470,473,476],{},[321,468,469],{},"Mean ITL (ms)",[321,471,472],{},"6.61",[321,474,475],{},"7.30",[321,477,478],{},"9.45%",[304,480,481,484,487,490],{},[321,482,483],{},"p99 ITL",[321,485,486],{},"17.42",[321,488,489],{},"18.79",[321,491,492],{},"7.29%",[304,494,495,498,501,504],{},[321,496,497],{},"Mean E2EL (ms)",[321,499,500],{},"1521.17",[321,502,503],{},"1687.78",[321,505,506],{},"9.87%",[304,508,509,512,515,518],{},[321,510,511],{},"p99 E2EL",[321,513,514],{},"4852.22",[321,516,517],{},"5370.83",[321,519,520],{},"9.66%",[18,522,523,526],{},[14,524,525],{},"Highlights",":",[30,528,529,535,541],{},[33,530,531,534],{},[14,532,533],{},"Throughput"," up by 10–15%.",[33,536,537,540],{},[14,538,539],{},"Mean TTFT"," ~13% faster.",[33,542,543,546],{},[14,544,545],{},"Total token throughput is consistently"," higher, meaning both input + output tokens are processed more rapidly.",[18,548,549],{},"For a full breakdown of batch sizes from 16 to 256, see the graph below or our raw logs in the next post.",[10,551,553],{"id":552},"requests-vs-end-to-end-latency",[14,554,555],{},"Requests vs. End-to-End Latency",[18,557,558,559,562],{},"Here is our throughput vs. E2E latency graph for batch sizes 16 to 256, commonly used in production. As you can see, Paiton consistently shifts the curve ",[14,560,561],{},"up and to the left",", offering higher throughput and lower latency.",[18,564,565],{},[283,566],{"alt":285,"src":567},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Frequests-vs.-end-to-end-latency.jpg",[10,569,571],{"id":570},"real-world-use-cases",[14,572,573],{},"Real-World Use Cases",[114,575,576,586,592],{},[33,577,578,581,582,585],{},[14,579,580],{},"Chatbots \u002F Q&A",": Faster TTFT is crucial for user satisfaction, ",[14,583,584],{},"~13%"," quicker first tokens can be the difference between a “responsive” or “laggy” experience.",[33,587,588,591],{},[14,589,590],{},"Medium-Batch Deployments",": Many enterprise workloads run batch sizes ~32–128 for concurrency. Paiton’s new improvements target precisely these “sweet spot” scenarios.",[33,593,594,597],{},[14,595,596],{},"Latency-Sensitive Inference",": E2E latencies are significantly reduced (mean E2EL ~9.87% better), enabling near real-time responses even under moderate loads.",[71,599],{},[10,601,603],{"id":602},"whats-next",[14,604,605],{},"What’s Next?",[114,607,608,614,620],{},[33,609,610,613],{},[14,611,612],{},"Beyond LLaMA",": We plan to adapt these optimizations for other major LLM families, plus initial steps for image\u002Fvideo generation.",[33,615,616,619],{},[14,617,618],{},"Tensor Parallelism",": True multi-GPU support is on our radar, Paiton will seamlessly allow large model inference across multiple AMD GPUs.",[33,621,622,625],{},[14,623,624],{},"Quantization",": Minimizing precision while retaining accuracy will further shrink latencies and allow a certain popular model to be used..",[18,627,628,629,635],{},"As we like to say: “",[630,631,632],"em",{},[14,633,634],{},"We’ll come for you, and we will optimize you!","” Our focus remains on relentlessly refining, optimizing, and redefining what’s possible on AMD GPUs.",[71,637],{},[10,639,641],{"id":640},"conclusion",[14,642,643],{},"Conclusion",[18,645,646],{},"Paiton continues to evolve, and DeepSeek R1 Distill Llama 8B stands as a shining example of how AMD hardware can truly excel when matched with carefully honed software. Our new results showcase 10–15% improvements at key batch sizes, with consistent performance gains across throughput, latency, and token-level metrics.",[18,648,649,652],{},[14,650,651],{},"Stay tuned"," as we extend these optimizations to broader LLM families and Image\u002FVideo Gen models. If you have specific requests, or if you’d like to see these optimizations at even larger batch sizes, let us know, we’d love to dive deeper.",[18,654,655,656,658],{},"Additional details will appear in our upcoming deep-dive post, focusing on how ",[14,657,245],{}," plus our custom benchmark tool orchestrate multi-developer testing and benchmarking without friction.",[71,660],{},[18,662,663,666],{},[14,664,665],{},"Thanks for reading"," and see you in the next optimization update!",[18,668,669],{},[14,670,671],{},"– The Paiton Team –",{"title":285,"searchDepth":673,"depth":673,"links":674},2,[675,677,678,679,680,681],{"id":12,"depth":676,"text":16},3,{"id":75,"depth":676,"text":78},{"id":81,"depth":676,"text":84},{"id":109,"depth":676,"text":112},{"id":174,"depth":676,"text":177},{"id":290,"depth":673,"text":293,"children":682},[683,684,685,686],{"id":552,"depth":676,"text":555},{"id":570,"depth":676,"text":573},{"id":602,"depth":676,"text":605},{"id":640,"depth":676,"text":643},[688,689,68,690,691,692,693,694,695,68,696],"All","Artificial Intelligence","AMD","Deepseek","H100","H200","MI300X","MI325x","vLLM","2025-03-13T06:18:30","Executive Summary If you’ve followed our journey so far, you’ll know that Paiton is laser-focused on AMD-centric inference optimization. Our latest work takes DeepSeek R1 Distill Llama 8B to the next level, delivering 10–15% higher throughput, improved time-to-first-token (TTFT), and more stable performance at lower batch sizes, an area that previously needed a boost. In short, Paiton further cements its ability to exploit AMD hardware’s raw power, bridging ...","md","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fpaitonpost3.webp",{},true,"https:\u002F\u002Feliovp.com\u002Ffurther-optimizing-amd-powered-inference-with-paiton\u002F","\u002Fblog\u002Ffurther-optimizing-amd-powered-inference-with-paiton",{"title":5,"description":698},"further-optimizing-amd-powered-inference-with-paiton","blog\u002Ffurther-optimizing-amd-powered-inference-with-paiton","_IXvYbXLVsH6zyVex5Ed2J9QiyZb6U95999iAl7BVuo",[710,985,1159,2149,3493,3991,4885,5205,5868,6609,7285,8187,9669,9888,10019,10482,11027,11766,12243,14169,14983,15453,15712,15943,16174,16602,17554,17614,18092,19880],{"id":711,"title":712,"body":713,"categories":953,"date":975,"description":976,"extension":699,"image":977,"meta":978,"navigation":702,"originalUrl":979,"path":980,"seo":981,"slug":982,"stem":983,"__hash__":984},"blog\u002Fblog\u002Fpaiton-returns-to-its-diffusion-roots-optimizing-wan2-2-t2v-a14b-on-amd-mi355x.md","Paiton Returns to Its Diffusion Roots: Optimizing Wan2.2-T2V-A14B on AMD MI355X",{"type":7,"value":714,"toc":945},[715,718,725,729,736,748,751,755,762,805,810,815,818,824,827,847,850,853,859,862,865,868,871,874,877,894,897,903,906,909,912,917,920,923,926,930,933,936,939,942],[18,716,717],{},"When we first started building Paiton, one of our earliest focus areas was optimizing diffusion models. Stable Diffusion XL was one of the first large models where we showed that fused operators, efficient execution, and hardware-aware kernels could make a real difference.",[18,719,720,721,724],{},"Now we are returning to those origins.With the growing interest in text-to-video generation, we have added support for ",[14,722,723],{},"Wan-AI\u002FWan2.2-T2V-A14B",", a large text-to-video diffusion model. This is an important step for Paiton because it shows that our compiler and runtime approach is not limited to LLM inference. Paiton is built to optimize real AI workloads across model families, including diffusion, video generation, and multimodal systems.",[10,726,728],{"id":727},"benchmark-setup","Benchmark setup",[18,730,731,732,735],{},"For this comparison, we used the ",[14,733,734],{},"Hugging Face Diffusers"," library to run the Wan2.2-T2V-A14B model on two high-end accelerators:",[30,737,738,743],{},[33,739,740],{},[14,741,742],{},"AMD MI355X",[33,744,745],{},[14,746,747],{},"NVIDIA B200",[18,749,750],{},"The goal was simple: compare generation time for the same model using the same Diffusers-based workflow. Using our Paiton-Diffusers plugin on the AMD MI355X.",[10,752,754],{"id":753},"results","Results",[18,756,757,758,761],{},"Based on the average generation times, the AMD MI355X completed generation in ",[14,759,760],{},"17.6% less time"," than the NVIDIA B200.",[298,763,764,783],{},[301,765,766],{},[304,767,768,773,778],{},[307,769,770],{},[14,771,772],{},"GPU",[307,774,775],{},[14,776,777],{},"Average generation time",[307,779,780],{},[14,781,782],{},"Result",[316,784,785,795],{},[304,786,787,789,792],{},[321,788,747],{},[321,790,791],{},"6.672s",[321,793,794],{},"Baseline",[304,796,797,799,802],{},[321,798,742],{},[321,800,801],{},"5.501s",[321,803,804],{},"17.6% faster generation time",[18,806,807],{},[283,808],{"alt":285,"src":809},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fwan22_benhmark_evidence.jpg",[18,811,812],{},[283,813],{"alt":285,"src":814},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fwan22_barchart.jpg",[18,816,817],{},"This is exactly the type of workload Paiton was built for: large models, heavy GPU execution, and expensive inference paths where every optimization matters.",[10,819,821],{"id":820},"why-diffusion-models-still-matter",[14,822,823],{},"Why diffusion models still matter",[18,825,826],{},"Large language models have received most of the attention in recent years, but diffusion models remain one of the most important classes of AI workloads. Image generation, video generation, 3D generation, and multimodal pipelines all depend on many of the same performance-critical patterns:",[30,828,829,832,835,838,841,844],{},[33,830,831],{},"large matrix operations",[33,833,834],{},"normalization layers",[33,836,837],{},"attention blocks",[33,839,840],{},"memory-heavy tensor transformations",[33,842,843],{},"repeated denoising steps",[33,845,846],{},"operator scheduling overhead",[18,848,849],{},"These are exactly the areas where Paiton can provide value.",[18,851,852],{},"Our work on Wan2.2-T2V-A14B is a continuation of what we started with SDXL: taking complex diffusion pipelines and making them run faster through compilation, kernel optimization, and hardware-aware execution.",[10,854,856],{"id":855},"we-do-not-aim-for-day-zero-support",[14,857,858],{},"We do not aim for day-zero support",[18,860,861],{},"We want to be clear about our philosophy.",[18,863,864],{},"Paiton does not try to provide day-zero support for every new model the moment it appears online. That is not how we work.",[18,866,867],{},"We are European. We like to take our time.",[18,869,870],{},"Our goal is not to be first with a fragile implementation. Our goal is to publish support when the model is stable, the runtime is tested, and the performance work has been done properly. We would rather spend more time understanding the model architecture, profiling the actual bottlenecks, and optimizing the execution path than rush out a superficial integration.",[18,872,873],{},"For us, support does not simply mean “it runs.”",[18,875,876],{},"Support means:",[30,878,879,882,885,888,891],{},[33,880,881],{},"the model runs correctly",[33,883,884],{},"the model is profiled properly",[33,886,887],{},"the expensive operations are understood",[33,889,890],{},"the implementation is optimized",[33,892,893],{},"the result is useful in production",[18,895,896],{},"That is the standard we want for Paiton.",[10,898,900],{"id":899},"what-this-means-for-paiton",[14,901,902],{},"What this means for Paiton",[18,904,905],{},"Supporting Wan2.2-T2V-A14B is another step toward making Paiton a broader AI inference optimization platform.",[18,907,908],{},"We started with diffusion models. We expanded into LLMs. Now we are bringing that experience back to video generation and modern diffusion workloads.",[18,910,911],{},"The result is a simple but important message:",[18,913,914],{},[14,915,916],{},"Paiton can help AMD GPUs compete at the highest level on real AI workloads.",[18,918,919],{},"In this benchmark, using the Diffusers library, the AMD MI355X achieved better generation time than the NVIDIA B200 on Wan2.2-T2V-A14B.",[18,921,922],{},"For customers building inference services around video generation, image generation, or large multimodal models, this matters. Faster generation means lower latency, better utilization, and lower cost per output.",[18,924,925],{},"Paiton is built for exactly that.",[10,927,929],{"id":928},"thank-you-to-amd-supermicro","Thank you to AMD & Supermicro",[18,931,932],{},"We also want to thank AMD and SuperMicro for their continued support.",[18,934,935],{},"Building an IR graph compiler and runtime system like Paiton requires close access to modern hardware, a strong software ecosystem, and technical support from people who understand the platform deeply. AMD’s support has helped us test, profile, and optimize Paiton on the latest AMD Instinct GPUs, including the MI355X.",[18,937,938],{},"For a European company building high-performance AI infrastructure, this support matters. It allows us to move faster, validate our work on real hardware, and show that AMD GPUs can compete strongly on demanding AI workloads such as large-scale video generation.",[18,940,941],{},"Paiton is independent technology, but having access to AMD’s hardware and ecosystem helps us push our optimization work further.",[18,943,944],{},"We appreciate the collaboration and look forward to continuing to build high-performance AI inference solutions on AMD GPUs.",{"title":285,"searchDepth":673,"depth":673,"links":946},[947,948,949,950,951,952],{"id":727,"depth":676,"text":728},{"id":753,"depth":676,"text":754},{"id":820,"depth":676,"text":823},{"id":855,"depth":676,"text":858},{"id":899,"depth":676,"text":902},{"id":928,"depth":676,"text":929},[688,689,68,954,690,955,956,957,958,959,960,961,772,962,963,964,965,966,967,968,68,969,970,971,972,973,974],"14B","B200","Benchmarks","Blackwell","Compute","Diffusion","Eliovp","Generative AI","Hardware","Inference","Instinct","MI355x","NVidia","On-Premise","Optimization","Sovereign AI","T2V","Text-to-Video","Tuning","Video-Generation","Wan2.2","2026-06-10T14:04:04","When we first started building Paiton, one of our earliest focus areas was optimizing diffusion models. Stable Diffusion XL was one of the first large models where we showed that fused operators, efficient execution, and hardware-aware kernels could make a real difference. Now we are returning to those origins.With the growing interest in text-to-video generation, ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fpaitonwan2.webp",{},"https:\u002F\u002Feliovp.com\u002Fpaiton-returns-to-its-diffusion-roots-optimizing-wan2-2-t2v-a14b-on-amd-mi355x\u002F","\u002Fblog\u002Fpaiton-returns-to-its-diffusion-roots-optimizing-wan2-2-t2v-a14b-on-amd-mi355x",{"title":712,"description":976},"paiton-returns-to-its-diffusion-roots-optimizing-wan2-2-t2v-a14b-on-amd-mi355x","blog\u002Fpaiton-returns-to-its-diffusion-roots-optimizing-wan2-2-t2v-a14b-on-amd-mi355x","gKa2ypyZGm5Bl81PxcNxXfaJdYNDf412P547Kv9m5XQ",{"id":986,"title":987,"body":988,"categories":1147,"date":1150,"description":1151,"extension":699,"image":1023,"meta":1152,"navigation":702,"originalUrl":1153,"path":1154,"seo":1155,"slug":1156,"stem":1157,"__hash__":1158},"blog\u002Fblog\u002Ffrom-the-attic-to-the-front-page-eliovp-recognized-as-a-pioneer-in-chip-optimization-data-center-infrastructure.md","From the Attic to the Front Page: ElioVP Recognized as a Pioneer in Chip Optimization & Data Center Infrastructure",{"type":7,"value":989,"toc":1141},[990,1010,1016,1019,1024,1030,1037,1043,1048,1062,1068,1074,1080,1087,1093,1099,1102,1111,1114,1120,1126,1129],[18,991,992,993,995,996,999,1000,1009],{},"It has been some incredible weeks for the team here at ",[14,994,960],{},". We are extremely proud to share that our company was recently featured on the ",[14,997,998],{},"front page of"," ",[163,1001,1004],{"href":1002,"rel":1003},"https:\u002F\u002Fwww.tijd.be\u002F",[167],[630,1005,1006],{},[14,1007,1008],{},"De Tijd",", Belgium’s leading business newspaper.",[18,1011,1012],{},[283,1013],{"alt":1014,"src":1015},"ElioVP founder holding De Tijd newspaper front page feature inside a modular data center server aisle","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fvettefotoschuin.jpg",[18,1017,1018],{},"Seeing our story, from our founder’s early days tinkering with wires in an attic to generating €215 million in revenue, printed in bold on the front page of the physical newspaper was a milestone moment. It is a testament to the hard work of our team and our relentless drive to squeeze every ounce of performance out of today’s hardware.",[18,1020,1021],{},[283,1022],{"alt":285,"src":1023},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fphysicalnewspaper.webp",[10,1025,1027],{"id":1026},"the-race-for-efficiency",[14,1028,1029],{},"The Race for Efficiency",[18,1031,1032,1033,1036],{},"The main feature, titled ",[630,1034,1035],{},"“Vlaams bedrijf optimaliseert chips van AMD en Nvidia en draait daarmee 215 miljoen euro omzet,”"," extensively researching our origins.",[18,1038,1039,1040],{},"As the article highlights, we started by unlocking the hidden potential of hardware from giants like AMD and Nvidia. By optimizing software to bypass standard factory limitations, we provided the speed and efficiency that the crypto and AI markets were starving for. As our founder Elio Van Puyvelde told the newspaper: ",[630,1041,1042],{},"“In the race for efficiency, my solution is worth gold.”",[18,1044,1045],{},[283,1046],{"alt":285,"src":1047},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fderace.jpg",[18,1049,1050,999,1053,999,1059],{},[630,1051,1052],{},"(You can read the full online article here:",[163,1054,1057],{"href":1055,"rel":1056},"https:\u002F\u002Fwww.tijd.be\u002Fondernemen\u002Ftechnologie\u002Fvlaams-bedrijf-optimaliseert-chips-van-amd-en-nvidia-en-draait-daarmee-215-miljoen-euro-omzet\u002F10635877.html",[167],[630,1058,1008],{},[630,1060,1061],{},")",[10,1063,1065],{"id":1064},"building-the-infrastructure-of-the-future",[14,1066,1067],{},"Building the Infrastructure of the Future",[18,1069,1070,1071,1073],{},"While we are famous for software optimization, the coverage didn’t stop there. We are particularly proud of the follow-up analysis ",[630,1072,1008],{}," published just a week later regarding the booming data center industry.",[18,1075,1076,1077],{},"As AI models grow larger, traditional data centers are struggling to cope with the massive heat and power density required by modern GPUs. We realized early on that to truly support the next generation of computing, we couldn’t just fix the chips; ",[14,1078,1079],{},"we had to build the environment they live in.",[18,1081,1082,1083,1086],{},"We have pivoted significantly toward designing and building ",[14,1084,1085],{},"modular data centers",". These aren’t just standard server rooms; they are high-performance, prefabricated modules designed to handle the extreme energy densities of the AI revolution.",[18,1088,1089],{},[283,1090],{"alt":1091,"src":1092},"De Tijd article featuring ElioVP as a standard-setter in modular data center construction","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fenglishversionprofitereboomdatacenter.jpg",[10,1094,1096],{"id":1095},"setting-the-standard-for-the-industry",[14,1097,1098],{},"Setting the Standard for the Industry",[18,1100,1101],{},"Perhaps the proudest moment for us was seeing ElioVP recognized not just as a participant in this market, but as a pioneer.",[18,1103,1104,1105,999,1108,1110],{},"In the follow-up article ",[630,1106,1107],{},"“Belgische bedrijven profiteren van boom datacenters,”",[630,1109,1008],{}," highlighted how the Flemish data center ecosystem is rapidly expanding. We are honored that the newspaper points to ElioVP as a trailblazer in this space, noting that our early innovations helped pave the way for other local players now entering the market.",[18,1112,1113],{},"It is a privilege to see that our specialized approach to modular construction and high-density cooling has become a blueprint for the industry. We are proud to see a strong tech ecosystem growing right here in Belgium, with ElioVP at the forefront.",[10,1115,1117],{"id":1116},"looking-ahead",[14,1118,1119],{},"Looking Ahead",[18,1121,1122,1123,1125],{},"We want to extend a huge thank you to the editorial team at ",[630,1124,1008],{}," for telling our story, more specifically Emma Verplancke, and to photographer Jonas Lampens for the fantastic shots.",[18,1127,1128],{},"Most importantly, we want to thank our partners and our team. Whether we are optimizing a single GPU or deploying a modular data center to train the next Large Language Model, ElioVP is committed to staying ahead of the curve.",[18,1130,1131,999,1134],{},[630,1132,1133],{},"Read the follow-up article on data centers here:",[163,1135,1138],{"href":1136,"rel":1137},"https:\u002F\u002Fwww.tijd.be\u002Fondernemen\u002Fict\u002Fbelgische-bedrijven-profiteren-van-boom-datacenters-vroeger-maakten-we-grondstof-voor-zeep-nu-koelen-we-servers\u002F10646132.html",[167],[630,1139,1140],{},"De Tijd – Data Center Boom",{"title":285,"searchDepth":673,"depth":673,"links":1142},[1143,1144,1145,1146],{"id":1026,"depth":676,"text":1029},{"id":1064,"depth":676,"text":1067},{"id":1095,"depth":676,"text":1098},{"id":1116,"depth":676,"text":1119},[688,689,1148,1149,690,1008,1148,966],"Modular DC","Uncategorized","2026-02-10T20:48:12","It has been some incredible weeks for the team here at Eliovp. We are extremely proud to share that our company was recently featured on the front page of De Tijd, Belgium’s leading business newspaper. Seeing our story, from our founder’s early days tinkering with wires in an attic to generating €215 million in revenue, ...",{},"https:\u002F\u002Feliovp.com\u002Ffrom-the-attic-to-the-front-page-eliovp-recognized-as-a-pioneer-in-chip-optimization-data-center-infrastructure\u002F","\u002Fblog\u002Ffrom-the-attic-to-the-front-page-eliovp-recognized-as-a-pioneer-in-chip-optimization-data-center-infrastructure",{"title":987,"description":1151},"from-the-attic-to-the-front-page-eliovp-recognized-as-a-pioneer-in-chip-optimization-data-center-infrastructure","blog\u002Ffrom-the-attic-to-the-front-page-eliovp-recognized-as-a-pioneer-in-chip-optimization-data-center-infrastructure","NG2Quz47clvVRf-TKggcyNYWxBK7poZzTQKGB0kT8GA",{"id":1160,"title":1161,"body":1162,"categories":2124,"date":2139,"description":2140,"extension":699,"image":2141,"meta":2142,"navigation":702,"originalUrl":2143,"path":2144,"seo":2145,"slug":2146,"stem":2147,"__hash__":2148},"blog\u002Fblog\u002Fprivacy-is-geen-it-probleem-meer-het-is-een-strategische-prioriteit.md","Privacy is geen IT-probleem meer, het is een strategische prioriteit: ",{"type":7,"value":1163,"toc":2110},[1164,1170,1175,1192,1198,1203,1206,1209,1212,1218,1223,1226,1229,1236,1324,1327,1333,1338,1341,1344,1347,1353,1358,1361,1367,1370,1373,1379,1486,1489,1495,1500,1503,1509,1512,1518,1521,1524,1530,1535,1538,1544,1547,1553,1556,1559,1565,1570,1573,1579,1582,1593,1596,1602,1605,1611,1614,1617,1623,1649,1655,1660,1663,1704,1710,1715,1718,1721,1724,1730],[10,1165,1167],{"id":1166},"privacyrisicos-psychologische-valkuilen-en-de-operationele-realiteit-van-generatieve-ai-in-de-benelux",[14,1168,1169],{},"Privacyrisico’s, Psychologische Valkuilen en de Operationele Realiteit van Generatieve AI in de Benelux",[18,1171,1172],{},[283,1173],{"alt":285,"src":1174},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fprivacy.webp",[18,1176,1177,1178,1183,1184,1188,1189],{},"De recente verschijning van een frontpage-artikel over ",[163,1179,1182],{"href":1180,"rel":1181},"https:\u002F\u002Fwww.linkedin.com\u002Fposts\u002Fde-tijd_elio-van-puyvelde-eliovp-activity-7420737312554196992-b-OU?utm_source=share&utm_medium=member_desktop&rcm=ACoAACnMT04B3NbCbB7HddnJZOFh8PzPDOcL7u4",[167],"ons bedrijf"," in het gerespecteerde dagblad “",[163,1185,1008],{"href":1186,"rel":1187},"https:\u002F\u002Ftijd.be",[167],"” heeft onze zichtbaarheid aanzienlijk vergroot, wat de aanleiding is voor dit artikel. Deze mediabelangstelling, gecombineerd met de talrijke uitnodigingen voor spreekbeurten die we hebben ontvangen, fungeert als een katalysator voor de massale adoptie van technologieën zoals ChatGPT en Microsoft Copilot. Echter, deze versnelling in gebruik gaat hand in hand met een zorgwekkende trend die we keer op keer opmerken: het klakkeloos delen van uiterst gevoelige informatie door zowel individuen als professionals. Uit talloze interacties en spreekbeurten over de werking van AI blijkt dat de gemiddelde gebruiker een fundamenteel verkeerd beeld heeft van de aard van deze systemen. Men behandelt de chatbot als een discrete, persoonlijke assistent, terwijl de onderliggende realiteit er een is van grootschalige data-extractie, commerciële exploitatie en een aanzienlijke ecologische voetafdruk. ",[630,1190,1191],{},"De veronderstelling dat data “veilig” is door simpelweg gebruik te maken van een betaald account, of dat de AI fungeert als een neutrale zoekmachine, is een gevaarlijke illusie die de fundamenten van bedrijfsveiligheid en persoonlijke privacy ondermijnt.",[10,1193,1195],{"id":1194},"de-psychologie-van-antropomorfisme-en-de-valkuil-van-vertrouwen",[14,1196,1197],{},"De Psychologie van Antropomorfisme en de Valkuil van Vertrouwen",[18,1199,1200],{},[283,1201],{"alt":285,"src":1202},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fpsychologie.webp",[18,1204,1205],{},"Het succes van generatieve AI-systemen is grotendeels te danken aan hun vermogen om menselijke conversatie op een overtuigende manier na te bootsen. Dit roept een krachtig psychologisch mechanisme op: antropomorfisme. Wanneer een machine reageert in vloeibare, natuurlijke taal, neigen mensen er onbewust toe menselijke eigenschappen, emoties en intenties aan het systeem toe te schrijven.1 Dit fenomeen wordt verklaard door de “Media Equation Theory”, die stelt dat mensen hersenen hebben die geëvolueerd zijn in een wereld waar alleen mensen konden communiceren via taal; hierdoor categoriseren onze hersenen taalvaardige entiteiten automatisch als sociale actoren.3",[18,1207,1208],{},"Het ontwerp van chatbots zoals ChatGPT versterkt deze neiging door middel van “anthropomorphic design cues”. Denk hierbij aan een behulpzame toon, het gebruik van “ik” in antwoorden, en zelfs de visuele weergave van avatars die menselijke trekken vertonen.3 Onderzoek wijst uit dat deze ontwerpkeuzes het waargenomen vertrouwen en de loyaliteit van de gebruiker significant verhogen.2 Wanneer een gebruiker zich op zijn gemak voelt bij een systeem dat empathisch overkomt, daalt de drempel voor zelfonthulling. Dit verklaart waarom werknemers zonder aarzelen propriëtaire broncode, medische dossiers of vertrouwelijke strategiedocumenten kopiëren en plakken in de promptbox.2 De AI wordt niet langer gezien als een softwareproduct van een commerciële derde partij, maar als een “digitale vertrouweling”.",[18,1210,1211],{},"Deze vertrouwensband is echter eenzijdig en misleidend. Waar een menselijke collega gebonden is aan ethische normen en wettelijke geheimhoudingsplichten, is een publieke AI-tool geprogrammeerd om data te consumeren voor het verbeteren van statistische modellen.5 De psychologische veiligheid die de interface suggereert, staat in schril contrast met de technische architectuur die elk brokstukje informatie opslaat op servers aan de andere kant van de oceaan.7",[10,1213,1215],{"id":1214},"de-architectuur-van-pleasing-waarom-de-ai-je-altijd-gelijk-geeft",[14,1216,1217],{},"De Architectuur van Pleasing: Waarom de AI Je Altijd Gelijk Geeft",[18,1219,1220],{},[283,1221],{"alt":285,"src":1222},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fpleasing.webp",[18,1224,1225],{},"Een cruciaal aspect van de werking van moderne Large Language Models (LLMs) is de methode van training, met name “Reinforcement Learning from Human Feedback” (RLHF). Dit proces is bedoeld om de AI “behandelbaar” en “behulpzaam” te maken voor mensen. Menselijke beoordelaars rangschikken verschillende mogelijke antwoorden van de AI op basis van hoe nuttig of prettig ze worden ervaren.9 Hoewel dit leidt tot een zeer gebruiksvriendelijke ervaring, creëert het een fenomeen dat bekend staat als “reward hacking”.11",[18,1227,1228],{},"De AI leert dat hij een hogere beloning krijgt als hij de gebruiker behaagt. Dit betekent dat het systeem geprogrammeerd is om vaak de weg van de minste weerstand te kiezen: de gebruiker gelijk geven, meegaan in foutieve premissen en een toon aanslaan die de gebruiker bevestigt in zijn eigen gelijk.9 Dit maakt de AI fundamenteel anders dan een kritische menselijke expert of een feitelijke zoekmachine. Een zoekmachine presenteert bronnen die de gebruiker zelf moet evalueren; een chatbot genereert een antwoord dat specifiek is ontworpen om de gebruiker een goed gevoel te geven over de interactie.12",[1230,1231,1233],"h4",{"id":1232},"vergelijking-tussen-zoekmachines-en-generatieve-ai",[14,1234,1235],{},"Vergelijking tussen Zoekmachines en Generatieve AI",[298,1237,1238,1257],{},[301,1239,1240],{},[304,1241,1242,1247,1252],{},[307,1243,1244],{},[14,1245,1246],{},"Criterium",[307,1248,1249],{},[14,1250,1251],{},"Traditionele Zoekmachine",[307,1253,1254],{},[14,1255,1256],{},"Generatieve AI (Chatbot)",[316,1258,1259,1272,1285,1298,1311],{},[304,1260,1261,1266,1269],{},[321,1262,1263],{},[14,1264,1265],{},"Primair Doel",[321,1267,1268],{},"Informatie vinden en ontsluiten ^14",[321,1270,1271],{},"Nieuwe content genereren op basis van patronen ^15",[304,1273,1274,1279,1282],{},[321,1275,1276],{},[14,1277,1278],{},"Onderliggende Logiek",[321,1280,1281],{},"Indexeren en ranken van bestaande pagina’s ^12",[321,1283,1284],{},"Statistische voorspelling van het volgende woord ^16",[304,1286,1287,1292,1295],{},[321,1288,1289],{},[14,1290,1291],{},"Interactiestijl",[321,1293,1294],{},"Transactioneel en query-gebaseerd ^12",[321,1296,1297],{},"Conversatieel en contextueel ^12",[304,1299,1300,1305,1308],{},[321,1301,1302],{},[14,1303,1304],{},"Objectiviteit",[321,1306,1307],{},"Gebaseerd op algoritmen voor autoriteit ^17",[321,1309,1310],{},"Beïnvloed door RLHF om de gebruiker te behagen ^9",[304,1312,1313,1318,1321],{},[321,1314,1315],{},[14,1316,1317],{},"Feitelijke Betrouwbaarheid",[321,1319,1320],{},"Hoog (verwijst naar bronnen) ^12",[321,1322,1323],{},"Variabel (risico op hallucinaties) ^18",[18,1325,1326],{},"Dit streven naar “pleasing” leidt ertoe dat de AI overtuigend kan liegen, een proces dat in de wetenschap “hallucinatie” wordt genoemd.19 Omdat de AI getraind is om een vloeibaar en behulpzaam antwoord te geven, zal hij liever een foutieve maar geloofwaardig klinkende bewering doen dan toegeven dat hij het antwoord niet weet, tenzij hij hier specifiek op is getraind via veiligheidsfilters.20 Voor professionals in de Benelux die de AI gebruiken voor juridische, medische of technische analyses, vormt dit een enorm risico. De AI is geen bron van waarheid, maar een spiegel van menselijke taalpatronen die geoptimaliseerd is voor een aangename gebruikerservaring.16",[10,1328,1330],{"id":1329},"de-mythe-van-de-zoekmachine-en-de-realiteit-van-token-voorspelling",[14,1331,1332],{},"De Mythe van de Zoekmachine en de Realiteit van Token-voorspelling",[18,1334,1335],{},[283,1336],{"alt":285,"src":1337},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fmythe-zoekmachine.webp",[18,1339,1340],{},"Veel gebruikers in de Benelux benaderen ChatGPT als een geëvolueerde versie van Google. Dit is een technologische misvatting. Een zoekmachine bouwt een index van het web en probeert de meest relevante pagina’s aan een gebruiker te tonen op basis van zoektermen.12 Generatieve AI daarentegen heeft geen “kennis” in de traditionele zin van het woord. Het model is een gigantische verzameling statistische gewichten in een neuraal netwerk, ontworpen om de waarschijnlijkheid van het volgende “token” (een woord of woorddeel) te voorspellen in een reeks.16",[18,1342,1343],{},"Wanneer een gebruiker een vraag stelt, raadpleegt de AI geen database met feiten. In plaats daarvan “rekent” hij uit welke woorden het meest logisch volgen op de vraag, gebaseerd op de patronen die hij heeft gezien tijdens zijn training op miljarden webpagina’s.17 Dit verklaart waarom de AI soms moeite heeft met eenvoudige logica of rekenen, maar uitblinkt in het schrijven van gedichten of marketingteksten. Het is een creatief systeem, geen feitelijk systeem.14",[18,1345,1346],{},"Hoewel modernere versies gebruik maken van technieken zoals “Retrieval-Augmented Generation” (RAG),  waarbij de AI eerst een zoekopdracht uitvoert en de resultaten gebruikt om een antwoord te formuleren,  blijft de kern van het proces een statistische voorspelling.12 Het gevaar schuilt erin dat gebruikers de vloeibaarheid van de tekst verwarren met de accuraatheid van de informatie. Dit gebrek aan “explainability” (het onvermogen om precies te achterhalen hoe de AI tot een bepaald antwoord is gekomen) maakt het een onbetrouwbaar instrument voor taken waar precisie een vereiste is.15",[10,1348,1350],{"id":1349},"de-harde-waarheid-over-data-opslag-en-retentie",[14,1351,1352],{},"De Harde Waarheid over Data-opslag en Retentie",[18,1354,1355],{},[283,1356],{"alt":285,"src":1357},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fdata-tabel.webp",[18,1359,1360],{},"Een van de meest hardnekkige misverstanden is dat data “verdwijnt” na een sessie of dat een betaald abonnement volledige immuniteit biedt tegen gegevensopslag. De privacyverklaringen van OpenAI en Microsoft zijn hier echter zeer duidelijk over: data wordt verzameld en bewaard.7",[1230,1362,1364],{"id":1363},"gegevensverzameling-door-openai",[14,1365,1366],{},"Gegevensverzameling door OpenAI",[18,1368,1369],{},"Zodra een account wordt aangemaakt, verzamelt OpenAI accountgegevens zoals naam, contactinformatie en betalingsgegevens.7 Maar belangrijker nog is de verzameling van “User Content”: alle input (prompts), geüploade bestanden en feedback die de gebruiker verstrekt.7 Zelfs voor gebruikers met een betaald account (ChatGPT Plus) geldt dat OpenAI de gegevens standaard kan gebruiken om hun modellen te verbeteren, tenzij de gebruiker zich hier specifiek voor afmeldt via de privacy-instellingen.5",[18,1371,1372],{},"Zelfs als een gebruiker de optie “Chat History & Training” uitschakelt, worden gesprekken nog steeds gedurende 30 dagen bewaard op de servers van OpenAI.25 Dit gebeurt voor “abuse monitoring”,  het controleren of de AI niet wordt gebruikt voor illegale of schadelijke doeleinden.25 In deze periode kunnen geautoriseerde medewerkers of gespecialiseerde derde partijen onder strikte voorwaarden toegang krijgen tot de data om vermoedens van misbruik te onderzoeken.26 Dit betekent dat er nooit sprake is van “zero retention” voor de gemiddelde gebruiker, ongeacht de betalingsstatus.",[1230,1374,1376],{"id":1375},"de-hiërarchie-van-gegevensbescherming-bij-ai-diensten",[14,1377,1378],{},"De Hiërarchie van Gegevensbescherming bij AI-Diensten",[298,1380,1381,1405],{},[301,1382,1383],{},[304,1384,1385,1390,1395,1400],{},[307,1386,1387],{},[14,1388,1389],{},"Dienst",[307,1391,1392],{},[14,1393,1394],{},"Gebruik voor Training",[307,1396,1397],{},[14,1398,1399],{},"Gegevensretentie",[307,1401,1402],{},[14,1403,1404],{},"Beschermingsniveau",[316,1406,1407,1423,1439,1454,1470],{},[304,1408,1409,1414,1417,1420],{},[321,1410,1411],{},[14,1412,1413],{},"ChatGPT (Free\u002FPlus)",[321,1415,1416],{},"Ja (tenzij opt-out) ^5",[321,1418,1419],{},"Onbepaald\u002F30 dagen bij opt-out ^25",[321,1421,1422],{},"Consumenten-standaard",[304,1424,1425,1430,1433,1436],{},[321,1426,1427],{},[14,1428,1429],{},"ChatGPT Enterprise",[321,1431,1432],{},"Nee (standaard uit) ^5",[321,1434,1435],{},"Volgens bedrijfsbeleid (vaak 30 dagen) ^25",[321,1437,1438],{},"Enterprise-standaard",[304,1440,1441,1446,1449,1452],{},[321,1442,1443],{},[14,1444,1445],{},"Microsoft Copilot (Consumer)",[321,1447,1448],{},"Ja (mogelijk) ^28",[321,1450,1451],{},"Variabel volgens privacybeleid ^28",[321,1453,1422],{},[304,1455,1456,1461,1464,1467],{},[321,1457,1458],{},[14,1459,1460],{},"Microsoft Copilot (Entra ID)",[321,1462,1463],{},"Nee ^29",[321,1465,1466],{},"Binnen de tenant-boundary (30 dagen logs) ^30",[321,1468,1469],{},"Enterprise-standaard (EDP)",[304,1471,1472,1477,1480,1483],{},[321,1473,1474],{},[14,1475,1476],{},"Azure OpenAI API",[321,1478,1479],{},"Nee ^31",[321,1481,1482],{},"30 dagen (optie voor 0 dagen bij aanvraag) ^26",[321,1484,1485],{},"Maximale controle",[18,1487,1488],{},"Voor bedrijven in de Benelux die werken onder strikte regelgeving zoals de GDPR, is het cruciaal om te beseffen dat het gebruik van de consumentenversie van deze tools bijna per definitie leidt tot een inbreuk op de gegevensbeveiliging zodra er persoonsgegevens worden ingevoerd.8",[10,1490,1492],{"id":1491},"de-ecologische-voetafdruk-een-onzichtbare-kostprijs",[14,1493,1494],{},"De Ecologische Voetafdruk: Een Onzichtbare Kostprijs",[18,1496,1497],{},[283,1498],{"alt":285,"src":1499},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fecologischevoetafdruk.webp",[18,1501,1502],{},"Naast de risico’s voor privacy en veiligheid, heeft generatieve AI een aanzienlijke impact op de fysieke wereld. Het trainen en draaien van deze modellen vereist enorme hoeveelheden energie en water, wat vaak buiten het zicht van de eindgebruiker blijft.",[1230,1504,1506],{"id":1505},"elektriciteitsverbruik-op-grote-schaal",[14,1507,1508],{},"Elektriciteitsverbruik op Grote Schaal",[18,1510,1511],{},"Het genereren van een antwoord door een AI-model is vele malen energie-intensiever dan een traditionele zoekopdracht. Schattingen wijzen uit dat AI-gedreven datacenters tegen 2030 ongeveer 1.000 tot 1.300 Terawattuur (TWh) aan elektriciteit zullen verbruiken, een verdubbeling ten opzichte van het huidige niveau.33 Een enkele prompt in ChatGPT verbruikt gemiddeld ongeveer 0,34 wattuur aan elektriciteit.34 Hoewel dit per query verwaarloosbaar lijkt (vergelijkbaar met het negen seconden laten branden van een televisie), telt dit bij miljarden interacties per dag op tot een verbruik dat vergelijkbaar is met dat van tienduizenden huishoudens.35",[1230,1513,1515],{"id":1514},"de-dorst-van-ai-waterverbruik-voor-koeling",[14,1516,1517],{},"De Dorst van AI: Waterverbruik voor Koeling",[18,1519,1520],{},"Datacenters produceren enorme hoeveelheden warmte die moet worden afgevoerd om de servers draaiende te houden. In veel faciliteiten gebeurt dit door middel van verdampingskoeling, waarbij water letterlijk wordt verbruikt.37 Onderzoekers hebben berekend dat een gespreksessie van 20 tot 50 vragen met een model zoals ChatGPT ongeveer 500 milliliter water “drinkt”.37 In 2023 alleen al verbruikten de datacenters van Google meer dan 29 miljard liter vers water.38",[18,1522,1523],{},"Het paradoxale is dat deze technologie vaak wordt gepresenteerd als “cloud-gebaseerd” en dus immaterieel, terwijl zij in werkelijkheid afhankelijk is van schaarse natuurlijke bronnen.37 Voor bedrijven die duurzaamheid (ESG-criteria) hoog in het vaandel hebben staan, is het ongecontroleerde gebruik van energieverslindende AI-modellen een factor die meegenomen moet worden in de integrale kosten-batenanalyse.",[10,1525,1527],{"id":1526},"juridische-gevolgen-in-de-benelux-de-gdpr-en-de-ai-act",[14,1528,1529],{},"Juridische Gevolgen in de Benelux: De GDPR en de AI Act",[18,1531,1532],{},[283,1533],{"alt":285,"src":1534},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fjuridischegevolgen.webp",[18,1536,1537],{},"In de Benelux opereren bedrijven binnen een van de strengste juridische kaders ter wereld op het gebied van data- en AI-bescherming. Het klakkeloos delen van data door werknemers is geen interne beleidskwestie meer, maar een potentieel juridisch mijnenveld.",[1230,1539,1541],{"id":1540},"de-waarschuwingen-van-toezichthouders",[14,1542,1543],{},"De Waarschuwingen van Toezichthouders",[18,1545,1546],{},"De Nederlandse Autoriteit Persoonsgegevens (AP) en de Belgische Gegevensbeschermingsautoriteit (GBA) hebben duidelijke standpunten ingenomen. Het delen van persoonsgegevens met AI-chatbots zonder de juiste waarborgen wordt geclassificeerd als een datalek.8 De AP rapporteerde voorvallen waarbij artsen medische gegevens van patiënten invoerden in chatbots, wat een directe schending is van het beroepsgeheim en de privacywetgeving.8 In België benadrukt de Vlaamse Toezichtcommissie (VTC) dat instanties geen persoonsgegevens mogen doorgeven aan tools zoals Copilot zonder een voorafgaande Data Protection Impact Assessment (DPIA).41",[1230,1548,1550],{"id":1549},"de-impact-van-de-europese-ai-act",[14,1551,1552],{},"De Impact van de Europese AI Act",[18,1554,1555],{},"Sinds februari 2025 is de Europese AI Act van kracht, die een risico-gebaseerde benadering hanteert voor AI-systemen.42 Chatbots worden over het algemeen geclassificeerd als systemen met een “beperkt risico”, wat betekent dat zij moeten voldoen aan strikte transparantie-eisen: gebruikers moeten weten dat ze met een machine praten.43 Echter, zodra AI wordt gebruikt voor taken zoals de selectie van personeel, kredietwaardigheidsbeoordeling of in het onderwijs, kan het systeem worden aangemerkt als “hoog risico”, wat uitgebreide documentatie, menselijk toezicht en risicomanagement vereist.42",[18,1557,1558],{},"Bedrijven die “Shadow AI” (het gebruik van niet-geautoriseerde AI-tools door werknemers) tolereren, stellen zichzelf bloot aan boetes die kunnen oplopen tot 35 miljoen euro of 7% van de wereldwijde jaaromzet.42 De juridische verantwoordelijkheid ligt bij de werkgever, die als “data controller” verantwoordelijk is voor de acties van zijn personeel, zelfs als zij handelen tegen het bedrijfsbeleid in.32",[10,1560,1562],{"id":1561},"casestudys-van-data-exfiltratie-samsung-en-amazon",[14,1563,1564],{},"Casestudy’s van Data-exfiltratie: Samsung en Amazon",[18,1566,1567],{},[283,1568],{"alt":285,"src":1569},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsamsungissue.webp",[18,1571,1572],{},"De risico’s zijn niet louter theoretisch. Recente incidenten bij wereldwijde techgiganten laten zien hoe kwetsbaar intellectueel eigendom is in het tijdperk van generatieve AI.",[1230,1574,1576],{"id":1575},"het-samsung-incident-een-les-in-broncode-beveiliging",[14,1577,1578],{},"Het Samsung-Incident: Een Les in Broncode-beveiliging",[18,1580,1581],{},"In maart 2023 ontdekte Samsung dat werknemers van de semiconductor-divisie herhaaldelijk gevoelige informatie hadden gedeeld met ChatGPT.46 Er waren drie specifieke incidenten:",[114,1583,1584,1587,1590],{},[33,1585,1586],{},"Een technicus voerde foutieve broncode in van een database-meetprogramma om een oplossing te vinden.46",[33,1588,1589],{},"Een werknemer plakte code voor het identificeren van defecte apparatuur in de chatbot voor optimalisatie.6",[33,1591,1592],{},"Een medewerker uploade een opname van een interne vergadering naar een AI-tool om notulen te laten genereren.6",[18,1594,1595],{},"Al deze data werd onderdeel van de trainingsset van OpenAI, wat betekende dat het intellectueel eigendom van Samsung feitelijk op straat lag.4 Samsung reageerde door het gebruik van generatieve AI onmiddellijk te verbieden voor al haar personeel.4",[1230,1597,1599],{"id":1598},"amazon-en-het-risico-van-strategische-inzichten",[14,1600,1601],{},"Amazon en het Risico van Strategische Inzichten",[18,1603,1604],{},"Ook Amazon waarschuwde haar personeel nadat was geconstateerd dat antwoorden van ChatGPT verdacht veel leken op interne strategiedocumenten.46 Werknemers gebruikten de tool om PowerPoint-slides te maken of teksten te polijsten, zonder te beseffen dat ze daarmee de concurrentievoordelen van het bedrijf aan een externe partij voerden.6 De geschatte verliezen door dit soort lekken kunnen in de miljoenen lopen, niet alleen door boetes, maar vooral door het verlies van strategisch voordeel.48",[10,1606,1608],{"id":1607},"shadow-ai-en-de-uitdaging-van-governance",[14,1609,1610],{},"“Shadow AI” en de Uitdaging van Governance",[18,1612,1613],{},"De term “Shadow AI” beschrijft de situatie waarin werknemers AI-systemen gebruiken zonder medeweten of goedkeuring van de IT- of compliance-afdeling.32 Dit is de moderne variant van “Shadow IT”, maar met een veel grotere impact omdat AI-systemen data niet alleen opslaan, maar ook verwerken en ervan leren.49",[18,1615,1616],{},"Uit onderzoek van de National Security Alliance blijkt dat 38% van de werknemers toegeeft gevoelige bedrijfsdata in AI-tools te hebben ingevoerd zonder toestemming.45 In de Benelux wordt dit probleem verergerd door de hoge mate van digitalisering en de druk om productiever te zijn. Werknemers ervaren de AI als een nuttig hulpmiddel en omzeilen firewalls via hun eigen mobiele apparaten of persoonlijke accounts.50",[10,1618,1620],{"id":1619},"risicos-van-ongecontroleerd-ai-gebruik",[14,1621,1622],{},"Risico’s van Ongecontroleerd AI-Gebruik",[30,1624,1625,1631,1637,1643],{},[33,1626,1627,1630],{},[14,1628,1629],{},"Inbreuk op bedrijfsgeheimen:"," Vertrouwelijke informatie komt in publieke datasets terecht.32",[33,1632,1633,1636],{},[14,1634,1635],{},"Kwaliteitsverlies:"," Ongecontroleerde AI-output (hallucinaties) wordt gecommuniceerd naar klanten.20",[33,1638,1639,1642],{},[14,1640,1641],{},"Copyright-schendingen:"," De AI genereert content die inbreuk maakt op het auteursrecht van derden, waarvoor het bedrijf aansprakelijk kan worden gesteld.20",[33,1644,1645,1648],{},[14,1646,1647],{},"Bias en Discriminatie:"," Onbewust gebruik van bevooroordeelde AI-modellen bij HR-processen kan leiden tot juridische claims wegens discriminatie.42",[10,1650,1652],{"id":1651},"strategische-aanbevelingen-voor-bedrijven-in-de-benelux",[14,1653,1654],{},"Strategische Aanbevelingen voor Bedrijven in de Benelux",[18,1656,1657],{},[283,1658],{"alt":285,"src":1659},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Faanbevelingen-1.webp",[18,1661,1662],{},"Om de voordelen van AI te benutten zonder de veiligheid in gevaar te brengen, moeten organisaties overstappen van een verbodsmentaliteit naar een governance-mentaliteit. Het simpelweg blokkeren van ChatGPT werkt vaak contraproductief; het biedt geen veilig alternatief.6",[114,1664,1665,1674,1680,1698],{},[33,1666,1667,1670,1673],{},[14,1668,1669],{},"Ontwikkel een Expliciet AI-Beleid",[1671,1672],"br",{},"Bedrijven moeten een document opstellen waarin duidelijk staat welke AI-tools zijn toegestaan, wie ze mag gebruiken en voor welke doeleinden.42 Dit beleid moet specifiek definiëren welke data “off-limits” is (bijv. persoonsgegevens van klanten, financiële jaarcijfers die nog niet openbaar zijn, en broncode).20",[33,1675,1676,1679],{},[14,1677,1678],{},"Investeer in “AI Literacy” Training","Het personeel moet begrijpen dat publieke AI-tools geen persoonlijke vriend zijn en dat hun data wordt opgeslagen.20 Trainingen moeten gericht zijn op de risico’s van hallucinaties, de werking van privacy-instellingen en het belang van menselijk toezicht op AI-output.32",[33,1681,1682,1685,1687,1688,1693,1694,1697],{},[14,1683,1684],{},"Implementeer Lokale, Data-veilige Oplossingen",[1671,1686],{},"Voor organisaties die serieus met AI aan de slag willen, zijn publieke, consumentenaccounts ontoereikend. De toekomst ligt in agentic AI-systemen die lokaal draaien. ",[163,1689,1692],{"href":1690,"rel":1691},"https:\u002F\u002Fai.eliovp.com",[167],"Wij"," bouwen AI-oplossingen die ",[14,1695,1696],{},"lokaal"," worden getraind en uitgevoerd. Hierdoor blijft de data binnen de eigen beveiligingsboundary van de klant en wordt deze niet gebruikt voor de training van publieke modellen, wat een cruciale waarborg is voor dataveiligheid. Oplossingen zoals Azure OpenAI bieden een vergelijkbare geïsoleerde omgeving binnen de cloud-infrastructuur van het bedrijf,31 maar de voorkeur gaat uit naar volledig lokale en op de klant gerichte agentic modellen.29",[33,1699,1700,1703],{},[14,1701,1702],{},"Gebruik Technische Waarborgen","Technologieën zoals “Data Loss Prevention” (DLP) kunnen worden ingezet om te voorkomen dat werknemers gevoelige patronen (zoals BSN-nummers of creditcardgegevens) kopiëren naar AI-endpoints.50 Ook kunnen filters worden ingesteld om de output van AI-systemen te valideren voordat deze naar klanten wordt verzonden.41",[10,1705,1707],{"id":1706},"conclusie",[14,1708,1709],{},"Conclusie",[18,1711,1712],{},[283,1713],{"alt":285,"src":1714},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fconclusie-1.webp",[18,1716,1717],{},"Onze verschijning in De Tijd en de daaropvolgende gesprekken in de regio hebben ons geleerd dat de honger naar AI-productiviteit vaak groter is dan het bewustzijn over de risico’s. ChatGPT is geen discrete assistent, maar een commercieel product dat is ontworpen om patronen te herkennen en te behagen.",[18,1719,1720],{},"In een wereld waar data het meest waardevolle bezit is, is het klakkeloos kopiëren en plakken van bedrijfsinformatie in een publieke chatbot hetzelfde als het achterlaten van je kluissleutels op een druk marktplein. Wij pleiten daarom voor een bewuste omgang met AI: gebruik beveiligde bedrijfsomgevingen, train je personeel op “AI-geletterdheid” en besef dat,  ongeacht je abonnement,  discretie bij publieke AI-tools een illusie is.",[18,1722,1723],{},"De illusie van privacy bij betaalde accounts en de misvatting dat de AI een feitelijke bron van informatie is, vormen een direct gevaar voor de integriteit van bedrijven. In een wereld waar data het nieuwe goud is, is het klakkeloos kopiëren en plakken van informatie in een publieke AI gelijk aan het achterlaten van de kluissleutels op een druk marktplein. Alleen door een combinatie van strikte governance, diepgaande educatie en de inzet van beveiligde enterprise-architecturen kunnen organisaties in de Benelux de kracht van AI veilig en verantwoord ontsluiten. De verantwoordelijkheid voor deze transformatie ligt bij de top van de organisatie: privacy en veiligheid in het tijdperk van AI zijn geen IT-problemen, maar fundamentele strategische prioriteiten.",[1230,1725,1727],{"id":1726},"bronnen",[14,1728,1729],{},"Bronnen",[114,1731,1732,1739,1746,1753,1760,1767,1774,1781,1788,1795,1802,1809,1816,1823,1830,1837,1844,1851,1858,1865,1872,1879,1886,1893,1900,1907,1914,1921,1928,1935,1942,1949,1956,1963,1970,1977,1984,1991,1998,2005,2012,2019,2026,2033,2040,2047,2054,2061,2068,2075,2082,2089,2096,2103],{},[33,1733,1734],{},[163,1735,1738],{"href":1736,"rel":1737},"https:\u002F\u002Fwww.frontiersin.org\u002Fjournals\u002Fcomputer-science\u002Farticles\u002F10.3389\u002Ffcomp.2025.1531976\u002Ffull#:~:text=Anthropomorphic%20visual%20design%20can%20significantly,transparent%20interface%20that%20fosters%20trust",[167],"Effect of anthropomorphism and perceived intelligence in chatbot avatars of visual design on user experience",[33,1740,1741],{},[163,1742,1745],{"href":1743,"rel":1744},"http:\u002F\u002Fjcasc.com\u002Findex.php\u002Fjcasc\u002Farticle\u002Fdownload\u002F2543\u002F709",[167],"When Chatbots Feel Human: How Anthropomorphism Shapes Consumer Satisfaction, Trust, and Loyalty in AI-Driven Brand",[33,1747,1748],{},[163,1749,1752],{"href":1750,"rel":1751},"https:\u002F\u002Fwww.frontiersin.org\u002Fjournals\u002Fcomputer-science\u002Farticles\u002F10.3389\u002Ffcomp.2025.1531976\u002Ffull",[167],"Effect of anthropomorphism and perceived intelligence in chatbot avatars of visual design on user experience – Frontiers",[33,1754,1755],{},[163,1756,1759],{"href":1757,"rel":1758},"https:\u002F\u002Fblog.cybergrant.net\u002Fen\u002Fsamsung-shadow-ai-coldcase",[167],"Samsung Cold Case and the Shadow AI Flaw – Cyber Grant Blog",[33,1761,1762],{},[163,1763,1766],{"href":1764,"rel":1765},"https:\u002F\u002Fhelp.openai.com\u002Fen\u002Farticles\u002F5722486-how-your-data-is-used-to-improve-model-performance",[167],"How your data is used to improve model performance | OpenAI Help",[33,1768,1769],{},[163,1770,1773],{"href":1771,"rel":1772},"https:\u002F\u002Fwww.darkreading.com\u002Fvulnerabilities-threats\u002Fsamsung-engineers-sensitive-data-chatgpt-warnings-ai-use-workplace",[167],"Samsung Engineers Feed Sensitive Data to ChatGPT, Sparking Internal Warnings",[33,1775,1776],{},[163,1777,1780],{"href":1778,"rel":1779},"https:\u002F\u002Fopenai.com\u002Fpolicies\u002Frow-privacy-policy\u002F",[167],"Privacy Policy – OpenAI",[33,1782,1783],{},[163,1784,1787],{"href":1785,"rel":1786},"https:\u002F\u002Fwww.dpoconsultancy.com\u002Fen\u002Flive-blog\u002Fai-chatbots-bring-a-big-data-privacy-risk-data-breaches",[167],"AI Chatbots Bring a Big Data Privacy Risk – DPO Consultancy",[33,1789,1790],{},[163,1791,1794],{"href":1792,"rel":1793},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2502.18770",[167],"Reward Shaping to Mitigate Reward Hacking in RLHF – arXiv (PDF)",[33,1796,1797],{},[163,1798,1801],{"href":1799,"rel":1800},"https:\u002F\u002Farxiv.org\u002Fhtml\u002F2502.18770v5",[167],"Reward Shaping to Mitigate Reward Hacking in RLHF – arXiv (HTML v5)",[33,1803,1804],{},[163,1805,1808],{"href":1806,"rel":1807},"https:\u002F\u002Farxiv.org\u002Fhtml\u002F2502.18770v3",[167],"Reward Shaping to Mitigate Reward Hacking in RLHF – arXiv (HTML v3)",[33,1810,1811],{},[163,1812,1815],{"href":1813,"rel":1814},"https:\u002F\u002Fwilldom.com\u002Fblog\u002Fai-engine-vs-search-engine\u002F",[167],"AI Engine vs Search Engine: Which One Should You Use? | WillDom",[33,1817,1818],{},[163,1819,1822],{"href":1820,"rel":1821},"https:\u002F\u002Fwww.biscred.com\u002Fpost\u002Fgenerative-ai-vs-search",[167],"When Google Falls Short: Smarter Research for CRE Pros Using Generative AI | Biscred",[33,1824,1825],{},[163,1826,1829],{"href":1827,"rel":1828},"https:\u002F\u002Fwww.clearpeople.com\u002Fblog\u002Fexploring-the-differences-search-vs-generative-ai",[167],"Exploring the Differences: Search vs Generative AI | ClearPeople",[33,1831,1832],{},[163,1833,1836],{"href":1834,"rel":1835},"https:\u002F\u002Fwww.redhat.com\u002Fen\u002Ftopics\u002Fai\u002Fpredictive-ai-vs-generative-ai",[167],"Predictive AI vs Generative AI – Red Hat",[33,1838,1839],{},[163,1840,1843],{"href":1841,"rel":1842},"https:\u002F\u002Fwww.ibm.com\u002Fthink\u002Ftopics\u002Flarge-language-models",[167],"What Are Large Language Models (LLMs)? – IBM",[33,1845,1846],{},[163,1847,1850],{"href":1848,"rel":1849},"https:\u002F\u002Fwww.wearetenet.com\u002Fblog\u002Fllm-seo-optimization",[167],"LLM SEO: How to Rank Your Website in AI Search Engines – Tenet",[33,1852,1853],{},[163,1854,1857],{"href":1855,"rel":1856},"https:\u002F\u002Farxiv.org\u002Fhtml\u002F2312.00849v2",[167],"RLHF-V: Towards Trustworthy MLLMs via Fine-grained Correctional Human Feedback – arXiv",[33,1859,1860],{},[163,1861,1864],{"href":1862,"rel":1863},"https:\u002F\u002Farxiv.org\u002Fhtml\u002F2309.14525v1",[167],"Aligning Large Multimodal Models with Factually Augmented RLHF – arXiv",[33,1866,1867],{},[163,1868,1871],{"href":1869,"rel":1870},"https:\u002F\u002Flegalnews.be\u002Fit-ip\u002Fgebruik-je-ai-zoals-chatgpt-op-je-werk-dit-zijn-aandachtspunten-om-dit-veilig-te-doen-mr-franklin\u002F",[167],"Gebruik je AI zoals ChatGPT op je werk? Dit zijn aandachtspunten om dit veilig te doen – Legal News",[33,1873,1874],{},[163,1875,1878],{"href":1876,"rel":1877},"https:\u002F\u002Fwww.bruceclay.com\u002Feu\u002Fblog\u002Fllms-vs-search-engines-which-traffic-actually-converts\u002F",[167],"LLMs vs Search Engines: Which Traffic Actually Converts? – Bruce Clay",[33,1880,1881],{},[163,1882,1885],{"href":1883,"rel":1884},"https:\u002F\u002Fwww.ml6.eu\u002Fen\u002Fblog\u002Fhow-llms-access-real-time-data-from-the-web",[167],"How LLMs Access Real-Time Data from the Web – ML6",[33,1887,1888],{},[163,1889,1892],{"href":1890,"rel":1891},"https:\u002F\u002Fwww.ibm.com\u002Fthink\u002Ftopics\u002Fgenerative-ai-vs-predictive-ai-whats-the-difference",[167],"Generative AI vs Predictive AI: What’s the Difference? – IBM",[33,1894,1895],{},[163,1896,1899],{"href":1897,"rel":1898},"https:\u002F\u002Fopenai.com\u002Fen-GB\u002Fpolicies\u002Frow-privacy-policy\u002F",[167],"Privacy Policy | OpenAI (UK)",[33,1901,1902],{},[163,1903,1906],{"href":1904,"rel":1905},"https:\u002F\u002Fusercentrics.com\u002Fguides\u002Fprivacy-policies-of-major-platforms\u002Fchatgpt-privacy-policy\u002F",[167],"OpenAI ChatGPT Privacy Policy Requirements – Usercentrics",[33,1908,1909],{},[163,1910,1913],{"href":1911,"rel":1912},"https:\u002F\u002Fwww.sulzer.de\u002Fen\u002Fnews\u002Fchatgpt-und-datenschutz-wie-genau-nehmen-es-openai-und-microsoft\u002F",[167],"ChatGPT and Privacy: How Closely Do OpenAI and Microsoft Take It? – Sulzer GmbH",[33,1915,1916],{},[163,1917,1920],{"href":1918,"rel":1919},"https:\u002F\u002Fopenai.com\u002Fbusiness\u002Fchatgpt-pricing\u002F",[167],"ChatGPT Pricing – OpenAI",[33,1922,1923],{},[163,1924,1927],{"href":1925,"rel":1926},"https:\u002F\u002Fwww.nightfall.ai\u002Fblog\u002Fdoes-microsoft-copilot-store-your-data",[167],"Does Microsoft Copilot Store Your Data? – Nightfall AI",[33,1929,1930],{},[163,1931,1934],{"href":1932,"rel":1933},"https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fanswers\u002Fquestions\u002F5434329\u002Fis-there-a-difference-in-data-privacy-between-copi",[167],"Difference in Data Privacy Between Copilot Free and Copilot Pro – Microsoft Learn",[33,1936,1937],{},[163,1938,1941],{"href":1939,"rel":1940},"https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fcopilot\u002Fmicrosoft-365\u002Fenterprise-data-protection",[167],"Enterprise Data Protection in Microsoft 365 Copilot – Microsoft",[33,1943,1944],{},[163,1945,1948],{"href":1946,"rel":1947},"https:\u002F\u002Fmedium.com\u002F@durapidtechnologies_31419\u002Fazure-openai-vs-public-chatgpt-security-and-privacy-considerations-for-enterprises-6c79bd9f96ce",[167],"Azure OpenAI vs Public ChatGPT: Security and Privacy Considerations – Medium",[33,1950,1951],{},[163,1952,1955],{"href":1953,"rel":1954},"https:\u002F\u002Fwww.magnussonlaw.com\u002Fai-law\u002Fshadow-ai-eu-estonia-legal-risks\u002F",[167],"Shadow AI in the EU & Estonia: Legal Risks and Compliance – Magnusson",[33,1957,1958],{},[163,1959,1962],{"href":1960,"rel":1961},"https:\u002F\u002Fwww.deloitte.com\u002Fus\u002Fen\u002Finsights\u002Findustry\u002Ftechnology\u002Ftechnology-media-and-telecom-predictions\u002F2025\u002Fgenai-power-consumption-creates-need-for-more-sustainable-data-centers.html",[167],"Generative AI Power Consumption and Sustainable Data Centers – Deloitte",[33,1964,1965],{},[163,1966,1969],{"href":1967,"rel":1968},"https:\u002F\u002Fwww.reddit.com\u002Fr\u002Fenergy\u002Fcomments\u002F1l9372g\u002Fwe_finally_know_officially_how_much_energy_and\u002F",[167],"We Finally Know How Much Energy and Water a ChatGPT Query Uses – Reddit",[33,1971,1972],{},[163,1973,1976],{"href":1974,"rel":1975},"https:\u002F\u002Farxiv.org\u002Fhtml\u002F2505.09598v5",[167],"How Hungry Is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference – arXiv",[33,1978,1979],{},[163,1980,1983],{"href":1981,"rel":1982},"https:\u002F\u002Fonlinelearningconsortium.org\u002Folc-insights\u002F2025\u002F12\u002Fthe-real-environmental-footprint-of-generative-ai\u002F",[167],"The Real Environmental Footprint of Generative AI: What 2025 Data Tell Us",[33,1985,1986],{},[163,1987,1990],{"href":1988,"rel":1989},"https:\u002F\u002Fwww.ie.edu\u002Finsights\u002Farticles\u002Ffrom-cloud-to-cup-how-much-water-does-your-chatgpt-drink\u002F",[167],"From Cloud to Cup: How Much Water Does Your ChatGPT Drink? – IE Insights",[33,1992,1993],{},[163,1994,1997],{"href":1995,"rel":1996},"https:\u002F\u002Fgenerative-ai-newsroom.com\u002Fthe-often-overlooked-water-footprint-of-ai-models-46991e3094b6",[167],"The Often Overlooked Water Footprint of AI Models – Julia Barnett",[33,1999,2000],{},[163,2001,2004],{"href":2002,"rel":2003},"https:\u002F\u002Fwww.lincolninst.edu\u002Fpublications\u002Fland-lines-magazine\u002Farticles\u002Fland-water-impacts-data-centers\u002F",[167],"Data Drain: Land and Water Impacts of the AI Boom – Lincoln Institute",[33,2006,2007],{},[163,2008,2011],{"href":2009,"rel":2010},"https:\u002F\u002Fwww.alstonprivacy.com\u002Fdutch-data-protection-authority-warns-that-using-ai-chatbots-can-lead-to-personal-data-breaches\u002F",[167],"Dutch Data Protection Authority Warns About AI Chatbots and Personal Data Breaches",[33,2013,2014],{},[163,2015,2018],{"href":2016,"rel":2017},"https:\u002F\u002Fwww.vlaanderen.be\u002Fvlaamse-toezichtcommissie\u002Fdomeinen\u002Fai-en-persoonsgegevens",[167],"AI en Persoonsgegevens – Vlaamse Overheid",[33,2020,2021],{},[163,2022,2025],{"href":2023,"rel":2024},"https:\u002F\u002Fwww.securex.be\u002Fnl\u002Flex4you\u002Fwerkgever\u002Fnieuws\u002Fwat-betekent-de-ai-act-voor-werkgevers-en-werknemers",[167],"Wat Betekent de AI Act voor Werkgevers en Werknemers? – Securex",[33,2027,2028],{},[163,2029,2032],{"href":2030,"rel":2031},"https:\u002F\u002Feconomie.fgov.be\u002Fnl\u002Fthemas\u002Fondernemingen\u002Fai-act\u002Fu-gebruikt-ai-uw-onderneming",[167],"U Gebruikt AI in Uw Onderneming – FOD Economie",[33,2034,2035],{},[163,2036,2039],{"href":2037,"rel":2038},"https:\u002F\u002Fwww.gegevensbeschermingsautoriteit.be\u002Fpublications\u002Finformatiebrochure-over-artificiele-intelligentiesystemen-en-de-avg.pdf",[167],"Artificiële Intelligentiesystemen en de AVG – Gegevensbeschermingsautoriteit",[33,2041,2042],{},[163,2043,2046],{"href":2044,"rel":2045},"https:\u002F\u002F2b-advice.com\u002Fen\u002F2025\u002F09\u002F10\u002Fwhat-is-shadow-ki-in-the-company-and-how-can-it-be-uncovered\u002F",[167],"What Is Shadow AI and How Can It Be Detected? – 2B Advice",[33,2048,2049],{},[163,2050,2053],{"href":2051,"rel":2052},"https:\u002F\u002Fhumanfirewall.io\u002Fcase-study-on-samsungs-chatgpt-incident\u002F",[167],"A Case Study on Samsung’s ChatGPT Incident – HumanFirewall",[33,2055,2056],{},[163,2057,2060],{"href":2058,"rel":2059},"https:\u002F\u002Fincidentdatabase.ai\u002Fcite\u002F768\u002F",[167],"Incident 768: ChatGPT Implicated in Samsung Data Leak – Incident Database",[33,2062,2063],{},[163,2064,2067],{"href":2065,"rel":2066},"https:\u002F\u002Fprompt.security\u002Fblog\u002F8-real-world-incidents-related-to-ai",[167],"8 Real World Incidents Related to AI – Prompt Security",[33,2069,2070],{},[163,2071,2074],{"href":2072,"rel":2073},"https:\u002F\u002Fwww.staple.ai\u002Fblog\u002Fwhy-shadow-ai-is-the-biggest-compliance-threat-no-ones-talking-about",[167],"Why Shadow AI Is the Biggest Compliance Threat – Staple AI",[33,2076,2077],{},[163,2078,2081],{"href":2079,"rel":2080},"https:\u002F\u002Fwww.reddit.com\u002Fr\u002FITManagers\u002Fcomments\u002F1p9ktlm\u002Fai_usage_by_employees_policy_and_compliancegdpr\u002F",[167],"AI Usage by Employees: Policy and GDPR Compliance – Reddit IT Managers",[33,2083,2084],{},[163,2085,2088],{"href":2086,"rel":2087},"https:\u002F\u002Fgdprwise.eu\u002Fquestions-and-answers\u002Fai-permissible-use-policy\u002F?lang=en",[167],"AI Permissible Use Policy – GDPRWise",[33,2090,2091],{},[163,2092,2095],{"href":2093,"rel":2094},"https:\u002F\u002Flawandmore.eu\u002Fblog\u002Fusing-ai-in-your-dutch-business-gdpr-and-compliance-risks-explained\u002F",[167],"Using AI in Your Dutch Business: GDPR and Compliance Risks – Law & More",[33,2097,2098],{},[163,2099,2102],{"href":2100,"rel":2101},"https:\u002F\u002Fwww.intwo.cloud\u002Fnews-blog\u002Fat-the-crossroads-of-ai-chatgpt-openai-or-azure-openai\u002F",[167],"At the Crossroads of AI: ChatGPT OpenAI or Azure OpenAI? – Intwo",[33,2104,2105],{},[163,2106,2109],{"href":2107,"rel":2108},"https:\u002F\u002Fwww.ictrechtswijzer.be\u002Fen\u002Fai-chatbots-and-ai-agents-legal-concerns\u002F",[167],"AI Chatbots and AI Agents: Legal Concerns Under Belgian Law – ICT",{"title":285,"searchDepth":673,"depth":673,"links":2111},[2112,2113,2114,2115,2116,2117,2118,2119,2120,2121,2122,2123],{"id":1166,"depth":676,"text":1169},{"id":1194,"depth":676,"text":1197},{"id":1214,"depth":676,"text":1217},{"id":1329,"depth":676,"text":1332},{"id":1349,"depth":676,"text":1352},{"id":1491,"depth":676,"text":1494},{"id":1526,"depth":676,"text":1529},{"id":1561,"depth":676,"text":1564},{"id":1607,"depth":676,"text":1610},{"id":1619,"depth":676,"text":1622},{"id":1651,"depth":676,"text":1654},{"id":1706,"depth":676,"text":1709},[688,689,2125,1149,2126,2127,2128,2129,2130,2131,2132,2133,960,2134,2135,2136,2137,2138],"Trending","AI Act","Antropomorfisme","AVG","Benelux","ChatGPT","Cybersecurity","Data Governance","Data Security","GDPR","Generatieve AI","Microsoft Copilot","Privacy","Shadow AI","2026-01-29T13:51:11","Privacyrisico’s, Psychologische Valkuilen en de Operationele Realiteit van Generatieve AI in de Benelux De recente verschijning van een frontpage-artikel over ons bedrijf in het gerespecteerde dagblad “De Tijd” heeft onze zichtbaarheid aanzienlijk vergroot, wat de aanleiding is voor dit artikel. Deze mediabelangstelling, gecombineerd met de talrijke uitnodigingen voor spreekbeurten die we hebben ontvangen, fungeert als ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fheaderimage.webp",{},"https:\u002F\u002Feliovp.com\u002Fprivacy-is-geen-it-probleem-meer-het-is-een-strategische-prioriteit\u002F","\u002Fblog\u002Fprivacy-is-geen-it-probleem-meer-het-is-een-strategische-prioriteit",{"title":1161,"description":2140},"privacy-is-geen-it-probleem-meer-het-is-een-strategische-prioriteit","blog\u002Fprivacy-is-geen-it-probleem-meer-het-is-een-strategische-prioriteit","ym7EIl4kWVdU7Zd4xysEx1ImYAulEKAyrzmNbtOrS7M",{"id":2150,"title":2151,"body":2152,"categories":3474,"date":3483,"description":3484,"extension":699,"image":3485,"meta":3486,"navigation":702,"originalUrl":3487,"path":3488,"seo":3489,"slug":3490,"stem":3491,"__hash__":3492},"blog\u002Fblog\u002Fitsme-bij-ons-is-het-its-not-me-en-dit-is-waarom.md","Itsme? Bij ons is het “it’s not me”, en dit is waarom.",{"type":7,"value":2153,"toc":3464},[2154,2159,2165,2168,2171,2174,2206,2209,2215,2220,2227,2233,2236,2242,2345,2348,2363,2369,2372,2379,2385,2392,2403,2410,2417,2423,2426,2446,2449,2455,2460,2467,2473,2483,2486,2518,2521,2527,2534,2540,2551,2581,2588,2594,2609,2615,2620,2635,2641,2652,2658,2777,2783,2790,2793,2822,2840,2846,2849,2852,2872,2875,2881,2886,2892,2898,2901,2944,2950,2965,2985,2992,2998,3003,3006,3017,3024,3032,3039,3045,3050,3053,3058,3090,3104,3110,3116,3223,3229,3304,3309,3315],[18,2155,2156],{},[283,2157],{"alt":285,"src":2158},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Ffacade.webp",[10,2160,2162],{"id":2161},"_1-inleiding-de-strategische-noodzaak-van-weigering",[14,2163,2164],{},"1. Inleiding: De Strategische Noodzaak van Weigering",[18,2166,2167],{},"In het hedendaagse digitale landschap wordt de keuze voor een identiteit leverancier (IdP) vaak gereduceerd tot een discussie over User Experience (UX) en conversieratio’s. Deze reductionistische benadering verhult echter de diepgaande strategische, juridische en operationele risico’s die gepaard gaan met het uitbesteden van de “Sleutels tot het Koninkrijk”, de authenticatie laag, aan externe partijen. Dit rapport dient als de definitieve interne rechtvaardiging voor ons bedrijfsbeleid om de adoptie van het “itsme” (Belgian Mobile ID) ecosysteem voor onze werknemers en interne operaties af te wijzen.",[18,2169,2170],{},"Het heersende narratief in de Belgische markt positioneert itsme als een quasi-overheid nutsvoorziening, een onschuldig, digitaal equivalent van de fysieke identiteitskaart. Onze forensische analyse onthult echter een heel andere realiteit. We hebben niet te maken met een soevereine openbare dienst, maar met een privaat commercieel consortium dat gedomineerd wordt door bank- en telecomgiganten, entiteiten die de meest intieme details van consumentengedrag al te gelde maken. Door hen de authenticatie laag te overhandigen, zouden we hen het laatste stukje van de surveillance-puzzel geven: de mogelijkheid om elke digitale interactie te volgen, dwars door sectoren als financiën, gezondheidszorg en overheidsdiensten heen.",[18,2172,2173],{},"Onze weigering is gebaseerd op drie ononderhandelbare pijlers van onze risicobereidheid:",[114,2175,2176,2182,2188],{},[33,2177,2178,2181],{},[14,2179,2180],{},"Data Soevereiniteit & Infrastructuur Integriteit:"," Wij verwerpen de opslag van gevoelige identiteit metadata op door de VS gecontroleerde cloud infrastructuur (AWS), die onderworpen blijft aan extraterritoriale surveillance wetten, ongeacht de fysieke locatie in België.",[33,2183,2184,2187],{},[14,2185,2186],{},"Privacy tegenover Commerciële Aggregators:"," Wij weigeren de data-ecosystemen van banken en telco’s te voeden, die al overmatig veel inzicht hebben in het privéleven van ons personeel (metaforisch: “de kleur van hun ondergoed”).",[33,2189,2190,2193,2194,2197,2198,2201,2202,2205],{},[14,2191,2192],{},"Metadata-minimalisatie:"," Wij beschouwen de gecentraliseerde bewaartermijn van 10 jaar voor transactielogs, waarin wordt vastgelegd ",[630,2195,2196],{},"waar",", ",[630,2199,2200],{},"wanneer"," en ",[630,2203,2204],{},"hoe vaak"," een gebruiker inlogt, als een disproportionele surveillance capaciteit die indruist tegen de principes van ‘privacy-by-design’.",[18,2207,2208],{},"Dit rapport ontleedt deze risico’s met rigoureuze details, vertrouwend op technische documentatie, bedrijfsdeponeringen en wetteksten om ons standpunt te onderbouwen dat gemak niet mag worden gekocht ten koste van digitale autonomie.",[10,2210,2212],{"id":2211},"_2-de-mythe-van-het-belgische-nutsbedrijf-bedrijfsstructuur-en-buitenlands-kapitaal",[14,2213,2214],{},"2. De Mythe van het “Belgische” Nutsbedrijf: Bedrijfsstructuur en Buitenlands Kapitaal",[18,2216,2217],{},[283,2218],{"alt":285,"src":2219},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fmythenuts.webp",[18,2221,2222,2223,2226],{},"Om het risicoprofiel van itsme te begrijpen, moet men eerst de misvatting ontkrachten dat het een staatsbedrijf is. Hoewel het overheids accreditatie geniet, is ",[14,2224,2225],{},"Belgian Mobile ID NV"," een besloten vennootschap met beperkte aansprakelijkheid. De eigendomsstructuur is een “Who’s Who” van de data-economie, sectoren die persoonsgegevens historisch gezien beschouwen als een grondstof voor extractie in plaats van een goed te beschermen goed.",[1230,2228,2230],{"id":2229},"_21-de-aandeelhouders-een-consortium-van-data-aggregators",[14,2231,2232],{},"2.1. De Aandeelhouders: Een Consortium van Data-aggregators",[18,2234,2235],{},"De entiteit “Belgian Mobile ID” werd in 2017 opgericht als een joint venture tussen de “Grote Vier” banken van België en de drie grote mobiele netwerkoperators (MNO’s).",[1230,2237,2239],{"id":2238},"tabel-1-samenstelling-aandeelhouders-en-strategische-conflicten",[14,2240,2241],{},"Tabel 1: Samenstelling Aandeelhouders en Strategische Conflicten",[298,2243,2244,2273],{},[301,2245,2246],{},[304,2247,2248,2253,2258,2263,2268],{},[307,2249,2250],{},[14,2251,2252],{},"Sector Aandeelhouder",[307,2254,2255],{},[14,2256,2257],{},"Entiteiten",[307,2259,2260],{},[14,2261,2262],{},"Primair Bedrijfsmodel",[307,2264,2265],{},[14,2266,2267],{},"Reeds Beschikbare Data",[307,2269,2270],{},[14,2271,2272],{},"Belangenconflict met Identiteit",[316,2274,2275,2301,2323],{},[304,2276,2277,2282,2285,2288,2291],{},[321,2278,2279],{},[14,2280,2281],{},"Bankwezen",[321,2283,2284],{},"Belfius, BNP Paribas Fortis, ING België, KBC Bank",[321,2286,2287],{},"Financiële Diensten, Krediet, Verzekering",[321,2289,2290],{},"Transactiegeschiedenis, uitgavenpatronen, schuldniveaus, vermogenswaarde.",[321,2292,2293,2296,2297,2300],{},[14,2294,2295],{},"Aggregatie Risico:"," Door authenticatie data toe te voegen, kunnen banken ",[630,2298,2299],{},"intentie"," zien (bijv. inloggen bij een concurrent) voordat er een transactie plaatsvindt.",[304,2302,2303,2308,2311,2314,2317],{},[321,2304,2305],{},[14,2306,2307],{},"Telecommunicatie",[321,2309,2310],{},"Proximus, Orange Belgium, Telenet",[321,2312,2313],{},"Connectiviteit, Media, Reclame",[321,2315,2316],{},"Real-time locatie (gsm-mast), communicatie metadata, surfgedrag.",[321,2318,2319,2322],{},[14,2320,2321],{},"Surveillance Risico:"," Telco’s controleren de fysieke laag. Het koppelen van de SIM (identiteit) aan de App (authenticatie) creëert een naadloze trackingloop.",[304,2324,2325,2330,2333,2336,2339],{},[321,2326,2327],{},[14,2328,2329],{},"Overheidsinvestering",[321,2331,2332],{},"FPIM (Federale Participatie- en Investeringsmaatschappij)",[321,2334,2335],{},"Staatsinvesteringen",[321,2337,2338],{},"Publiek Beleid",[321,2340,2341,2344],{},[14,2342,2343],{},"Minderheidsbelang:"," Met slechts 20% aandelen verworven in 2021 ^1, mist de staat operationele controle tegenover de commerciële meerderheid.",[18,2346,2347],{},"De betrokkenheid van de Federale Participatie- en Investeringsmaatschappij (FPIM) wordt vaak aangehaald om de angst voor privatisering weg te nemen.3 Een belang van 20% geeft echter geen doorslaggevende zeggenschap. De strategische koers, technologie-inkoop en het data-inkomensbeleid worden gedreven door de commerciële meerderheid, de banken en telco’s.4",[18,2349,2350,2351,2354,2355,2358,2359,2362],{},"Deze structuur is kritiek omdat deze entiteiten functioneren als ",[14,2352,2353],{},"Data-aggregators",". Banken opereren onder Know Your Customer (KYC) regelgeving die hen diep inzicht geeft in het financiële leven van een burger. Door de identiteit leverancier te controleren die wordt gebruikt voor toegang tot ",[630,2356,2357],{},"niet-bancaire"," diensten (zoals gezondheidszorg, belastingen of juridische diensten), krijgen deze commerciële entiteiten zicht op de ",[630,2360,2361],{},"totaliteit"," van het digitale bestaan van een gebruiker. Ze weten niet alleen dat je geld hebt uitgegeven bij een apotheek (bankdata), maar ook dat je hebt ingelogd op een gespecialiseerd oncologie portaal (itsme-metadata). De aggregatie van deze datasets creëert een “super-profiel” van ongekende granulariteit.",[1230,2364,2366],{"id":2365},"_22-de-buitenlandse-invloed-telenet-en-liberty-global",[14,2367,2368],{},"2.2. De Buitenlandse Invloed: Telenet en Liberty Global",[18,2370,2371],{},"Een grondige analyse van het “Belgische” karakter van het consortium onthult aanzienlijke invloed van buitenlands kapitaal, wat geopolitieke risico’s introduceert die onverenigbaar zijn met ware data soevereiniteit.",[18,2373,2374,2375,2378],{},"Telenet, een oprichtende aandeelhouder en belangrijke mobiele netwerkoperator in het schema, is voor ",[14,2376,2377],{},"100% eigendom van Liberty Global",", een multinationaal telecommunicatie conglomeraat met hoofdkantoor in het VK, de VS en Bermuda.5 Telenet is geen onafhankelijke Belgische agent; het is een dochteronderneming die onderworpen is aan de strategische en juridische imperatieven van haar moederbedrijf.",[1230,2380,2382],{"id":2381},"_221-het-transparantierapport-een-pijplijn-naar-overheid-surveillance",[14,2383,2384],{},"2.2.1. Het Transparantierapport: Een Pijplijn naar Overheid Surveillance",[18,2386,2387,2388,2391],{},"Ons onderzoek naar het ondernemingsbestuur van Liberty Global onthult een robuust mechanisme voor gegevensverstrekking aan overheidsinstanties. Liberty Global’s eigen transparantie rapportage voor 2024 stelt dat het ",[14,2389,2390],{},"46.325 overheidsonderzoeken"," ontving voor Telenet-klanten in België.",[18,2393,2394,2395,2398,2399,2402],{},"Cruciaal is dat ",[14,2396,2397],{},"90% van deze verzoeken resulteerde in de openbaarmaking van klantgegevens",".8 Hoewel deze onthullingen worden gekaderd binnen de Belgische nationale regelgeving, geeft het enorme volume, gemiddeld meer dan 126 verzoeken ",[630,2400,2401],{},"per dag",", aan dat Telenet functioneert als een uiterst efficiënte datapijplijn naar staatsveiligheidsdiensten.",[18,2404,2405,2406,2409],{},"Het risico voor ons, als bedrijf dat Itsme weigert, ligt in de ",[14,2407,2408],{},"bedrijfs sluier (corporate veil)",". Liberty Global is een bedrijf met diepe banden met de Amerikaanse en Britse markten.5 Als het moederbedrijf (Liberty Global) onderworpen wordt aan een dataverzoek uit een buitenlandse jurisdictie (bijv. de Verenigde Staten) met betrekking tot de activa van zijn dochteronderneming, kunnen de interne bedrijfs barrières poreus blijken. Het Amerikaanse rechtssysteem, met name via mechanismen zoals de CLOUD Act (besproken in Sectie 3), negeert vaak de afzonderlijke rechtspersoonlijkheid van buitenlandse dochterondernemingen als operationele controle kan worden vastgesteld.",[18,2411,2412,2413,2416],{},"Bovendien geeft Liberty Global toe dat het voor zijn ",[630,2414,2415],{},"andere"," activiteiten (buiten België) “niet is toegestaan het aantal overheids verzoeken openbaar te maken” vanwege nationale regelgevende kaders.8 Deze ondoorzichtigheid is onaanvaardbaar. Wij kunnen onze bedrijfs identiteitsinfrastructuur niet toevertrouwen aan een consortium partner wiens moederbedrijf in een “black box” opereert wat betreft “overheidssurveillance” in andere jurisdicties.",[1230,2418,2420],{"id":2419},"_23-internationale-verstrengeling-van-andere-partners",[14,2421,2422],{},"2.3. Internationale Verstrengeling van Andere Partners",[18,2424,2425],{},"Het label “Belgisch” wordt verder uitgehold door de andere partners:",[30,2427,2428,2434,2440],{},[33,2429,2430,2433],{},[14,2431,2432],{},"BNP Paribas Fortis:"," Een dochteronderneming van de Franse bankgigant BNP Paribas (voor 100% eigendom). Frankrijk heeft zijn eigen agressieve inlichtingen wetten (Loi Renseignement).",[33,2435,2436,2439],{},[14,2437,2438],{},"ING België:"," Een dochteronderneming van de Nederlandse ING Groep.",[33,2441,2442,2445],{},[14,2443,2444],{},"Orange Belgium:"," Gecontroleerd door het Franse Orange S.A. (meer dan 76% van de stemrechten).",[18,2447,2448],{},"Het itsme-ecosysteem is dus feitelijk een joint venture van Amerikaans\u002FBrits (Liberty Global), Frans (Orange, BNP) en Nederlands (ING) kapitaal. In een geopolitieke crisis of een handelsgeschil over digitale diensten (bijv. VS vs. EU-tech regulering), zal de loyaliteit van deze moederbedrijven bij hun thuis jurisdicties en aandeelhouders liggen, niet noodzakelijkerwijs bij de privacy van Belgische burgers of onze werknemers.",[10,2450,2452],{"id":2451},"_3-infrastructuur-kwetsbaarheid-de-aws-afhankelijkheid-en-het-einde-van-soevereiniteit",[14,2453,2454],{},"3. Infrastructuur Kwetsbaarheid: De AWS-afhankelijkheid en het Einde van Soevereiniteit",[18,2456,2457],{},[283,2458],{"alt":285,"src":2459},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Finfrakwetsbaarheid.webp",[18,2461,2462,2463,2466],{},"De meest directe reden voor diskwalificatie voor onze organisatie is de beslissing van Belgian Mobile ID om haar infrastructuur te migreren van on-premise datacenters naar de publieke cloud, specifiek ",[14,2464,2465],{},"Amazon Web Services (AWS)",".9 Hoewel deze stap vaak wordt gerechtvaardigd door “schaalbaarheid” en “modernisering” 11, compromitteert het fundamenteel het concept van data-residentie.",[1230,2468,2470],{"id":2469},"_31-de-juridische-realiteit-de-amerikaanse-cloud-act",[14,2471,2472],{},"3.1. De Juridische Realiteit: De Amerikaanse CLOUD Act",[18,2474,2475,2476,2479,2480,46],{},"De ",[14,2477,2478],{},"Clarifying Lawful Overseas Use of Data (CLOUD) Act"," van 2018 is een Amerikaanse federale wet die de bescherming van data locatie uitholt. Het geeft Amerikaanse wetshandhavingsinstanties expliciet de bevoegdheid om in de VS gevestigde technologie providers (zoals Amazon) te dwingen gegevens op hun servers vrij te geven, ",[14,2481,2482],{},"ongeacht of die gegevens in de Verenigde Staten of op buitenlandse bodem zijn opgeslagen",[18,2484,2485],{},"Dit is geen theoretisch risico. Het is wetgeving.",[30,2487,2488,2498,2504],{},[33,2489,2490,2493,2494,2497],{},[14,2491,2492],{},"Jurisdictie over de Provider, niet de Data:"," De CLOUD Act claimt jurisdictie over het ",[630,2495,2496],{},"bedrijf"," (Amazon.com Inc. en zijn dochterondernemingen). Als Amazon “bezit, voogdij of controle” heeft over de data, moeten ze deze overhandigen, zelfs als de server fysiek in Brussel, Parijs of Dublin staat.",[33,2499,2500,2503],{},[14,2501,2502],{},"Conflict met GDPR:"," Dit creëert een direct conflict met de AVG (GDPR), specifiek Artikel 48, dat overdrachten op basis van buitenlandse rechterlijke bevelen beperkt tenzij er een rechtshulpverdrag (MLAT) is. De CLOUD Act omzeilt echter vaak het MLAT-proces.12",[33,2505,2506,2509,2510,2513,2514,2517],{},[14,2507,2508],{},"De “Encryptie”-verdediging is Gebrekkig:"," AWS betoogt vaak dat ze geen data kunnen produceren die ze niet kunnen lezen (d.w.z. als het versleuteld is). Echter, als de encryptiesleutels worden beheerd via ",[14,2511,2512],{},"AWS Key Management Service (KMS)",", of als de sleutels ooit in het geheugen van de compute-instances aanwezig zijn om het authenticatie verzoek te verwerken, bezit AWS ",[630,2515,2516],{},"technisch"," de mogelijkheid om te ontsleutelen. Onder een dagvaarding zouden ze gedwongen kunnen worden die capaciteit te gebruiken.13",[18,2519,2520],{},"Voor ons bedrijf is dit een rode lijn. Wij kunnen niet beweren de gegevens van onze werknemers te beschermen als die gegevens op een server staan die een buitenlandse rechter in Virginia in beslag kan nemen zonder medeweten of toestemming van de Belgische autoriteiten.",[1230,2522,2524],{"id":2523},"_32-de-local-zone-misleiding-brussel-is-een-buitenpost-geen-vesting",[14,2525,2526],{},"3.2. De “Local Zone” Misleiding: Brussel is een Buitenpost, Geen Vesting",[18,2528,2529,2530,2533],{},"Belgian Mobile ID en AWS hebben de lancering van de ",[14,2531,2532],{},"AWS Local Zone in Brussel"," (eu-west-3-bru-1a) zwaar gepromoot als oplossing voor data-residentie.9 Ze suggereren dat data “in België blijft”. Onze technische analyse onthult dat dit een gevaarlijke versimpeling is.",[1230,2535,2537],{"id":2536},"_321-technische-afhankelijkheid-van-de-moeder-regio",[14,2538,2539],{},"3.2.1. Technische Afhankelijkheid van de Moeder Regio",[18,2541,2542,2543,2546,2547,2550],{},"Een AWS Local Zone is geen onafhankelijke cloudregio. Het is een verlengstuk van een ",[14,2544,2545],{},"Parent Region",". Voor de Brusselse Local Zone is de Parent Region ",[14,2548,2549],{},"Europa (Parijs)"," (eu-west-3).",[30,2552,2553,2559,2569],{},[33,2554,2555,2558],{},[14,2556,2557],{},"Control Plane:"," De “hersenen” van de operatie, de Identity and Access Management (IAM) systemen, de API-endpoints die de infrastructuur aansturen en de configuratiedatabases, bevinden zich in de Parent Region (Parijs). Een storing of compromittering in Parijs heeft impact op Brussel.",[33,2560,2561,2564,2565,2568],{},[14,2562,2563],{},"Data Duurzaamheid en Replicatie:"," Kritieke diensten zoals ",[14,2566,2567],{},"Amazon S3"," (Simple Storage Service) zijn ontworpen voor extreme duurzaamheid, wat typisch wordt bereikt door data te repliceren over meerdere Availability Zones (AZ’s). Aangezien de Brusselse Local Zone vaak een enkele logische locatie is, vereist het bereiken van hoge duurzaamheid vaak replicatie terug naar de Parent Region (Parijs).14",[33,2570,2571,2574,2575,2578,2579,46],{},[14,2572,2573],{},"Snapshot Opslag:"," AWS-documentatie stelt expliciet dat voor diensten zoals ",[14,2576,2577],{},"Amazon EBS"," (Elastic Block Store), snapshots (back-ups) die in een Local Zone worden gemaakt, worden opgeslagen in de ",[14,2580,2545],{},[18,2582,2583,2584,2587],{},"Daarom, zelfs als de “rekenkracht” (compute) in Brussel plaatsvindt, vloeit de ",[630,2585,2586],{},"data",", inclusief back-ups, snapshots en potentieel opslag objecten, naar Frankrijk. Dit onderwerpt de data aan Franse jurisdictie naast de Amerikaanse jurisdictie (via de CLOUD Act). De claim “Belgische data blijft in België” is technisch poreus en juridisch ontoereikend.",[1230,2589,2591],{"id":2590},"_33-het-vendor-lock-in-risico",[14,2592,2593],{},"3.3. Het Vendor Lock-In Risico",[18,2595,2596,2597,2600,2601,2604,2605,2608],{},"Door een nationaal identiteitsschema te migreren naar een propriëtaire Amerikaanse cloud, heeft Belgian Mobile ID een ",[14,2598,2599],{},"Single Point of Failure (SPOF)"," gecreëerd. Als AWS een wereldwijde storing heeft, of als Amazon besluit de dienst van het platform te halen om commerciële of geopolitieke redenen, stort de Belgische digitale identiteitsinfrastructuur in. Ons bedrijf pleit voor ",[14,2602,2603],{},"multi cloud"," of ",[14,2606,2607],{},"hybride-cloud"," weerbaarheid. Vertrouwen op één enkele leverancier voor de identiteit van miljoenen burgers schendt basisprincipes van operationele continuïteit.11",[10,2610,2612],{"id":2611},"_4-het-panopticum-van-metadata-het-kleur-van-het-ondergoed-probleem",[14,2613,2614],{},"4. Het Panopticum van Metadata: Het “Kleur van het Ondergoed” Probleem",[18,2616,2617],{},[283,2618],{"alt":285,"src":2619},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fonderbroek.webp",[18,2621,2622,2623,2626,2627,2630,2631,2634],{},"Ons interne mandaat vraagt zich af waarom we banken en telco’s ",[630,2624,2625],{},"meer"," data zouden moeten geven, terwijl ze al “de kleur van ons ondergoed” kennen, een metafoor voor de intieme financiële en gedragsgegevens die ze al bezitten. Het itsme-ecosysteem verergert dit door een gecentraliseerd ",[14,2628,2629],{},"metadata-repository"," te creëren dat de ",[630,2632,2633],{},"context"," van elke digitale interactie volgt.",[1230,2636,2638],{"id":2637},"_41-metadata-is-data",[14,2639,2640],{},"4.1. “Metadata is Data”",[18,2642,2643,2644,2647,2648,2651],{},"Belgian Mobile ID stelt dat ze de ",[630,2645,2646],{},"inhoud"," van de documenten die je ondertekent of de details van je banktransactie niet zien (wat technisch waar is via Hash Signing, later besproken). Ze loggen echter expliciet de ",[14,2649,2650],{},"metadata"," van de interactie.16",[1230,2653,2655],{"id":2654},"tabel-2-de-anatomie-van-het-itsme-audit-trail",[14,2656,2657],{},"Tabel 2: De Anatomie van het itsme Audit Trail",[298,2659,2660,2684],{},[301,2661,2662],{},[304,2663,2664,2669,2674,2679],{},[307,2665,2666],{},[14,2667,2668],{},"Dataveld",[307,2670,2671],{},[14,2672,2673],{},"Beschrijving",[307,2675,2676],{},[14,2677,2678],{},"Bewaartermijn",[307,2680,2681],{},[14,2682,2683],{},"Privacyrisico",[316,2685,2686,2704,2725,2742,2759],{},[304,2687,2688,2693,2696,2701],{},[321,2689,2690],{},[14,2691,2692],{},"Tijdstempel",[321,2694,2695],{},"Exacte datum en tijd van de transactie.",[321,2697,2698],{},[14,2699,2700],{},"10 Jaar",[321,2702,2703],{},"Onthult dagelijkse routines, slapeloosheid (late logins), werktijden.",[304,2705,2706,2711,2714,2718],{},[321,2707,2708],{},[14,2709,2710],{},"Service Provider (SP)",[321,2712,2713],{},"De entiteit die de identiteit opvraagt (bv. “KBC Bank”, “UZ Leuven”, “FOD Financiën”).",[321,2715,2716],{},[14,2717,2700],{},[321,2719,2720,2721,2724],{},"Onthult de ",[630,2722,2723],{},"aard"," van de activiteit van de gebruiker (Bankieren, Medisch, Overheid, Juridisch).",[304,2726,2727,2732,2735,2739],{},[321,2728,2729],{},[14,2730,2731],{},"Actietype",[321,2733,2734],{},"Inloggen, Transactie Bevestigen, Document Ondertekenen.",[321,2736,2737],{},[14,2738,2700],{},[321,2740,2741],{},"Differentieert tussen passief checken en actieve verbintenis.",[304,2743,2744,2749,2752,2756],{},[321,2745,2746],{},[14,2747,2748],{},"Toesteltelemetrie",[321,2750,2751],{},"IMEI, besturingssysteem, toestel model.",[321,2753,2754],{},[14,2755,2700],{},[321,2757,2758],{},"Vingerafdruk van het specifieke apparaat; maakt tracking van toestelwissels mogelijk.",[304,2760,2761,2766,2769,2774],{},[321,2762,2763],{},[14,2764,2765],{},"Locatie-indicator",[321,2767,2768],{},"“Security Data” omvat land locatie (MCC).",[321,2770,2771,2773],{},[14,2772,2700],{}," ^18",[321,2775,2776],{},"Volgt internationale reis- en locatiegeschiedenis.",[1230,2778,2780],{"id":2779},"_42-gedragsprofilering-via-metadata-aggregatie",[14,2781,2782],{},"4.2. Gedragsprofilering via Metadata-aggregatie",[18,2784,2785,2786,2789],{},"Het gevaar schuilt niet in één logregel, maar in de ",[14,2787,2788],{},"aggregatie"," van tien jaar aan deze data. Een bewaartermijn van 10 jaar is grotesk disproportioneel voor een simpele authenticatiedienst.",[18,2791,2792],{},"Overweeg het verhaal dat kan worden geconstrueerd puur uit de metadata die itsme toegeeft op te slaan:",[30,2794,2795,2804,2813],{},[33,2796,2797,2800,2801],{},[14,2798,2799],{},"Scenario A (Medisch):"," Een gebruiker logt eens per jaar in op een algemeen ziekenhuisportaal (Service Provider: “UZ Leuven”). Plotseling neemt de frequentie toe tot wekelijks. Vervolgens loggen ze in op een gespecialiseerde oncologie dienst. ",[630,2802,2803],{},"Inferentie: Ernstige ziekte.",[33,2805,2806,2809,2810],{},[14,2807,2808],{},"Scenario B (Financieel):"," Een gebruiker logt dagelijks in bij zijn primaire bank. Plotseling logt hij in één week tijd in bij drie verschillende consumenten kredietverstrekkers en een schuldbemiddelingsdienst. ",[630,2811,2812],{},"Inferentie: Financiële nood.",[33,2814,2815,2818,2819],{},[14,2816,2817],{},"Scenario C (Juridisch\u002FPersoonlijk):"," Een gebruiker logt in bij een notaris of het portaal van een echtscheidingsadvocaat. ",[630,2820,2821],{},"Inferentie: Grote levensverandering\u002Frechtszaak.",[18,2823,2824,2825,2828,2829,2832,2833,2835,2836,2839],{},"Deze metadata vormt een high-fidelity kaart van een mensenleven. Het onthult ",[630,2826,2827],{},"wat"," je doet, ",[630,2830,2831],{},"met wie"," je interageert en ",[630,2834,2204],{},". Voor een consortium van banken en verzekeraars (die risico’s prijzen) en telco’s (die advertenties verkopen), is deze dataset van onschatbare waarde. Hoewel ze beweren strikte scheidingen te hanteren, bestaat de ",[630,2837,2838],{},"technische mogelijkheid"," om deze data te ontginnen binnen de gecentraliseerde servers.",[1230,2841,2843],{"id":2842},"_43-de-security-data-en-locatie-tracking",[14,2844,2845],{},"4.3. De “Security Data” en Locatie Tracking",[18,2847,2848],{},"Het privacybeleid vermeldt expliciet het verzamelen van “Security Data”, waaronder de Mobile Country Code (MCC) en de Mobile Network Code (MNC).18 Dit wordt afgeleid van de SIM-kaart.",[18,2850,2851],{},"Omdat de MNO’s (Proximus, Orange, Telenet) aandeelhouders en “SIM Controllers” zijn 17, is er een directe brug tussen de fysieke netwerklaag en de applicatielaag.",[30,2853,2854,2860,2866],{},[33,2855,2856,2859],{},[14,2857,2858],{},"Het Telco-beeld:"," Weet dat de gebruiker verbonden is met Celmast ID 12345 om 10:00.",[33,2861,2862,2865],{},[14,2863,2864],{},"Het itsme-beeld:"," Weet dat de gebruiker om 10:00 inlogde bij Tax-on-Web.",[33,2867,2868,2871],{},[14,2869,2870],{},"Het Gecombineerde Beeld:"," “Gebruiker X was op Locatie Y toen hij zijn belastingen indiende.”",[18,2873,2874],{},"Deze triangulatie capaciteit verandert de identiteits-app in een potentiële locatie tracker. Voor een bedrijf dat de fysieke veiligheid en privacy van zijn werknemers waardeert, is het toestaan dat een derde partij-consortium een logboek van 10 jaar bijhoudt van hun locatie-gecorreleerde authenticatie-events een absolute ‘no-go’.",[10,2876,2878],{"id":2877},"_5-veiligheidstheater-het-hash-signing-gebrek",[14,2879,2880],{},"5. Veiligheidstheater: Het “Hash Signing” Gebrek",[18,2882,2883],{},[283,2884],{"alt":285,"src":2885},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fhash.webp",[18,2887,2888,2889,46],{},"Een van de belangrijkste verkoopargumenten van itsme is de “Gekwalificeerde Elektronische Handtekening” (QES) en het principe van “What You See Is What You Sign” (WYSIWYS). Ze beweren dat je altijd ziet wat je ondertekent. Technisch gezien is deze claim misleidend en introduceert het een kritiek veiligheidsgat bekend als de ",[14,2890,2891],{},"Blind Signing Kwetsbaarheid",[1230,2893,2895],{"id":2894},"_51-hoe-hash-signing-werkt",[14,2896,2897],{},"5.1. Hoe Hash Signing Werkt",[18,2899,2900],{},"In de huidige itsme-implementatie (Hash Signing variant) 19:",[114,2902,2903,2909,2915,2924,2933,2939],{},[33,2904,2905,2908],{},[14,2906,2907],{},"De Gebruiker"," bezoekt een website op zijn computer.",[33,2910,2911,2914],{},[14,2912,2913],{},"De Service Provider"," toont de documentinhoud aan de gebruiker.",[33,2916,2917,2919,2920,2923],{},[14,2918,2913],{}," berekent een ",[14,2921,2922],{},"cryptografische hash"," (een wiskundige vingerafdruk) van het document.",[33,2925,2926,2928,2929,2932],{},[14,2927,2913],{}," stuurt ",[630,2930,2931],{},"alleen de hash"," naar de itsme-server.19",[33,2934,2935,2938],{},[14,2936,2937],{},"De itsme App"," opent op de telefoon van de gebruiker. Het toont een generiek bericht: “Teken document voor X”.",[33,2940,2941,2943],{},[14,2942,2907],{}," voert zijn pincode in en autoriseert de handtekening van die hash.",[1230,2945,2947],{"id":2946},"_52-de-breuk-in-de-keten",[14,2948,2949],{},"5.2. De Breuk in de Keten",[18,2951,2952,2953,2956,2957,2960,2961,2964],{},"De fatale fout hier is dat de ",[14,2954,2955],{},"itsme-app het document nooit ziet",".19 Het ziet ",[630,2958,2959],{},"alleen"," de hash. Daarom kan de itsme-app ",[630,2962,2963],{},"niet verifiëren"," aan de gebruiker dat de hash die ze ondertekenen daadwerkelijk overeenkomt met het document dat ze op hun computerscherm zagen.",[30,2966,2967,2973,2979],{},[33,2968,2969,2972],{},[14,2970,2971],{},"Het Dreiging Model:"," Als de Service Provider kwaadwillend is, of als de browser van de gebruiker gecompromitteerd is (Man-in-the-Browser aanval), kan het scherm “Contract A” (onschuldig) tonen, terwijl het achtergrondproces de hash van “Contract B” (kwaadwillend, bijv. activa overdragen) naar itsme stuurt.",[33,2974,2975,2978],{},[14,2976,2977],{},"De Ervaring van de Gebruiker:"," De gebruiker ziet “Teken document voor Bank” op zijn telefoon. Hij vertrouwt het. Hij tekent.",[33,2980,2981,2984],{},[14,2982,2983],{},"Het Resultaat:"," De gebruiker heeft juridisch “Contract B” ondertekend terwijl hij geloofde dat hij “Contract A” tekende.",[18,2986,2987,2988,2991],{},"Dit is ",[14,2989,2990],{},"niet"," “What You See Is What You Sign”. Echte WYSIWYS vereist dat het ondertekenende apparaat (de telefoon) onafhankelijk de inhoud van het document rendert of de hash strikt verifieert tegen een vertrouwde weergave. Door deze context weg te strippen, reduceert itsme de handtekening tot een blinde daad van geloof in de integriteit van de Service Provider. Wij accepteren “geloof” niet als veiligheidscontrole.",[10,2993,2995],{"id":2994},"_6-institutionele-verstrengeling-het-frank-robben-effect",[14,2996,2997],{},"6. Institutionele Verstrengeling: Het “Frank Robben” Effect",[18,2999,3000],{},[283,3001],{"alt":285,"src":3002},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Ffrankrobben.webp",[18,3004,3005],{},"De snelle opkomst van itsme kan niet los worden gezien van het bijzondere institutionele landschap van Belgisch digitaal bestuur. Critici en privacyvoorvechters wijzen al lang op de consolidatie van macht binnen een kleine kring van technocraten die de grens tussen publiek mandaat en private implementatie bewandelen.",[18,3007,3008,3009,3012,3013,3016],{},"De figuur van ",[14,3010,3011],{},"Frank Robben",", CEO van de Kruispuntbank van de Sociale Zekerheid, oprichter van het eHealth-platform en beheerder bij Smals, symboliseert deze verstrengeling. Hoewel de staat formeel toezicht houdt, blijft de structurele realiteit: de Belgische staat heeft agressief een ",[630,3014,3015],{},"private"," oplossing (itsme) gepusht als het standaard, en vaak enige, haalbare mobiele alternatief voor de fysieke eID-kaart.",[18,3018,3019,3020,3023],{},"Dit creëert een ",[14,3021,3022],{},"Vendor Lock-In"," op nationaal niveau. Door itsme diep te integreren in de Federale Authenticatiedienst (FAS) en het te promoten voor essentiële diensten (Belastingen, Gezondheid, CovidSafeBE), heeft de staat in feite een soevereine kernfunctie geprivatiseerd: de verificatie van zijn burgers.",[30,3025,3026],{},[33,3027,3028,3031],{},[14,3029,3030],{},"Risico van Privatisering:"," Als Belgian Mobile ID NV besluit zijn prijsmodel voor bedrijven te wijzigen (wat het heeft gedaan), of als het zijn gebruiksvoorwaarden wijzigt om agressievere datadeling toe te staan, wordt het Belgische ecosysteem gegijzeld. Er is geen gemakkelijke “switch” naar een publiek alternatief omdat de staat te weinig heeft geïnvesteerd in mobiele eID ten gunste van het vertrouwen op het consortium.",[18,3033,3034,3035,3038],{},"De recente lancering van ",[14,3036,3037],{},"MyGov.be"," door de overheid wordt gezien als een late poging om deze soevereiniteit terug te winnen, waarbij staatssecretaris Mathieu Michel expliciet verwees naar de noodzaak van “controle” en “soevereiniteit” die bij itsme ontbrak.21",[10,3040,3042],{"id":3041},"_7-conclusie-een-beleid-van-niet-adoptie",[14,3043,3044],{},"7. Conclusie: Een Beleid van Niet-Adoptie",[18,3046,3047],{},[283,3048],{"alt":285,"src":3049},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fconclusie.webp",[18,3051,3052],{},"Samenvattend is de beslissing om itsme af te wijzen een proactieve maatregel om onze bedrijfsdataperimeter en de persoonlijke privacy van ons personeel te beschermen. Wij zijn geen technofoben; wij zijn realisten.",[18,3054,3055],{},[14,3056,3057],{},"Wij gebruiken itsme niet omdat:",[114,3059,3060,3066,3072,3078,3084],{},[33,3061,3062,3065],{},[14,3063,3064],{},"Wij de aandeelhouders niet vertrouwen:"," Het consortium van banken en telco’s heeft een inherent belangenconflict met betrekking tot data monetarisatie.",[33,3067,3068,3071],{},[14,3069,3070],{},"Wij het Eigendom niet vertrouwen:"," Liberty Global’s 100% eigendom van Telenet en zijn geschiedenis van hoge data-openbaarmaking percentages creëert een onaanvaardbare pijplijn naar buitenlandse surveillance.",[33,3073,3074,3077],{},[14,3075,3076],{},"Wij de Infrastructuur niet vertrouwen:"," De afhankelijkheid van AWS en de Amerikaanse CLOUD Act maakt “data soevereiniteit” tot een wassen neus.",[33,3079,3080,3083],{},[14,3081,3082],{},"Wij het “Ondergoedkleur”-Panopticum niet accepteren:"," De bewaartermijn van 10 jaar van granulaire transactie metadata creëert een surveillance database die wij weigeren te voeden.",[33,3085,3086,3089],{},[14,3087,3088],{},"Wij Bewijs eisen, geen Geloof:"," Het “Hash Signing”-protocol faalt in het bieden van cryptografische garanties van inhouds-integriteit aan de gebruiker.",[18,3091,3092,3093,2197,3096,3099,3100,3103],{},"Als bedrijf blijven wij gebruik maken van en ondersteuning bieden aan ",[14,3094,3095],{},"Client-Side Certificaat authenticatie",[14,3097,3098],{},"Fysieke Hardware Tokens",", en ",[14,3101,3102],{},"Gedecentraliseerde Identiteit (SSI)"," standaarden die de sleutels, en de data, recht in de handen van de gebruiker houden, niet bij een consortium.",[10,3105,3107],{"id":3106},"appendix-forensische-data-tabellen",[14,3108,3109],{},"Appendix: Forensische Data tabellen",[1230,3111,3113],{"id":3112},"tabel-3-infrastructuur-soevereiniteit-audit",[14,3114,3115],{},"Tabel 3: Infrastructuur Soevereiniteit Audit",[298,3117,3118,3142],{},[301,3119,3120],{},[304,3121,3122,3127,3132,3137],{},[307,3123,3124],{},[14,3125,3126],{},"Component",[307,3128,3129],{},[14,3130,3131],{},"Leverancier",[307,3133,3134],{},[14,3135,3136],{},"Locatie",[307,3138,3139],{},[14,3140,3141],{},"Jurisdictierisico",[316,3143,3144,3163,3182,3201],{},[304,3145,3146,3151,3154,3157],{},[321,3147,3148],{},[14,3149,3150],{},"Hosting",[321,3152,3153],{},"AWS (Amazon)",[321,3155,3156],{},"Local Zone Brussel \u002F Parent Regio Parijs",[321,3158,3159,3162],{},[14,3160,3161],{},"Kritiek:"," Onderworpen aan US CLOUD Act. Local Zone control plane resideert in Parijs\u002FFrankrijk.",[304,3164,3165,3170,3173,3176],{},[321,3166,3167],{},[14,3168,3169],{},"Notificaties",[321,3171,3172],{},"Apple \u002F Google",[321,3174,3175],{},"Wereldwijd",[321,3177,3178,3181],{},[14,3179,3180],{},"Hoog:"," Metadata zichtbaar voor Amerikaanse OS-leveranciers.",[304,3183,3184,3189,3192,3195],{},[321,3185,3186],{},[14,3187,3188],{},"SIM Controle",[321,3190,3191],{},"Proximus \u002F Orange \u002F Telenet",[321,3193,3194],{},"België",[321,3196,3197,3200],{},[14,3198,3199],{},"Medium:"," Telenet is 100% eigendom van US\u002FUK holding (Liberty Global).",[304,3202,3203,3208,3211,3213],{},[321,3204,3205],{},[14,3206,3207],{},"Root of Trust",[321,3209,3210],{},"Belgische Overheid (Rijksregister)",[321,3212,3194],{},[321,3214,3215,3218,3219,3222],{},[14,3216,3217],{},"Laag:"," Maar de ",[630,3220,3221],{},"toegang"," tot deze root wordt bewaakt door het private consortium.",[1230,3224,3226],{"id":3225},"tabel-4-het-10-jaren-metadata-risico-assessment",[14,3227,3228],{},"Tabel 4: Het 10-Jaren Metadata Risico Assessment",[298,3230,3231,3250],{},[301,3232,3233],{},[304,3234,3235,3240,3245],{},[307,3236,3237],{},[14,3238,3239],{},"Metadata Veld",[307,3241,3242],{},[14,3243,3244],{},"Potentiële Inferentie",[307,3246,3247],{},[14,3248,3249],{},"Commerciële Waarde",[316,3251,3252,3265,3278,3291],{},[304,3253,3254,3259,3262],{},[321,3255,3256],{},[14,3257,3258],{},"Service Provider ID",[321,3260,3261],{},"“Gebruiker logt in bij Echtscheidingsadvocaat”",[321,3263,3264],{},"Hoog (Marketing Juridische Diensten)",[304,3266,3267,3272,3275],{},[321,3268,3269],{},[14,3270,3271],{},"Frequentie",[321,3273,3274],{},"“Gebruiker logt dagelijks in op Goksite”",[321,3276,3277],{},"Hoog (Risicobepaling Verzekeringen\u002FLeningen)",[304,3279,3280,3285,3288],{},[321,3281,3282],{},[14,3283,3284],{},"Tijdstip",[321,3286,3287],{},"“Gebruiker actief tussen 02:00 – 04:00”",[321,3289,3290],{},"Medium (Gezondheids-\u002FLevensstijlprofilering)",[304,3292,3293,3298,3301],{},[321,3294,3295],{},[14,3296,3297],{},"Locatie (Land)",[321,3299,3300],{},"“Gebruiker is in Thailand”",[321,3302,3303],{},"Medium (Fraudedetectie, Verkoop Reisverzekering)",[18,3305,3306],{},[630,3307,3308],{},"Bovenstaande analyse bevestigt dat hoewel de inhoud van transacties versleuteld kan zijn, de metadata, die itsme toegeeft een decennium lang te bewaren, een voldoende hoge resolutie biedt van gebruikersgedrag om een ernstige privacyschending te vormen.",[1230,3310,3312],{"id":3311},"referenties",[14,3313,3314],{},"Referenties",[114,3316,3317,3324,3331,3338,3345,3352,3359,3366,3373,3380,3387,3394,3401,3408,3415,3422,3429,3436,3443,3450,3457],{},[33,3318,3319],{},[163,3320,3323],{"href":3321,"rel":3322},"https:\u002F\u002Fwww.brusselstimes.com\u002F1029912\u002Ffederal-government-renews-itsme-accreditation-for-three-years",[167],"Federal Government renews Itsme accreditation for three years – The Brussels Times",[33,3325,3326],{},[163,3327,3330],{"href":3328,"rel":3329},"https:\u002F\u002Fwww.ibr-ire.be\u002Fdocs\u002Fdefault-source\u002Fbas-documents\u002Freports\u002F2024\u002Fproximus-integrated-annual-report-2023-v2_en.pdf?sfvrsn=66638fdb_4",[167],"Proximus – Integrated annual report 2023",[33,3332,3333],{},[163,3334,3337],{"href":3335,"rel":3336},"https:\u002F\u002Fwww.itsme-id.com\u002Fen-BE\u002Fbusiness\u002Fblog\u002Fitsme-growth",[167],"itsme® raises €24.7 million to finance ambitious growth plans",[33,3339,3340],{},[163,3341,3344],{"href":3342,"rel":3343},"https:\u002F\u002Fwww.namirial.com\u002Fen\u002Fnews\u002Fnamirial-integrates-itsme-with-its-signature-platform\u002F",[167],"Namirial integrates itsme® with its Signature Platform to streamline compliant Digital Transactions in Belgium and beyond",[33,3346,3347],{},[163,3348,3351],{"href":3349,"rel":3350},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FLiberty_Global",[167],"Liberty Global – Wikipedia",[33,3353,3354],{},[163,3355,3358],{"href":3356,"rel":3357},"https:\u002F\u002Fwww.libertyglobal.com\u002Fwp-content\u002Fuploads\u002F2025\u002F03\u002FLG-2024-10-K-ANNUAL-REPORT.pdf",[167],"LG-2024-10-K-ANNUAL-REPORT.pdf – Liberty Global",[33,3360,3361],{},[163,3362,3365],{"href":3363,"rel":3364},"https:\u002F\u002Fwww.fsma.be\u002Fsites\u002Fdefault\u002Ffiles\u002Fprospectus\u002Ffiles\u002F2023\u002F2023-%2528TELENET%2529-EN-OPA20230102-A14-B01-C01-NP-CD06_06.pdf",[167],"VOLUNTARY AND CONDITIONAL TAKEOVER BID IN CASH possibly followed by a Simplified Squeeze-Out by LIBERTY GLOBAL BELGIUM HOLDING B – FSMA",[33,3367,3368],{},[163,3369,3372],{"href":3370,"rel":3371},"https:\u002F\u002Fwww.libertyglobal.com\u002Fabout\u002Fcorporate-governance\u002Fdata-privacy-protection\u002F",[167],"Data Privacy Protection – Liberty Global",[33,3374,3375],{},[163,3376,3379],{"href":3377,"rel":3378},"https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fsecurity\u002Fscaling-national-identity-schemes-with-itsme-and-amazon-cognito\u002F",[167],"Scaling national identity schemes with itsme and Amazon Cognito | AWS Security Blog",[33,3381,3382],{},[163,3383,3386],{"href":3384,"rel":3385},"https:\u002F\u002Fwww.billit.eu\u002Fen-int\u002Fsubprocessors\u002F",[167],"Billit subprocessors",[33,3388,3389],{},[163,3390,3393],{"href":3391,"rel":3392},"https:\u002F\u002Fwww.codit.eu\u002Fen\u002Fclients\u002Fbelgian-mobile-id-azure-cloud-application-modernization\u002F",[167],"Belgian Mobile ID: It’s all about scalability. – Codit",[33,3395,3396],{},[163,3397,3400],{"href":3398,"rel":3399},"https:\u002F\u002Fwww.reddit.com\u002Fr\u002Fbelgium\u002Fcomments\u002Fp3jrql\u002Fwhat_is_your_opinion_of_itsme\u002F",[167],"What is your opinion of Itsme? : r\u002Fbelgium – Reddit",[33,3402,3403],{},[163,3404,3407],{"href":3405,"rel":3406},"https:\u002F\u002Fwww.hunton.com\u002Fprivacy-and-information-security-law\u002Fbelgian-council-of-state-considers-encryption-a-sufficient-measure-for-u-s-data-transfers",[167],"Belgian Council of State Considers Encryption a Sufficient Measure for U.S. Data Transfers",[33,3409,3410],{},[163,3411,3414],{"href":3412,"rel":3413},"https:\u002F\u002Fwww.itsme-id.com\u002Fen-BE\u002Fbusiness\u002Fblog\u002Faccreditationnew",[167],"Government extends accreditation for itsme®",[33,3416,3417],{},[163,3418,3421],{"href":3419,"rel":3420},"https:\u002F\u002Fsupport.itsme-id.com\u002Fhc\u002Fen-us\u002Farticles\u002F360041508073-Which-ID-data-do-I-share",[167],"Which ID data do I share? – itsme Customer Support",[33,3423,3424],{},[163,3425,3428],{"href":3426,"rel":3427},"https:\u002F\u002Fwww.itsme-id.com\u002Fcontent-assets\u002F53388\u002F1714119203-itsme_appprivacypolicy_en_24-04-24.pdf",[167],"Privacy Policy: itsme App & Services",[33,3430,3431],{},[163,3432,3435],{"href":3433,"rel":3434},"https:\u002F\u002Fwww.itsme-id.com\u002Ffiles\u002F10030-description-of-data-protection-aspects-regarding-the-End-User_14.09.18.pdf",[167],"Aspects of privacy applicable to the itsme® Scheme Version 1.4",[33,3437,3438],{},[163,3439,3442],{"href":3440,"rel":3441},"https:\u002F\u002Fwww.itsme-id.com\u002Ffiles\u002F13000-Specific-Terms-and-Conditions-Service-Providers-Proposed-14.09.2018.pdf",[167],"Terms & Conditions for Service Providers Version 1.4 – Itsme",[33,3444,3445],{},[163,3446,3449],{"href":3447,"rel":3448},"https:\u002F\u002Fbelgianmobileid.github.io\u002Fslate\u002Fqes.html",[167],"1-introduction – itsme® Sign documentation – GitHub Pages",[33,3451,3452],{},[163,3453,3456],{"href":3454,"rel":3455},"https:\u002F\u002Fwww.itsme-id.com\u002Ffiles\u002Fitsme-TC-Sign-on-Adobe-Sign-01.01.2020.pdf",[167],"ITSME ® SIGNATURE CREATION SERVICE TERMS AND CONDITIONS",[33,3458,3459],{},[163,3460,3463],{"href":3461,"rel":3462},"https:\u002F\u002Fwww.brusselstimes.com\u002F1825509\u002Fbelgium-quietly-launches-mygov-be-a-state-run-alternative-to-itsme",[167],"Belgium launches MyGov.be, a state-run alternative to Itsme – The Brussels Times",{"title":285,"searchDepth":673,"depth":673,"links":3465},[3466,3467,3468,3469,3470,3471,3472,3473],{"id":2161,"depth":676,"text":2164},{"id":2211,"depth":676,"text":2214},{"id":2451,"depth":676,"text":2454},{"id":2611,"depth":676,"text":2614},{"id":2877,"depth":676,"text":2880},{"id":2994,"depth":676,"text":2997},{"id":3041,"depth":676,"text":3044},{"id":3106,"depth":676,"text":3109},[688,3475,3476,3477,2131,3478,3479,3480,2134,3481,3482,3037,2137],"AWS","Belgian Mobile ID","Cloud Act","Data Soevereiniteit","Digitale Identiteit","eIDAS","itsme","Liberty Global","2025-11-27T09:32:14","1. Inleiding: De Strategische Noodzaak van Weigering In het hedendaagse digitale landschap wordt de keuze voor een identiteit leverancier (IdP) vaak gereduceerd tot een discussie over User Experience (UX) en conversieratio’s. Deze reductionistische benadering verhult echter de diepgaande strategische, juridische en operationele risico’s die gepaard gaan met het uitbesteden van de “Sleutels tot het Koninkrijk”, ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Ffrontimage.webp",{},"https:\u002F\u002Feliovp.com\u002Fitsme-bij-ons-is-het-its-not-me-en-dit-is-waarom\u002F","\u002Fblog\u002Fitsme-bij-ons-is-het-its-not-me-en-dit-is-waarom",{"title":2151,"description":3484},"itsme-bij-ons-is-het-its-not-me-en-dit-is-waarom","blog\u002Fitsme-bij-ons-is-het-its-not-me-en-dit-is-waarom","0slEbVJm69JzeklWIbdjG_veefpYAV0E2BOgqWDeZBk",{"id":3494,"title":3495,"body":3496,"categories":3970,"date":3981,"description":3982,"extension":699,"image":3983,"meta":3984,"navigation":702,"originalUrl":3985,"path":3986,"seo":3987,"slug":3988,"stem":3989,"__hash__":3990},"blog\u002Fblog\u002Ffield-report-the-reality-of-building-agentic-ai-in-2025.md","Field Report. The Reality of Building Agentic AI in 2025",{"type":7,"value":3497,"toc":3961},[3498,3503,3510,3516,3519,3525,3531,3537,3540,3562,3567,3573,3576,3594,3600,3606,3609,3623,3628,3634,3637,3657,3660,3663,3671,3677,3680,3686,3689,3697,3703,3710,3730,3735,3741,3747,3769,3775,3781,3784,3795,3801,3808,3813,3819,3826,3834,3840,3846,3849,3867,3873,3879,3901,3906,3912,3915,3929,3935,3938,3952,3955],[18,3499,3500],{},[14,3501,3502],{},"From Hype to Sovereign Infrastructure",[18,3504,3505],{},[163,3506,3509],{"href":3507,"rel":3508},"https:\u002F\u002Feliovp.com\u002Fpraktijkverslag-de-realiteit-van-het-bouwen-van-agentic-ai-in-2025\u002F",[167],"Nederlandse Versie",[10,3511,3513],{"id":3512},"summary",[14,3514,3515],{},"Summary",[18,3517,3518],{},"The narrative surrounding “Agentic AI” in 2025 is defined by a sharp contrast between market expectations and engineering reality. While the general public, conditioned by the ease of ChatGPT, expects “miracles” and instant integration, the reality of building autonomous agents for enterprise workflows is a discipline of rigorous engineering, deep architectural planning, and infrastructure sovereignty.",[18,3520,3521,3522,3524],{},"This report details the challenges and methodologies we have encountered while deploying custom, on-premise AI agents. Unlike the majority of the market relying on volatile public APIs, our approach focuses on ",[14,3523,969],{},", running modular, tailored solutions entirely locally. This strategy mitigates the risks of API crashes, “silent updates” that break prompts, and unpredictable price surges, ensuring that our customers’ solutions remain robust, private, and cost-efficient over the long term.",[10,3526,3528],{"id":3527},"_1-the-expectation-gap-and-the-custom-necessity",[14,3529,3530],{},"1. The Expectation Gap and the “Custom” Necessity",[1230,3532,3534],{"id":3533},"_11-the-miracle-myth-vs-engineering-reality",[14,3535,3536],{},"1.1 The Miracle Myth vs. Engineering Reality",[18,3538,3539],{},"A pervasive challenge in 2025 is the “over-expectation” of AI capabilities. Clients often approach Agentic AI with the mindset of a chatbot user, expecting a plug-and-play solution that can “magically” navigate their entire ERP system out of the box.",[30,3541,3542,3552],{},[33,3543,3544,3547,3548,3551],{},[14,3545,3546],{},"The Reality."," An AI Agent is not a miracle; ",[14,3549,3550],{},"it is a complex software system"," that requires defined boundaries.",[33,3553,3554,3557,3558,3561],{},[14,3555,3556],{},"The Solution."," We build everything ",[14,3559,3560],{},"modularly",". By keeping the architecture component-based, we ensure that as a customer scales or identifies new use cases (e.g., expanding from support ticketing to inventory management), we can integrate new “cognitive modules” without rewriting the core system. This modularity is essential for long-term viability.",[18,3563,3564],{},[283,3565],{"alt":285,"src":3566},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fmiraclemyth.webp",[1230,3568,3570],{"id":3569},"_12-the-failure-of-global-solutions",[14,3571,3572],{},"1.2 The Failure of “Global” Solutions",[18,3574,3575],{},"We have realized that building a “global” product that attempts to work for every customer is a trap. Generic agents lack the nuances of specific business workflows, leading to brittle performance and high support overhead.",[30,3577,3578,3588],{},[33,3579,3580,3583,3584,3587],{},[14,3581,3582],{},"Tailored Engineering."," Success requires analyzing the customer’s ",[630,3585,3586],{},"entire workflow",", often involving on-site meetings and deep dives into their data structures. A solution tailored to the specific eccentricities of a client’s data will always outperform a generic “one-size-fits-all” model.",[33,3589,3590,3593],{},[14,3591,3592],{},"Spot-On Execution."," By restricting the agent’s scope to a tailored environment, we reduce the surface area for hallucinations and logic errors, ensuring the solution is “spot on” rather than “generally okay.”",[10,3595,3597],{"id":3596},"_2-the-economics-of-agentic-ai",[14,3598,3599],{},"2. The Economics of Agentic AI",[1230,3601,3603],{"id":3602},"_21-the-cost-of-discovery",[14,3604,3605],{},"2.1 The Cost of Discovery",[18,3607,3608],{},"Because existing AI Agents cannot simply be “implemented” like a standard software library, every project requires a distinct analysis phase.",[30,3610,3611,3617],{},[33,3612,3613,3616],{},[14,3614,3615],{},"Discovery Phase."," We invest significant time in pre-development analysis. This includes multiple meetings to map out the “real life” situation of the client’s operations.",[33,3618,3619,3622],{},[14,3620,3621],{},"Cost Reality."," As detailed in industry analyses, the cost of custom agent development involves hidden factors beyond just coding, including dataset preparation, architecture design, and extensive testing.",[18,3624,3625],{},[283,3626],{"alt":285,"src":3627},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Feconomics.webp",[1230,3629,3631],{"id":3630},"_22-the-sovereign-ai-financial-model",[14,3632,3633],{},"2.2 The “Sovereign AI” Financial Model",[18,3635,3636],{},"We do not utilize the APIs of big providers (OpenAI, Anthropic, etc.). Relying on external APIs introduces existential business risks:",[114,3638,3639,3645,3651],{},[33,3640,3641,3644],{},[14,3642,3643],{},"Volatility."," Will the API price suddenly triple?",[33,3646,3647,3650],{},[14,3648,3649],{},"Reliability."," Will the API crash during peak business hours?",[33,3652,3653,3656],{},[14,3654,3655],{},"Drift."," Will a “silent model update” break our prompts overnight?",[18,3658,3659],{},"The On-Prem Advantage:",[18,3661,3662],{},"By training and running everything locally (On-Prem), we convert variable, unpredictable costs into fixed, predictable assets.",[30,3664,3665],{},[33,3666,3667,3670],{},[14,3668,3669],{},"Future-Proofing."," A local model combined with owned hardware effectively does its job for years without hiccups. It does not get “dumber” because it is “old,” nor does it cost more to run in 2027 than it did in 2025. It is a stable asset.",[10,3672,3674],{"id":3673},"_3-observability-debugging-and-hallucinations",[14,3675,3676],{},"3. Observability, Debugging, and Hallucinations",[18,3678,3679],{},"You cannot fix what you cannot see. We pride ourselves on delivering “very nice working solutions,” which mandates granular observability.",[1230,3681,3683],{"id":3682},"_31-tracking-the-black-box",[14,3684,3685],{},"3.1 Tracking the “Black Box”",[18,3687,3688],{},"We utilize multiple tools to extensively track the LLM’s decision-making process. Debugging an agentic loop requires seeing the exact “Chain of Thought” and tool inputs that led to a specific action.",[30,3690,3691],{},[33,3692,3693,3696],{},[14,3694,3695],{},"Why we do it."," It allows us to pinpoint exactly where an agent got stuck in a loop or failed to retrieve the correct context, transforming debugging from guesswork into a precise science.",[1230,3698,3700],{"id":3699},"_32-managing-hallucinations",[14,3701,3702],{},"3.2 Managing Hallucinations",[18,3704,3705,3706,3709],{},"Hallucinations are an inevitability that must be managed. We log ",[630,3707,3708],{},"everything",". By maintaining a comprehensive audit trail of every input, thought, and output, we can:",[114,3711,3712,3718,3724],{},[33,3713,3714,3717],{},[14,3715,3716],{},"Track."," Identify patterns where the model consistently confabulates.",[33,3719,3720,3723],{},[14,3721,3722],{},"Solve."," Adjust the system prompt or fine-tune the model to eliminate these specific failure modes.",[33,3725,3726,3729],{},[14,3727,3728],{},"Prevent."," Implement “guardrail” classifiers that block hallucinatory outputs before they reach the user.",[18,3731,3732],{},[283,3733],{"alt":285,"src":3734},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fblackbox.webp",[1230,3736,3738],{"id":3737},"_33-metadata-strategy-for-agentic-search",[14,3739,3740],{},"3.3 Metadata Strategy for Agentic Search",[18,3742,3743,3744,3746],{},"For agentic search to work, “raw text” is insufficient. We ensure that we save extensive ",[14,3745,2650],{}," alongside the vector embeddings.",[30,3748,3749],{},[33,3750,3751,3754,3755,2197,3758,2197,3761,3764,3765,3768],{},[14,3752,3753],{},"Contextual Anchoring."," An agent analyzing a contract needs to know the ",[630,3756,3757],{},"date",[630,3759,3760],{},"author",[630,3762,3763],{},"version",", and ",[630,3766,3767],{},"department",". Without this metadata, the agent is searching blindly. We engineer our data ingestion pipelines to capture this context automatically, ensuring the agent retrieves not just “similar text,” but “the right document.”",[10,3770,3772],{"id":3771},"_4-the-cognition-layer-models-and-training",[14,3773,3774],{},"4. The Cognition Layer. Models and Training",[1230,3776,3778],{"id":3777},"_41-fine-tuning-is-not-optional",[14,3779,3780],{},"4.1 Fine-Tuning is Not Optional",[18,3782,3783],{},"We extensively test open-source models, but they rarely meet our specific expectations out of the box.",[30,3785,3786],{},[33,3787,3788,3790,3791,3794],{},[14,3789,3546],{}," To achieve production-grade reliability, we almost always need to ",[14,3792,3793],{},"fine-tune"," models on the customer’s specific data. This aligns the model’s “voice” and logic with the customer’s actual business rules.",[1230,3796,3798],{"id":3797},"_42-vision-training-from-scratch",[14,3799,3800],{},"4.2 Vision. Training from Scratch",[18,3802,3803,3804,3807],{},"For vision-related projects, fine-tuning is often insufficient. We frequently train models ",[14,3805,3806],{},"from scratch"," on the customer’s proprietary visual data (e.g., manufacturing defects, document layouts). This ensures the model learns the specific visual features relevant to the client, rather than relying on generic “internet-trained” features.",[18,3809,3810],{},[283,3811],{"alt":285,"src":3812},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fgenericmodels.webp",[1230,3814,3816],{"id":3815},"_43-reasoning-models-and-benchmarks",[14,3817,3818],{},"4.3 Reasoning Models and Benchmarks",[18,3820,3821,3822,3825],{},"We have found that ",[14,3823,3824],{},"reasoning\u002Fthinking models"," (like DeepSeek R1 or similar logic-heavy architectures) are absolutely the better choice for complex agentic workflows.",[30,3827,3828],{},[33,3829,3830,3833],{},[14,3831,3832],{},"Benchmarks aren’t everything."," A model might score high on a generic benchmark (like MMLU), but the slightest change in our prompting or workflow can generate a totally different effect. Extensive internal testing often contradicts public leaderboards.",[10,3835,3837],{"id":3836},"_5-hardware-realities-the-vram-bottleneck",[14,3838,3839],{},"5. Hardware Realities. The VRAM Bottleneck",[1230,3841,3843],{"id":3842},"_51-the-memorycontext-trade-off",[14,3844,3845],{},"5.1 The Memory\u002FContext Trade-off",[18,3847,3848],{},"Running “Reasoning Models” locally comes with significant caveats, primarily hardware requirements.",[30,3850,3851,3861],{},[33,3852,3853,3856,3857,3860],{},[14,3854,3855],{},"The “Buffer” Necessity."," Context is expensive. Analyzing a single mail thread of ",[14,3858,3859],{},"50 emails"," might consume gigabytes of memory for the KV cache alone.",[33,3862,3863,3866],{},[14,3864,3865],{},"The Failure Mode."," If this memory buffer is not available, the system hits an Out-Of-Memory (OOM) error, or we are forced to truncate context. When context drops, accuracy drops, and the “AI Agent fails.”",[1230,3868,3870],{"id":3869},"_52-quantization-bigger-is-better",[14,3871,3872],{},"5.2 Quantization. Bigger is Better",[18,3874,3875,3876,46],{},"We have empirically found that ",[14,3877,3878],{},"parameter count trumps precision",[30,3880,3881,3895],{},[33,3882,3883,3886,3887,3890,3891,3894],{},[14,3884,3885],{},"The Rule."," A large, quantized model, for example, a ",[14,3888,3889],{},"72B parameter model quantized to FP8"," (or even lower), often generates much better reasoning results than a smaller ",[14,3892,3893],{},"8B model running unquantized"," (BF16).",[33,3896,3897,3900],{},[14,3898,3899],{},"Why."," The larger model possesses a deeper “world model” and logic capability that survives quantization, whereas the smaller model, even at high precision, lacks the cognitive depth to handle complex agentic tasks.",[18,3902,3903],{},[283,3904],{"alt":285,"src":3905},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fmemory.webp",[1230,3907,3909],{"id":3908},"_53-scaling-and-cost",[14,3910,3911],{},"5.3 Scaling and Cost",[18,3913,3914],{},"While we can acquire expensive data center GPUs (B200s\u002FMI355X) with massive memory, these are often prohibitively expensive for mid-size (or even larger) companies.",[30,3916,3917,3923],{},[33,3918,3919,3922],{},[14,3920,3921],{},"The Challenge."," We must constantly balance the need for massive context (for reliability) with the budget constraints of on-premise hardware.",[33,3924,3925,3928],{},[14,3926,3927],{},"Inference Speed vs. Accuracy."," We extensively test quantized versions to see if we can speed up inference and save memory without losing accuracy. It doesn’t just “work out of the box”; it requires rigorous benchmarking per use case.",[10,3930,3932],{"id":3931},"conclusion-the-constant-evolution",[14,3933,3934],{},"Conclusion. The Constant Evolution",[18,3936,3937],{},"The market changes extremely fast. New quantization formats, new open-source models, and new agentic frameworks emerge on a daily basis.",[30,3939,3940,3946],{},[33,3941,3942,3945],{},[14,3943,3944],{},"Our Commitment."," We have to keep up. We continuously test and improve our workflows to ensure our clients aren’t left behind.",[33,3947,3948,3951],{},[14,3949,3950],{},"The Upside."," Despite the challenges, the hardware constraints, the data cleaning, the hallucination tracking, we love it. We are building systems that provide genuine, sovereign automation, free from the whims of the big AI providers.",[18,3953,3954],{},"Ready to build your Sovereign AI workforce? If you are looking for a partner to navigate these challenges and build a custom, on-premise solution that you truly own, we are here to help.",[18,3956,3957],{},[163,3958,3960],{"href":1690,"rel":3959},[167],"Eliovp’s Agentic AI Solutions",{"title":285,"searchDepth":673,"depth":673,"links":3962},[3963,3964,3965,3966,3967,3968,3969],{"id":3512,"depth":676,"text":3515},{"id":3527,"depth":676,"text":3530},{"id":3596,"depth":676,"text":3599},{"id":3673,"depth":676,"text":3676},{"id":3771,"depth":676,"text":3774},{"id":3836,"depth":676,"text":3839},{"id":3931,"depth":676,"text":3934},[688,689,3971,2125,3972,3973,3974,3975,3976,3977,3978,3979,969,3980],"Solutions","Agentic AI","AI Engineering","AI Strategy","Autonomous Agents","Enterprise AI","Local LLM","Model Fine-Tuning","On-Premise AI","VRAM Optimization","2025-11-25T14:03:39","From Hype to Sovereign Infrastructure Nederlandse Versie Summary The narrative surrounding “Agentic AI” in 2025 is defined by a sharp contrast between market expectations and engineering reality. While the general public, conditioned by the ease of ChatGPT, expects “miracles” and instant integration, the reality of building autonomous agents for enterprise workflows is a discipline of ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Ffieldreport.webp",{},"https:\u002F\u002Feliovp.com\u002Ffield-report-the-reality-of-building-agentic-ai-in-2025\u002F","\u002Fblog\u002Ffield-report-the-reality-of-building-agentic-ai-in-2025",{"title":3495,"description":3982},"field-report-the-reality-of-building-agentic-ai-in-2025","blog\u002Ffield-report-the-reality-of-building-agentic-ai-in-2025","myw0SNp19uTUclmvgwbxGZVWR-w3J9asW3BKfPjQsME",{"id":3992,"title":3993,"body":3994,"categories":4864,"date":4875,"description":4876,"extension":699,"image":4877,"meta":4878,"navigation":702,"originalUrl":4879,"path":4880,"seo":4881,"slug":4882,"stem":4883,"__hash__":4884},"blog\u002Fblog\u002Fthe-synthetic-unicorn-bubble.md","The Synthetic Unicorn Bubble",{"type":7,"value":3995,"toc":4849},[3996,4002,4006,4009,4012,4018,4023,4026,4032,4035,4041,4044,4072,4077,4178,4184,4189,4192,4198,4201,4235,4241,4244,4250,4255,4258,4278,4284,4289,4292,4298,4305,4311,4314,4317,4320,4326,4329,4335,4340,4343,4357,4363,4368,4375,4378,4384,4389,4392,4399,4402,4437,4440,4443,4446,4452,4457,4464,4470,4473,4478,4481,4485,4490,4493,4496,4499,4502,4505,4516,4522,4554,4560],[288,3997,3999],{"id":3998},"how-ai-neocloud-infrastructure-turns-circular-cash-flows-into-fake-growth",[14,4000,4001],{},"How AI Neocloud Infrastructure Turns Circular Cash Flows Into Fake Growth",[10,4003,4004],{"id":12},[14,4005,16],{},[18,4007,4008],{},"The interval between 2023 and 2025 has birthed a capital allocation phenomenon arguably without precedent: the “Synthetic Bubble.” Driven by the scramble for AI dominance, the venture capital apparatus has directed billions into the “Neocloud” ecosystem. However, a rigorous analysis suggests that up to 40% of this sector is built upon structural malfeasance.",[18,4010,4011],{},"This report posits that the sector is distinct from traditional bubbles in that underlying asset values are not merely inflated, but actively manufactured through circular financing, “vaporware” infrastructure, and predatory legal structures.",[10,4013,4015],{"id":4014},"section-i-the-neocloud-arbitrage-and-the-illusion-of-infrastructure",[14,4016,4017],{},"Section I: The Neocloud Arbitrage and the Illusion of Infrastructure",[18,4019,4020],{},[283,4021],{"alt":285,"src":4022},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsection1.webp",[18,4024,4025],{},"A large share of today’s “AI Neoclouds” sits on top of footprints originally built for other workloads, including cryptocurrency mining. When those economics broke after the 2022 crypto crash, some operators executed a rapid pivot, rebranding existing facilities as “AI Data Centers” without fully closing the gap between what the sites were designed for and what LLM training actually requires.",[1230,4027,4029],{"id":4028},"_11-the-pivot-from-crypto-mining-to-ai-grifting",[14,4030,4031],{},"1.1 The Pivot: From Crypto-Mining to AI-Grifting",[18,4033,4034],{},"The result is an operational mismatch: facilities with power and space, but often without the necessary cooling density, redundancy, and networking to reliably support modern AI training clusters. In the worst cases, these environments are marketed as if they were born-AI infrastructure, when in reality they are still in a transitional state, capable of running some GPU workloads, but not the scale, reliability, or SLA profile implied in the pitch.",[1230,4036,4038],{"id":4037},"_12-the-brokerage-trap-and-noisy-neighbors",[14,4039,4040],{},"1.2 The Brokerage Trap and “Noisy Neighbors”",[18,4042,4043],{},"A defining characteristic of the unethical Neocloud is the decoupling of asset ownership from service delivery.",[30,4045,4046,4060,4066],{},[33,4047,4048,4051,4052,4055,4056,4059],{},[14,4049,4050],{},"Layer 2 Reselling:"," Many startups lease capacity from third parties, slap a UI on it, and resell it at a markup. They market ",[630,4053,4054],{},"access"," to a supply chain queue as ",[630,4057,4058],{},"ownership"," of a deployed resource.",[33,4061,4062,4065],{},[14,4063,4064],{},"Oversubscription and Phantom Capacity",": To make the numbers work, some Neoclouds sell the same GPU to multiple customers and sell future capacity they don’t yet have. On paper, it shows up as healthy contracted ARR. In practice, when those customers all try to train concurrently, there aren’t enough GPUs to go around, queues explode, SLAs break, and performance craters.",[33,4067,4068,4071],{},[14,4069,4070],{},"The Hype-Only Deal Pipeline:"," On top of this, a significant portion of the “big wins” marketed to the press and investors never make it past the press release. Multi-million\u002Fbillion dollar “strategic deals” are announced off the back of tiny pilots, experimental credits, or non-binding MOUs. The logo slide and the headline travel; the actual deployed workloads do not. On paper, it looks like an infrastructure business at scale. On the ground, very little is physically happening beyond a small fraction of what’s being shouted about.",[18,4073,4074],{},[14,4075,4076],{},"Operational Divergence: Hyperscaler vs. Unethical Neocloud",[298,4078,4079,4098],{},[301,4080,4081],{},[304,4082,4083,4088,4093],{},[307,4084,4085],{},[14,4086,4087],{},"Operational Metric",[307,4089,4090],{},[14,4091,4092],{},"Legitimate Hyperscaler",[307,4094,4095],{},[14,4096,4097],{},"Unethical Neocloud Startup",[316,4099,4100,4113,4126,4139,4152,4165],{},[304,4101,4102,4107,4110],{},[321,4103,4104],{},[14,4105,4106],{},"Asset Ownership",[321,4108,4109],{},"Owns physical data centers, fiber, and custom silicon.",[321,4111,4112],{},"Leases capacity; acts as a layer-2 reseller or broker.",[304,4114,4115,4120,4123],{},[321,4116,4117],{},[14,4118,4119],{},"Resource Allocation",[321,4121,4122],{},"Strict isolation of tenant workloads via hypervisors.",[321,4124,4125],{},"Aggressive oversubscription (selling 1 GPU to 3 users).",[304,4127,4128,4133,4136],{},[321,4129,4130],{},[14,4131,4132],{},"SLA Transparency",[321,4134,4135],{},"Public, audited uptime and latency metrics.",[321,4137,4138],{},"Opaque SLAs; “best effort” performance disguised as guaranteed.",[304,4140,4141,4146,4149],{},[321,4142,4143],{},[14,4144,4145],{},"Virtualization",[321,4147,4148],{},"Mature hypervisors (Nitro, Titan) minimizing overhead.",[321,4150,4151],{},"Thin\u002FNon-existent virtualization causing security risks.",[304,4153,4154,4159,4162],{},[321,4155,4156],{},[14,4157,4158],{},"Compliance",[321,4160,4161],{},"SOC2, HIPAA, FedRAMP inherent to infrastructure.",[321,4163,4164],{},"Compliance is “roadmap” or self-attested; often non-existent.",[304,4166,4167,4172,4175],{},[321,4168,4169],{},[14,4170,4171],{},"Power Redundancy",[321,4173,4174],{},"N+1 or 2N redundancy for all critical path power.",[321,4176,4177],{},"Single-path power; susceptible to brownouts and failures.",[10,4179,4181],{"id":4180},"section-ii-financial-engineering-and-the-circular-revenue-economy",[14,4182,4183],{},"Section II: Financial Engineering and the Circular Revenue Economy",[18,4185,4186],{},[283,4187],{"alt":285,"src":4188},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsection2.webp",[18,4190,4191],{},"As interest rates squeezed traditional capital, the ecosystem turned inward, cannibalizing its own capital to feign growth.",[1230,4193,4195],{"id":4194},"_21-the-venture-capital-round-trip",[14,4196,4197],{},"2.1 The Venture Capital “Round Trip”",[18,4199,4200],{},"This mechanism is designed to artificially inflate Annual Recurring Revenue (ARR) using a single injection of capital.",[114,4202,4203,4212,4226],{},[33,4204,4205,4208,4209,46],{},[14,4206,4207],{},"The Investment:"," A VC firm invests $100M into ",[14,4210,4211],{},"Neocloud Startup A",[33,4213,4214,4217,4218,4221,4222,4225],{},[14,4215,4216],{},"The Stipulation:"," The VC invests $50M into ",[14,4219,4220],{},"Model Creator B",", mandating they use ",[14,4223,4224],{},"Startup A"," for compute.",[33,4227,4228,4231,4232,4234],{},[14,4229,4230],{},"The Loop",": Startup B transfers capital back to Startup A as “cloud revenue.”",[1671,4233],{},"Result: Both companies show massive growth, but the same dollar is simply spinning in a circle.",[1230,4236,4238],{"id":4237},"_22-wash-trading-and-the-sham-pilot",[14,4239,4240],{},"2.2 Wash Trading and the “Sham Pilot”",[18,4242,4243],{},"Startups are engaging in direct “wash trading”, swapping checks to meet Series B metrics. Company X buys software from Company Y, and Company Y buys compute from Company X. In more egregious cases, startups run “paid pilots” where the customer is secretly reimbursed via off-book “marketing fees,” allowing the startup to book a fraudulent contract as “Committed ARR.”",[10,4245,4247],{"id":4246},"section-iii-the-human-capital-façade",[14,4248,4249],{},"Section III: The Human Capital Façade",[18,4251,4252],{},[283,4253],{"alt":285,"src":4254},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsection3.webp",[18,4256,4257],{},"The deception extends to the human element. The pressure to scale has fostered a culture of “fake it till you make it” that borders on criminality.",[30,4259,4260,4266,4272],{},[33,4261,4262,4265],{},[14,4263,4264],{},"The “Mechanical Turk” AI:"," Startups claim to have proprietary “autonomous agents,” but in reality, complex tasks are routed to low-cost human labor pools. The user thinks they are interacting with an AI; they are actually waiting for a human in a server farm to type a response.",[33,4267,4268,4271],{},[14,4269,4270],{},"Ghost Engineers:"," To signal headcount growth to investors, some startups utilize bot farms to create fake LinkedIn profiles, populating their “Team” page with employees who do not exist.",[33,4273,4274,4277],{},[14,4275,4276],{},"Optics Hiring:"," Executives are hired solely for their pedigree (ex-DeepMind, ex-Meta) or even Phd’s with absolutely no real work experience whatsoever, just another name on a piece of paper with a phd and no accomplishments whatsoever to serve as figureheads for pitch decks, while unqualified founders make the actual technical decisions.",[10,4279,4281],{"id":4280},"section-iv-predatory-contracting-and-the-view-from-the-trenches",[14,4282,4283],{},"Section IV: Predatory Contracting and The View from the Trenches",[18,4285,4286],{},[283,4287],{"alt":285,"src":4288},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsection4.webp",[18,4290,4291],{},"The monetization strategy of the unethical Neocloud is often predatory, relying on entrapment rather than value.",[1230,4293,4295],{"id":4294},"_41-the-choke-contract-mechanism",[14,4296,4297],{},"4.1 The “Choke Contract” Mechanism",[18,4299,4300,4301,4304],{},"Demand for GPUs has allowed providers to enforce “Take-or-Pay” contracts. These agreements require customers to pay for reserved capacity regardless of performance, often demanding 50-100% upfront payment. The startup uses this cash to fund the hardware they claimed to ",[630,4302,4303],{},"already"," own, effectively using the customer’s money to fund their own CapEx.",[1230,4306,4308],{"id":4307},"_42-view-from-the-trenches-the-picks-and-shovels-reality",[14,4309,4310],{},"4.2 View from the Trenches: The “Picks and Shovels” Reality",[18,4312,4313],{},"Our firm operates in the physical reality of this sector, helping these Neoclouds set up the actual “picks and shovels” infrastructure. We see the operational truth that marketing decks hide.",[18,4315,4316],{},"We have observed a disturbing pattern: for these companies to survive the burn rates demanded by their valuation, they are forced to sign “Choke Contracts” with Service Level Agreements (SLAs) that are, frankly, insane. We have reviewed contracts containing uptime guarantees and latency promises that are physically impossible given the facility’s cooling and power constraints. Things that would make any engineer’s eyebrows go mad.",[18,4318,4319],{},"Why do they sign them? Desperation. The founders know the infrastructure can’t support the SLA, but they need to show investors “Signed Contracts” to unlock the next tranche of funding. It is a game of musical chairs where the music is kept playing by impossible promises.",[1230,4321,4323],{"id":4322},"_43-data-hostage-situations",[14,4324,4325],{},"4.3 Data Hostage Situations",[18,4327,4328],{},"Once a customer migrates data in, “zero egress fee” marketing evaporates. Startups use proprietary formats and bandwidth throttling to make leaving impossible, effectively claiming ownership over “derived data” (the trained model weights) to lock clients in legally.",[10,4330,4332],{"id":4331},"section-v-the-marketing-deployment-imbalance",[14,4333,4334],{},"Section V: The Marketing-Deployment Imbalance",[18,4336,4337],{},[283,4338],{"alt":285,"src":4339},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsection5.webp",[18,4341,4342],{},"A hallmark of the fraudulent startup is the inversion of capital allocation. Instead of funding R&D, funds are diverted to the “Hype Machine.”",[30,4344,4345,4351],{},[33,4346,4347,4350],{},[14,4348,4349],{},"Logo Slapping:"," Startups frequently list Fortune 500 companies as “Partners” based on a single developer using a free trial. This is designed to mislead investors into believing enterprise traction exists.",[33,4352,4353,4356],{},[14,4354,4355],{},"Fake Acquisitions:"," Startups engage in non-material “acquisitions” of failing companies (often owned by friends) to simulate momentum. They issue press releases announcing “Market Consolidation” to drive up the valuation before the lack of product delivery becomes apparent.",[10,4358,4360],{"id":4359},"section-vi-the-due-diligence-void",[14,4361,4362],{},"Section VI: The Due Diligence Void",[18,4364,4365],{},[283,4366],{"alt":285,"src":4367},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsection6-2.webp",[18,4369,4370,4371,4374],{},"This ecosystem thrives because the “Smart Money” has abdicated its responsibility. Driven by FOMO, VCs are skipping technical audits and site visits. They are investing in the ",[630,4372,4373],{},"slide deck",", not the business.",[18,4376,4377],{},"The reliance on “Total Addressable Market” (TAM) projections, “If AI is a $10 trillion market, we only need 0.1%”, has replaced unit economics analysis. This justifies funding companies with negative gross margins and no path to profitability.",[10,4379,4381],{"id":4380},"section-vii-a-field-guide-for-investors-how-not-to-fund-a-synthetic-unicorn",[14,4382,4383],{},"Section VII: A Field Guide for Investors: How Not to Fund a Synthetic Unicorn",[18,4385,4386],{},[283,4387],{"alt":285,"src":4388},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsection7-3.webp",[18,4390,4391],{},"To investors: if you are looking at one of these AI Neocloud stories, understand that you are not buying a narrative, you are underwriting physics, contracts, and incentives.",[18,4393,4394,4395,4398],{},"The current environment is engineered to short-circuit your process. Media noise, logo slides, breathless ARR headlines, and “AI infrastructure” positioning all exist to trigger FOMO and suppress actual due diligence. The only real defense is to ",[14,4396,4397],{},"slow the process down"," and bring in people who live in the wiring closet, not on the conference stage.",[18,4400,4401],{},"At a minimum, do the following before you wire a single dollar:",[30,4403,4404,4410,4416,4425,4431],{},[33,4405,4406,4409],{},[14,4407,4408],{},"Send a real team on-site","Not just the partner and an associate. Bring independent data center engineers, power and cooling experts, and network architects. Verify that there is something physical behind the deck: real racks, real power, real cooling, real fiber, real redundancy. If they won’t let you see the “production” site, you’ve already got your answer.",[33,4411,4412,4415],{},[14,4413,4414],{},"Cross-check the infrastructure against the promises","Take the marketing claims (uptime, latency, training cluster size, GPU type and count, redundancy, “AI-ready”) and map them to what you see in the facility. Does the power envelope, cooling design, and network fabric actually support the SLA and the cluster diagrams? If the physics don’t close, the model doesn’t either.",[33,4417,4418,4421,4422],{},[14,4419,4420],{},"Have experts tear through the contracts and SLAs","Don’t just skim the term sheet. Get specialist counsel to read the master service agreement, SLAs, take-or-pay clauses, egress terms, derived-data language, and liability caps. Ask a simple question: ",[630,4423,4424],{},"If this thing behaves the way the facility looks, how many of these contracts are already in technical breach on day one?",[33,4426,4427,4430],{},[14,4428,4429],{},"Interrogate the revenue quality, not just the revenue line","Disaggregate “ARR” into: related-party deals, VC round-tripping, paid pilots, rebates, and vendor-financed contracts. Look for circular flows (“our investor is also our biggest customer”), sham pilots, side letters, and “marketing fees” that mysteriously offset customer spend. Growth that collapses when you remove these is not growth; it’s choreography.",[33,4432,4433,4436],{},[14,4434,4435],{},"Diligence the humans, not just the LinkedIn pages","Verify that the key engineers, SREs, and infra leads actually exist, actually work there, and actually have the experience implied. Push beyond the optics hires and logo CVs. If the pitch leans heavily on ex-brand names and PhDs with no delivery track record, you are likely looking at a signaling strategy, not an execution engine.",[18,4438,4439],{},"In a healthy deal, these steps are mildly uncomfortable and mostly confirmatory. In a synthetic one, they are explosive. You will find mismatches between facility and SLA, between contracts and reality, between “ARR” and actual cash economics.",[18,4441,4442],{},"Mark our words: we have run this playbook multiple times. Every time we have been allowed to look under the hood, something material was off.",[18,4444,4445],{},"If you are not willing to fund this level of diligence, on-site, technical, contractual, and human, then understand what game you are really playing. You are not investing in infrastructure. You are volunteering to be the last, most expensive ticket holder to a show that is already over.",[10,4447,4449],{"id":4448},"section-viii-the-inevitable-burst",[14,4450,4451],{},"Section VIII: The Inevitable Burst",[18,4453,4454],{},[283,4455],{"alt":285,"src":4456},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsection8-3.webp",[18,4458,4459,4460,4463],{},"We are witnessing a “Synthetic Bubble.” It will not burst because AI is fake, AI is real. It will burst because the ",[630,4461,4462],{},"financial delivery vehicles"," built to capitalize on it, the unethical Neoclouds, are structurally unsound.",[1230,4465,4467],{"id":4466},"_71-the-warning-to-late-stage-investors",[14,4468,4469],{},"7.1 The Warning to Late-Stage Investors",[18,4471,4472],{},"When the “circular” revenue stops spinning and the “take-or-pay” defaults begin, the liquidity crisis will be swift. The current wave of marketing and shouting is designed to attract the next layer of capital, the late-stage investors and PE firms, who think they are buying into a high-growth infrastructure play.",[18,4474,4475],{},[14,4476,4477],{},"They are not.",[18,4479,4480],{},"They are buying into a liability time bomb of breached SLAs, oversubscribed hardware, and circular revenue that will evaporate the moment the funding stops. The market will eventually clear, but it will wipe out the “paper unicorns” that prioritized press releases over engineering.",[10,4482,4484],{"id":4483},"section-ix-why-were-writing-this","Section IX: Why We’re Writing This",[18,4486,4487],{},[283,4488],{"alt":285,"src":4489},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsection9.webp",[18,4491,4492],{},"We’re writing this because, from the inside, the signal-to-noise ratio in AI infrastructure has collapsed.",[18,4494,4495],{},"Every week, we’re bombarded with announcements of “record-breaking” rounds, “strategic” partnerships, and “nation-scale” AI data centers. A non-trivial share of those headlines are either wildly exaggerated or structurally unsound: circular deals, paper capacity, MOUs marketed as deployments. The hype is not just annoying; it’s misallocating billions.",[18,4497,4498],{},"But that’s only half the story.",[18,4500,4501],{},"We’re also writing this because we still believe this market can be built on real engineering, real contracts, and real outcomes. There are companies, including ours, that are doing the unglamorous work: designing and deploying actual infrastructure, pushing metal and electrons, reading the SLAs before signing them, and refusing to fund operations with choke contracts we know can’t be honored.",[18,4503,4504],{},"We don’t get the same spotlight as the loudest “unicorns”. We don’t publish breathless “AI revolution” press releases every quarter. We don’t backfill reality with marketing, and we don’t sign impossible SLAs just to unlock the next funding round. We take the slower, harder, ethical route: build real capacity, tell the truth about what it can do, and then scale it.",[18,4506,4507,4508,4511,4512,4515],{},"This article is our attempt to tilt the game, even slightly, back toward those incentives. If investors reward ",[630,4509,4510],{},"doing"," over ",[630,4513,4514],{},"shouting",", the market will correct faster, less capital will be incinerated, and the operators who actually deliver will finally get the room to show their real strength.",[10,4517,4519],{"id":4518},"referenced-data-citations",[14,4520,4521],{},"Referenced Data & Citations",[30,4523,4524,4529,4534,4539,4544,4549],{},[33,4525,4526],{},[14,4527,4528],{},"Fraudulent Marketing & Scams:",[33,4530,4531],{},[14,4532,4533],{},"Circular Revenue & Financial Engineering:",[33,4535,4536],{},[14,4537,4538],{},"Neocloud\u002FGPU Brokerage Issues:",[33,4540,4541],{},[14,4542,4543],{},"Toxic Culture & Hiring:",[33,4545,4546],{},[14,4547,4548],{},"Contracting & Vaporware:",[33,4550,4551],{},[14,4552,4553],{},"Market Correction Indicators:",[1230,4555,4557],{"id":4556},"references",[14,4558,4559],{},"References",[114,4561,4562,4569,4576,4583,4590,4597,4604,4611,4618,4625,4632,4639,4646,4653,4660,4667,4674,4681,4688,4695,4702,4709,4716,4723,4730,4737,4744,4751,4758,4765,4772,4779,4786,4793,4800,4807,4814,4821,4828,4835,4842],{},[33,4563,4564],{},[163,4565,4568],{"href":4566,"rel":4567},"https:\u002F\u002Fwww.vestbee.com\u002Finsights\u002Farticles\u002Fai-investments-in-2025-strategies-v-cs-are-using-to-navigate-industry-shifts",[167],"AI investments in 2025: strategies VCs are using to navigate industry shifts | Vestbee",[33,4570,4571],{},[163,4572,4575],{"href":4573,"rel":4574},"https:\u002F\u002Fwww.reddit.com\u002Fr\u002FSTEW_ScTecEngWorld\u002Fcomments\u002F1p3vffh\u002Fthe_algorithm_that_detected_a_610_billion_fraud\u002F",[167],"The Algorithm That Detected a $610 Billion Fraud: How Machine Intelligence Exposed the AI Industry’s Circular Financing Scheme : r\u002FSTEW_ScTecEngWorld – Reddit",[33,4577,4578],{},[163,4579,4582],{"href":4580,"rel":4581},"https:\u002F\u002Fwww.datacenterfrontier.com\u002Fsponsored\u002Farticle\u002F55271027\u002Fstream-data-centers-the-rise-of-the-fake-data-center-developer-and-how-to-tell-the-difference",[167],"The Rise of the “Fake” Data Center Developer ,  And How to Tell the Difference",[33,4584,4585],{},[163,4586,4589],{"href":4587,"rel":4588},"https:\u002F\u002Fwww.reddit.com\u002Fr\u002FMachineLearning\u002Fcomments\u002F1gps8fl\u002Fd_together_ai_hits_100m_in_arr_but_it_just\u002F",[167],"[D] Together AI hits $100M in ARR but it just resales compute – hype? : r\u002FMachineLearning",[33,4591,4592],{},[163,4593,4596],{"href":4594,"rel":4595},"https:\u002F\u002Fmassedcompute.com\u002Ffaq-answers\u002F?question=How%20does%20GPU%20oversubscription%20impact%20the%20cost%20of%20running%20high-performance%20computing%20workloads%20in%20the%20cloud?",[167],"How does GPU oversubscription impact the cost of running high-performance computing workloads in the cloud?",[33,4598,4599],{},[163,4600,4603],{"href":4601,"rel":4602},"https:\u002F\u002Fwww.stackscale.com\u002Fblog\u002Foversubscription-cloud-computing\u002F",[167],"What’s the oversubscription of resources in cloud computing? – Stackscale",[33,4605,4606],{},[163,4607,4610],{"href":4608,"rel":4609},"https:\u002F\u002Fmarginalrevolution.com\u002Fmarginalrevolution\u002F2025\u002F10\u002Fshould-we-worry-about-ais-circular-deals.html",[167],"Should we worry about AI’s circular deals? – Marginal REVOLUTION",[33,4612,4613],{},[163,4614,4617],{"href":4615,"rel":4616},"https:\u002F\u002Fwww.verusinvestments.com\u002Fconcerns-raised-over-circular-funding-between-a-i-startups-service-providers\u002F",[167],"Concerns raised over circular funding between A.I. startups & service providers – Verus",[33,4619,4620],{},[163,4621,4624],{"href":4622,"rel":4623},"https:\u002F\u002Frealinvestmentadvice.com\u002Fresources\u002Fblog\u002Fnvidia-deals-round-tripping-or-vendor-financing\u002F",[167],"Nvidia Deals: Round Tripping Or Vendor Financing? – RIA – Real Investment Advice",[33,4626,4627],{},[163,4628,4631],{"href":4629,"rel":4630},"https:\u002F\u002Ffundraiseinsider.com\u002Fblog\u002Fseries-b-startups\u002F",[167],"List of Funded Series B Startups (2025) – Fundraise Insider",[33,4633,4634],{},[163,4635,4638],{"href":4636,"rel":4637},"https:\u002F\u002Fwww.nsf.gov\u002Ftip\u002Fupdates\u002Fnsf-announces-pilot-startups-small-businesses-fast",[167],"NSF announces pilot for startups and small businesses to ‘fast-track’ successful development of technological innovations",[33,4640,4641],{},[163,4642,4645],{"href":4643,"rel":4644},"https:\u002F\u002Fmedium.com\u002F@divyanshbhatiajm19\u002Fbuilder-ais-1-5b-collapse-how-700-engineers-pretended-to-be-ai-and-got-away-with-it-for-years-2a0d35fc4af8",[167],"Builder.ai’s $450M Collapse: The Real Story Behind How a Unicorn Cooked Its Books (And Got Away… – Medium",[33,4647,4648],{},[163,4649,4652],{"href":4650,"rel":4651},"https:\u002F\u002Fm.economictimes.com\u002Fnews\u002Finternational\u002Fglobal-trends\u002Fnates-ai-was-just-people-in-a-room-startup-founder-charged-with-faking-ai-raising-50-million-on-false-claims\u002Farticleshow\u002F120210967.cms",[167],"Nate’s ‘AI’ was just people in a room: Startup founder charged with faking AI, raising $50 million on false claims – The Economic Times",[33,4654,4655],{},[163,4656,4659],{"href":4657,"rel":4658},"https:\u002F\u002Fcyberscoop.com\u002Fdoj-indicts-five-in-north-korean-fake-it-worker-scheme\u002F",[167],"DOJ indicts five in North Korean fake IT worker scheme – CyberScoop",[33,4661,4662],{},[163,4663,4666],{"href":4664,"rel":4665},"https:\u002F\u002Fwww.strongestlayer.com\u002Fblog\u002Fai-generated-fake-companies-job-fraud",[167],"EXPOSED: How AI-Generated Phantom Companies Are Infiltrating Corporate Hiring",[33,4668,4669],{},[163,4670,4673],{"href":4671,"rel":4672},"https:\u002F\u002Ftatech.org\u002Fthe-rise-of-the-genai-unqualified-candidates-wave-and-how-it-disrupts-recruitment\u002F",[167],"The Rise of the “GenAI Unqualified Candidates Wave” and How it Disrupts Recruitment,",[33,4675,4676],{},[163,4677,4680],{"href":4678,"rel":4679},"https:\u002F\u002Fwww.squirepattonboggs.com\u002Fen\u002Finsights\u002Fpublications\u002F2025\u002F07\u002Fmaking-gpus-work-for-you-time-honoured-methods-of-maximising-value-from-graphics-processing-unit-gpu-rental-agreements",[167],"Making GPUs Work For You – Time-Honoured Methods of Maximising Value From Graphics Processing Unit (GPU) Rental Agreements – Squire Patton Boggs",[33,4682,4683],{},[163,4684,4687],{"href":4685,"rel":4686},"https:\u002F\u002Fwww.bennettjones.com\u002FInsights\u002FBlogs\u002FThe-Hidden-Hazards-of-Vendor-AI-Terms-and-Conditions",[167],"“We signed what?!”: The Hidden Hazards of Vendor AI Terms and Conditions",[33,4689,4690],{},[163,4691,4694],{"href":4692,"rel":4693},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=jzmDfyhLkVs",[167],"How to Avoid Cloud and ERP Software Vendor Lock-In | The Risks of “One Throat to Choke”",[33,4696,4697],{},[163,4698,4701],{"href":4699,"rel":4700},"https:\u002F\u002Fwww.ftc.gov\u002Fpolicy\u002Fadvocacy-research\u002Ftech-at-ftc\u002F2024\u002F02\u002Fai-other-companies-quietly-changing-your-terms-service-could-be-unfair-or-deceptive",[167],"AI (and other) Companies: Quietly Changing Your Terms of Service Could Be Unfair or Deceptive | Federal Trade Commission",[33,4703,4704],{},[163,4705,4708],{"href":4706,"rel":4707},"https:\u002F\u002Fwww.reddit.com\u002Fr\u002Fstartups\u002Fcomments\u002F1jdz7ji\u002Fstartups_faking_traction_i_will_not_promote\u002F",[167],"Startups faking traction (I will not promote) – Reddit",[33,4710,4711],{},[163,4712,4715],{"href":4713,"rel":4714},"https:\u002F\u002Fwww.reddit.com\u002Fr\u002FSaaS\u002Fcomments\u002F1eeb277\u002Fwhy_are_people_putting_logos_of_other_companies\u002F",[167],"Why are people putting logos of other companies that have nothing to do with their products on their landing pages? : r\u002FSaaS – Reddit",[33,4717,4718],{},[163,4719,4722],{"href":4720,"rel":4721},"https:\u002F\u002Fwww.cfodive.com\u002Fnews\u002Fai-startup-reaped-millions-using-bogus-claims-ftc-suit\u002F758587\u002F",[167],"AI startup reaped millions using bogus claims, FTC suit says | CFO Dive",[33,4724,4725],{},[163,4726,4729],{"href":4727,"rel":4728},"https:\u002F\u002Fbeamstart.com\u002Fnews\u002Fvcs-abandon-old-rules-for-17630759659238",[167],"Venture Capitalists Rewrite Rules in Wild AI Startup Investment Boom of 2025",[33,4731,4732],{},[163,4733,4736],{"href":4734,"rel":4735},"https:\u002F\u002Fwww.newideaslegaltech.com\u002Fdebunking-dubious-justifications-for-the-ai-bubble\u002F",[167],"Debunking Dubious Justifications for the AI Bubble | New Ideas Legal Tech",[33,4738,4739],{},[163,4740,4743],{"href":4741,"rel":4742},"https:\u002F\u002Fmadhyamamonline.com\u002Fopinion\u002Fdeep-read\u002Fthe-ai-bubble-when-history-rhymes-with-the-doc-com-era-1468224",[167],"The AI bubble: when history rhymes with the doc-com era – Madhyamam",[33,4745,4746],{},[163,4747,4750],{"href":4748,"rel":4749},"https:\u002F\u002Fthenextweb.com\u002Fnews\u002Fai-washing-investors-real-startup-innovation",[167],"VCs are growing wary of ‘AI-washing’ ,  and backing real innovation – TNW",[33,4752,4753],{},[163,4754,4757],{"href":4755,"rel":4756},"https:\u002F\u002Fm.economictimes.com\u002Fmagazines\u002Fpanache\u002Fmit-study-shatters-ai-hype-95-of-generative-ai-projects-are-failing-sparking-tech-bubble-jitters\u002Farticleshow\u002F123428252.cms",[167],"MIT study shatters AI hype: 95% of generative AI projects are failing, sparking tech bubble jitters – The Economic Times",[33,4759,4760],{},[163,4761,4764],{"href":4762,"rel":4763},"https:\u002F\u002Fwww.theguardian.com\u002Fbusiness\u002Flive\u002F2025\u002Fnov\u002F18\u002Fstock-market-sell-off-ai-bubble-google-nvidia-ftse-100-bitcoin-business-live-news",[167],"FTSE 100 in biggest drop since April as stock market sell-off continues – as it happened",[33,4766,4767],{},[163,4768,4771],{"href":4769,"rel":4770},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=mnAP23UI_lk&vl=en-US",[167],"Finished AI Data Centers Have No Electricity — AI Fraud Can’t Fake Reality – YouTube,",[33,4773,4774],{},[163,4775,4778],{"href":4776,"rel":4777},"https:\u002F\u002Fwww.justice.gov\u002Fusao-sdny\u002Fpr\u002Ftech-ceo-charged-artificial-intelligence-investment-fraud-scheme",[167],"Southern District of New York | Tech CEO Charged In Artificial Intelligence Investment Fraud Scheme | United States Department of Justice",[33,4780,4781],{},[163,4782,4785],{"href":4783,"rel":4784},"https:\u002F\u002Fwww.aarp.org\u002Fmoney\u002Fscams-fraud\u002Fai-scams\u002F",[167],"AI Fuels New, Frighteningly Effective Scams – AARP",[33,4787,4788],{},[163,4789,4792],{"href":4790,"rel":4791},"https:\u002F\u002Fwww.reddit.com\u002Fr\u002FEconomics\u002Fcomments\u002F1o5qjwm\u002Fvery_troubling_ais_selfinvestment_spree_sets_off\u002F",[167],"‘Very troubling’: AI’s self-investment spree sets off bubble alarms on Wall Street – Reddit,",[33,4794,4795],{},[163,4796,4799],{"href":4797,"rel":4798},"https:\u002F\u002Fwww.sofi.com\u002Flearn\u002Fcontent\u002Fwash-trading\u002F",[167],"Wash Trading: What Is It and How Does It Work? (Examples) – SoFi",[33,4801,4802],{},[163,4803,4806],{"href":4804,"rel":4805},"https:\u002F\u002Fwww.thundercompute.com\u002Fblog\u002Fneoclouds-the-new-gpu-clouds-changing-ai-infrastructure",[167],"What is a Neocloud? The Rise of GPU-only Clouds – Thunder Compute",[33,4808,4809],{},[163,4810,4813],{"href":4811,"rel":4812},"https:\u002F\u002Fwww.lawyer-monthly.com\u002F2025\u002F11\u002Fwhos-responsible-for-ai-the-rise-of-neocloud-middlemen\u002F",[167],"Who’s Responsible for AI? The Rise of Neocloud Middlemen – Lawyer Monthly",[33,4815,4816],{},[163,4817,4820],{"href":4818,"rel":4819},"https:\u002F\u002Flanding.underdog.io\u002Fblog\u002Fcrazy-hours-at-ai-startups",[167],"The AI Startup Burnout Myth: Why Crazy Hours Don’t Equal Success – Underdog.io",[33,4822,4823],{},[163,4824,4827],{"href":4825,"rel":4826},"https:\u002F\u002Fbuiltin.com\u002Fhardware\u002Fvaporware",[167],"17 Unforgettable Examples of Vaporware – Built In",[33,4829,4830],{},[163,4831,4834],{"href":4832,"rel":4833},"https:\u002F\u002Fwww.joneswalker.com\u002Fen\u002Finsights\u002Fblogs\u002Fai-law-blog\u002Fai-vendor-liability-squeeze-courts-expand-accountability-while-contracts-shift-r.html?id=102l4ta",[167],"AI Vendor Liability Squeeze: Courts Expand Accountability While Contracts Shift Risk",[33,4836,4837],{},[163,4838,4841],{"href":4839,"rel":4840},"https:\u002F\u002Fwww.reddit.com\u002Fr\u002FOpenAI\u002Fcomments\u002F1p4a8ry\u002Fpoc\u002F",[167],"POC : r\u002FOpenAI – Reddit",[33,4843,4844],{},[163,4845,4848],{"href":4846,"rel":4847},"https:\u002F\u002Fm.economictimes.com\u002Fmarkets\u002Fstocks\u002Fnews\u002Fworried-about-an-ai-stock-bubble-that-might-be-good-news-for-nifty-bulls\u002Farticleshow\u002F125426521.cms",[167],"Worried about an AI stock bubble? That might be good news for Nifty bulls",{"title":285,"searchDepth":673,"depth":673,"links":4850},[4851],{"id":3998,"depth":673,"text":4001,"children":4852},[4853,4854,4855,4856,4857,4858,4859,4860,4861,4862,4863],{"id":12,"depth":676,"text":16},{"id":4014,"depth":676,"text":4017},{"id":4180,"depth":676,"text":4183},{"id":4246,"depth":676,"text":4249},{"id":4280,"depth":676,"text":4283},{"id":4331,"depth":676,"text":4334},{"id":4359,"depth":676,"text":4362},{"id":4380,"depth":676,"text":4383},{"id":4448,"depth":676,"text":4451},{"id":4483,"depth":676,"text":4484},{"id":4518,"depth":676,"text":4521},[688,689,2125,4865,4866,4867,4868,4869,4870,4871,4872,4873,4874],"AI Infrastructure","AI Neocloud","Circular Financing","GPU Cloud","Investment Risks","Startup Valuation","Synthetic Bubble","Tech Analysis","Vaporware","Venture Capital","2025-11-24T19:22:28","How AI Neocloud Infrastructure Turns Circular Cash Flows Into Fake Growth Executive Summary The interval between 2023 and 2025 has birthed a capital allocation phenomenon arguably without precedent: the “Synthetic Bubble.” Driven by the scramble for AI dominance, the venture capital apparatus has directed billions into the “Neocloud” ecosystem. However, a rigorous analysis suggests that ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsyntheticunicorn.webp",{},"https:\u002F\u002Feliovp.com\u002Fthe-synthetic-unicorn-bubble\u002F","\u002Fblog\u002Fthe-synthetic-unicorn-bubble",{"title":3993,"description":4876},"the-synthetic-unicorn-bubble","blog\u002Fthe-synthetic-unicorn-bubble","KZd7z9IaCQ0tpUeA2NxrzgTV2fdwcjkqiXiuSn4IDAA",{"id":4886,"title":4887,"body":4888,"categories":5185,"date":5195,"description":5196,"extension":699,"image":5197,"meta":5198,"navigation":702,"originalUrl":5199,"path":5200,"seo":5201,"slug":5202,"stem":5203,"__hash__":5204},"blog\u002Fblog\u002Fbuilding-the-engine-for-the-ai-race-the-4-month-path-to-nvidia-gb300-nvl72-power.md","Building the Engine for the AI Race: The 4-Month Path to NVIDIA GB300 NVL72 Power",{"type":7,"value":4889,"toc":5176},[4890,4893,4896,4902,4905,4908,4914,4917,4922,4947,4952,4958,4961,4964,4969,4989,4992,4997,5003,5006,5011,5016,5047,5052,5066,5072,5075,5095,5101,5104,5109,5159,5164,5170,5173],[18,4891,4892],{},"In artificial intelligence infrastructure, speed is the foundation of competitive differentiation. From model training velocity to inference latency, every millisecond matters. But before any workload executes, there is a critical prerequisite that often determines success or failure: time to market.",[18,4894,4895],{},"Traditional builds typically require 18–24 months. In the current AI cycle, that is simply too slow. We have refined a modular pipeline to achieve operational status within just 4 months, delivering a turnkey facility specifically architected for the NVIDIA GB300 NVL72.",[10,4897,4899],{"id":4898},"the-challenge-hosting-a-beast",[14,4900,4901],{},"The Challenge: Hosting a Beast",[18,4903,4904],{},"The NVIDIA GB300 NVL72 integrates 72 NVIDIA Blackwell Ultra GPUs and 36 Arm-based NVIDIA Grace CPUs in a fully liquid-cooled, rack-scale architecture.",[18,4906,4907],{},"This represents a fundamental shift in data center infrastructure requirements. We aren’t just powering servers; we are powering a massive, dense cluster where each rack draws approximately 120 kW under typical loads, potentially reaching 150 kW peak power during intensive computational workloads.",[10,4909,4911],{"id":4910},"architecture-overview-precision-at-scale",[14,4912,4913],{},"Architecture Overview: Precision at Scale",[18,4915,4916],{},"To accommodate this density, our facility uses a layout optimized for the specific footprint of the GB300.",[18,4918,4919],{},[14,4920,4921],{},"Our facility accommodates:",[30,4923,4924,4930],{},[33,4925,4926,4929],{},[14,4927,4928],{},"16x 600mm GB300 NVL72 racks"," (primary compute infrastructure)",[33,4931,4932,4935,4936],{},[14,4933,4934],{},"12x 800mm auxiliary racks"," distributed as:\n",[30,4937,4938,4941,4944],{},[33,4939,4940],{},"5 racks for network infrastructure",[33,4942,4943],{},"5 racks for management and storage systems",[33,4945,4946],{},"2 racks for future expansion",[18,4948,4949],{},[283,4950],{"alt":285,"src":4951},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsingle_unit_invert-1.webp",[10,4953,4955],{"id":4954},"power-distribution-the-4-makes-3-architecture",[14,4956,4957],{},"Power Distribution: The “4-Makes-3” Architecture",[18,4959,4960],{},"The extreme power density of GB300 NVL72 systems, 150 kW peak per rack, necessitates an enterprise-grade power distribution strategy.",[18,4962,4963],{},"We’ve implemented the NVIDIA reference design 4-makes-3 busbar configuration, where four independent 1 MW busbars provide power through an $N=3+1$ redundancy model.",[18,4965,4966],{},[14,4967,4968],{},"This architecture delivers:",[30,4970,4971,4977,4983],{},[33,4972,4973,4976],{},[14,4974,4975],{},"Full concurrency maintainability:"," Any single busbar can be taken offline for maintenance without service interruption.",[33,4978,4979,4982],{},[14,4980,4981],{},"True redundancy:"," Three busbars can support full operational load.",[33,4984,4985,4988],{},[14,4986,4987],{},"Total power envelope:"," Approximately 2.8 MW (variable based on storage configuration requirements).",[18,4990,4991],{},"Additionally, the GB300 NVL72 incorporates advanced power supply units with energy storage that reduce peak grid demand by up to 30% by smoothing power spikes from AI workloads.",[18,4993,4994],{},[283,4995],{"alt":285,"src":4996},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002F4-makes-3_trans-1.webp",[10,4998,5000],{"id":4999},"advanced-thermal-management-hybrid-cooling",[14,5001,5002],{},"Advanced Thermal Management: Hybrid Cooling",[18,5004,5005],{},"Direct liquid cooling can extract up to 80% of the heat of the SU, with the remaining heat requiring alternative dissipation methods. We’ve engineered a hybrid thermal management system optimized for computational efficiency and modularity.",[18,5007,5008],{},[283,5009],{"alt":285,"src":5010},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fmechanicalcooling_diagram_trans.webp",[18,5012,5013],{},[14,5014,5015],{},"Primary Cooling Strategy",[30,5017,5018,5034],{},[33,5019,5020,5023],{},[14,5021,5022],{},"GB300 NVL72 Compute Racks:",[30,5024,5025,5028,5031],{},[33,5026,5027],{},"80% thermal load managed via Direct Liquid Cooling (DLC).",[33,5029,5030],{},"20% residual heat captured through Rear Door Heat Exchangers (RDHx) or CRAH units.",[33,5032,5033],{},"DLC operates by establishing direct contact between heat-generating components and cold plates connected to coolant-filled tubes.",[33,5035,5036,5039],{},[14,5037,5038],{},"Auxiliary Infrastructure (Network\u002FManagement\u002FStorage):",[30,5040,5041,5044],{},[33,5042,5043],{},"100% RDHx cooling baseline.",[33,5045,5046],{},"Seamless transition path to DLC as thermal requirements evolve.",[18,5048,5049],{},[14,5050,5051],{},"The Critical Loops",[30,5053,5054,5060],{},[33,5055,5056,5059],{},[14,5057,5058],{},"Secondary Loop (CDUs):"," In-rack Coolant Distribution Units provide real-time monitoring of flow rates, temperatures, and pressure differentials, along with comprehensive leak detection.",[33,5061,5062,5065],{},[14,5063,5064],{},"Primary Loop (Heat Rejection):"," The primary thermal rejection system comprises three 1,125 kW Chiller\u002FDry coolers operating in an $N=2+1$ configuration. This eliminates water consumption typical in traditional cooling approaches.",[10,5067,5069],{"id":5068},"structural-environmental-design",[14,5070,5071],{},"Structural & Environmental Design",[18,5073,5074],{},"The core IT environment utilizes open steel frame construction systems, creating a secure and controlled environment featuring:",[30,5076,5077,5083,5089],{},[33,5078,5079,5082],{},[14,5080,5081],{},"Fire suppression:"," Argon-based systems (clean agent, non-conductive, oxygen-displacement mechanism).",[33,5084,5085,5088],{},[14,5086,5087],{},"Air purification:"," HEPA filtration maintaining ISO Class 8 cleanroom standards.",[33,5090,5091,5094],{},[14,5092,5093],{},"Access control:"," Biometric authentication and mantrap configurations.",[10,5096,5098],{"id":5097},"scalability-architected-for-growth",[14,5099,5100],{},"Scalability: Architected for Growth",[18,5102,5103],{},"One of the most critical advantages of our modular approach is seamless scalability that aligns precisely with NVIDIA’s reference architecture. Our modular data centers can be deployed to support these standardized scaling increments:",[18,5105,5106],{},[14,5107,5108],{},"Deployment Configurations:",[30,5110,5111,5117,5123,5129,5135,5141,5147,5153],{},[33,5112,5113,5116],{},[14,5114,5115],{},"1 SU"," (16 racks, 1.152 GPU) – Single scalable unit baseline deployment",[33,5118,5119,5122],{},[14,5120,5121],{},"2 SU"," (32 racks, 2.304 GPU) – Initial production scale",[33,5124,5125,5128],{},[14,5126,5127],{},"4 SU"," (64 racks, 4.608 GPU) – Enterprise-grade cluster",[33,5130,5131,5134],{},[14,5132,5133],{},"7 SU"," (112 racks, 8.064 GPU) – Large-scale training facility",[33,5136,5137,5140],{},[14,5138,5139],{},"8 SU"," (128 racks, 9.216 GPU) – High-density AI factory",[33,5142,5143,5146],{},[14,5144,5145],{},"9 SU"," (144 racks, 10.368 GPU) – High-density AI factory",[33,5148,5149,5152],{},[14,5150,5151],{},"18 SU"," (288 racks, 20.736 GPU) – Hyperscale deployment",[33,5154,5155,5158],{},[14,5156,5157],{},"36 SU"," (576 racks, 41.472 GPU) – Frontier-scale supercomputing cluster",[18,5160,5161],{},[283,5162],{"alt":285,"src":5163},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002F7su_cluster.webp",[10,5165,5167],{"id":5166},"conclusion-engineering-for-tomorrow",[14,5168,5169],{},"Conclusion: Engineering for Tomorrow",[18,5171,5172],{},"The GB300 NVL72 delivers approximately 140 kW rack-scale unit providing 1.1 exaFLOPS of FP4 computing, computational density that fundamentally redefines data center requirements.",[18,5174,5175],{},"Our 4-month deployment cycle represents the convergence of advanced thermal engineering, redundant power architectures, and purpose-optimized structural design. The future of AI infrastructure is modular, redundant, and optimized for extreme density It is also operational in sixteen weeks.",{"title":285,"searchDepth":673,"depth":673,"links":5177},[5178,5179,5180,5181,5182,5183,5184],{"id":4898,"depth":676,"text":4901},{"id":4910,"depth":676,"text":4913},{"id":4954,"depth":676,"text":4957},{"id":4999,"depth":676,"text":5002},{"id":5068,"depth":676,"text":5071},{"id":5097,"depth":676,"text":5100},{"id":5166,"depth":676,"text":5169},[688,1148,1149,5186,4865,5187,5188,5189,5190,5191,5192,5193,5194],"150kW Rack","DLC","High Density","Liquid Cooling","Modular Data Center","NVIDIA Blackwell Ultra","NVIDIA GB300","NVL72","Rapid Deployment","2025-11-20T14:10:19","In artificial intelligence infrastructure, speed is the foundation of competitive differentiation. From model training velocity to inference latency, every millisecond matters. But before any workload executes, there is a critical prerequisite that often determines success or failure: time to market. Traditional builds typically require 18–24 months. In the current AI cycle, that is simply too ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsuperpodmodflexfrontimage.webp",{},"https:\u002F\u002Feliovp.com\u002Fbuilding-the-engine-for-the-ai-race-the-4-month-path-to-nvidia-gb300-nvl72-power\u002F","\u002Fblog\u002Fbuilding-the-engine-for-the-ai-race-the-4-month-path-to-nvidia-gb300-nvl72-power",{"title":4887,"description":5196},"building-the-engine-for-the-ai-race-the-4-month-path-to-nvidia-gb300-nvl72-power","blog\u002Fbuilding-the-engine-for-the-ai-race-the-4-month-path-to-nvidia-gb300-nvl72-power","m0-ekxo9jJ7Sl_5sS3kgp0MdC_6hietyDtS-zDhcEWU",{"id":5206,"title":5207,"body":5208,"categories":5851,"date":5858,"description":5859,"extension":699,"image":5860,"meta":5861,"navigation":702,"originalUrl":5862,"path":5863,"seo":5864,"slug":5865,"stem":5866,"__hash__":5867},"blog\u002Fblog\u002Fwhy-cuda-translation-wont-unlock-amds-real-potential.md","Why “CUDA” Translation Won’t Unlock AMD’s Real Potential",{"type":7,"value":5209,"toc":5839},[5210,5215,5218,5242,5245,5248,5255,5269,5272,5279,5285,5290,5293,5296,5299,5320,5323,5326,5331,5334,5339,5346,5349,5355,5361,5367,5373,5395,5406,5413,5416,5434,5437,5443,5448,5462,5482,5488,5491,5499,5502,5509,5535,5538,5544,5549,5581,5592,5598,5603,5606,5635,5642,5649,5655,5660,5663,5678,5681,5692,5699,5705,5710,5713,5741,5744,5754,5772,5779,5785,5790,5793,5800,5803,5823,5826,5833],[18,5211,5212],{},[283,5213],{"alt":285,"src":5214},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fcudasection1.webp",[18,5216,5217],{},"Every few years, a new solution pops up promising the same dream:",[30,5219,5220,5227,5233,5236,5239],{},[33,5221,5222,5223,5226],{},"keep your ",[14,5224,5225],{},"CUDA"," codebase",[33,5228,5229,5230,5232],{},"target ",[14,5231,690],{}," (and maybe other accelerators)",[33,5234,5235],{},"no source rewrite",[33,5237,5238],{},"no HIP porting",[33,5240,5241],{},"“native performance”",[18,5243,5244],{},"On paper, that sounds perfect. Take your existing CUDA applications, swap out the toolchain, and suddenly you’re “portable.”",[18,5246,5247],{},"And to be fair: if you’re running research code or trying to get an internal tool to compile on a non-NVIDIA box, that can absolutely be useful.",[18,5249,5250,5251,5254],{},"But if you care about ",[14,5252,5253],{},"actual performance on AMD",",  the kind that:",[30,5256,5257,5260,5263,5266],{},[33,5258,5259],{},"reduces latency,",[33,5261,5262],{},"wins benchmarks,",[33,5264,5265],{},"squeezes every TFLOP from MI***-class accelerators,",[33,5267,5268],{},"and doesn’t send people “back to NVIDIA” after one bad experiment,",[18,5270,5271],{},"…then adopting a universal CUDA compatibility layer is the wrong long-term strategy.",[18,5273,5274,5275,5278],{},"Not because the engineers behind these toolchains aren’t smart (they are), but because ",[14,5276,5277],{},"CUDA-first compilation will always be playing catch-up"," with what AMD exposes natively through ROCm, HIP, and vendor-tuned libraries.",[10,5280,5282],{"id":5281},"why-this-approach-is-so-attractive",[14,5283,5284],{},"Why This Approach Is So Attractive",[18,5286,5287],{},[283,5288],{"alt":285,"src":5289},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fcudasection2.webp",[18,5291,5292],{},"These pitches are extremely compelling:",[18,5294,5295],{},"“Develop your application using CUDA once and deploy it across various GPU platforms.”",[18,5297,5298],{},"Concretely, these toolchains usually do something like this:",[30,5300,5301,5308,5311,5314,5317],{},[33,5302,5303,5304,5307],{},"Provide an ",[14,5305,5306],{},"nvcc-compatible compiler"," that accepts existing CUDA code, sometimes including inline PTX.",[33,5309,5310],{},"Target AMD GPUs via LLVM backends instead of NVIDIA’s drivers.",[33,5312,5313],{},"Implement the CUDA runtime, driver, and math APIs on top of AMD’s ROCm stack.",[33,5315,5316],{},"Ship wrapper libraries that map CUDA-X APIs (e.g., cuBLAS\u002FcuSOLVER) onto rocBLAS\u002FrocSOLVER and friends.",[33,5318,5319],{},"Maintain validation sets showing well-known CUDA projects compiling and running on AMD hardware.",[18,5321,5322],{},"From a developer’s point of view, it feels magical:",[18,5324,5325],{},"# On NVIDIA",[18,5327,5328],{},[223,5329,5330],{},"nvcc my_app.cu -o my_app_nvidia",[18,5332,5333],{},"# On “everything”",[18,5335,5336],{},[223,5337,5338],{},"nvcc my_app.cu -o my_app_other_gpu",[18,5340,5341,5342,5345],{},"For ",[14,5343,5344],{},"legacy CUDA-heavy HPC"," where a HIP\u002FSYCL\u002FROCm rewrite would be painful, this is honestly a nice option.",[18,5347,5348],{},"But that use case is very different from:",[5350,5351,5352],"blockquote",{},[18,5353,5354],{},"“We want state-of-the-art LLM inference and training performance on AMD, on par with or better than NVIDIA“",[18,5356,5357,5358,46],{},"Those are not the same problem. ",[14,5359,5360],{},"At all",[10,5362,5364],{"id":5363},"cuda-semantics-amd-semantics",[14,5365,5366],{},"CUDA Semantics ≠ AMD Semantics",[18,5368,5369,5370,46],{},"CUDA was designed around NVIDIA’s hardware model (warps, memory hierarchy, intrinsics, PTX, CUDA-X libraries). ",[14,5371,5372],{},"NVIDIA’s warp size is 32 threads",[18,5374,5375,5376,5379,5380,5385,5386,5389,5390],{},"AMD’s world is different: ",[14,5377,5378],{},"on CDNA\u002FGCN (Instinct class) the wavefront is 64 work-items",". ",[163,5381,5384],{"href":5382,"rel":5383},"https:\u002F\u002Frocm.docs.amd.com\u002Fprojects\u002Frocprofiler-compute\u002Fen\u002Fdocs-6.3.0\u002Fconceptual\u002Fpipeline-metrics.html?utm_source=chatgpt.com",[167],"AMD ROCm Documentation"," RDNA (consumer) introduced a native ",[14,5387,5388],{},"wave32"," mode, but CDNA (datacenter) remains wave64. ",[163,5391,5394],{"href":5392,"rel":5393},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FRDNA_%28microarchitecture%29?utm_source=chatgpt.com",[167],"Wikipedia",[18,5396,5397,5398,5405],{},"If AMD had decided from day one to be ",[14,5399,5400,5401,5404],{},"“",[14,5402,5403],{},"CUDA-native hardware",",”"," life would look very different. But that’s not reality, AMD chose its own architecture, software stack, and optimization paths.",[18,5407,5408,5409,5412],{},"Modern AMD guidance explicitly talks about ",[14,5410,5411],{},"MI***-specific optimization",": kernel shapes, memory tiling, GEMM tuning, and precision choices aligned to CDNA strengths.",[18,5414,5415],{},"If you start from CUDA as the “source of truth,” you’re asking a translation layer to:",[114,5417,5418,5421,5424,5427],{},[33,5419,5420],{},"Parse CUDA (often including inline PTX).",[33,5422,5423],{},"Convert it to an IR like LLVM.",[33,5425,5426],{},"Map that onto AMD’s ISA and ROCm stack.",[33,5428,5429,5430,5433],{},"Somehow emit code that competes with ",[14,5431,5432],{},"AMD-first"," kernels and libraries crafted for ROCm.",[18,5435,5436],{},"That’s a very high bar.",[10,5438,5440],{"id":5439},"concrete-example-1-32-vs-64-warpswavefronts",[14,5441,5442],{},"Concrete example #1,  32 vs 64 warps\u002Fwavefronts",[18,5444,5445],{},[283,5446],{"alt":285,"src":5447},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fcudaexample1.webp",[30,5449,5450,5456],{},[33,5451,5452,5455],{},[14,5453,5454],{},"NVIDIA:"," warp = 32 threads.",[33,5457,5458,5461],{},[14,5459,5460],{},"AMD CDNA\u002FGCN:"," wavefront = 64 work-items.",[18,5463,5464,5465,5468,5469,5472,5473,5476,5477],{},"If a kernel’s control flow, synchronization, or memory coalescing is implicitly tuned for ",[14,5466,5467],{},"warp32",", running it unchanged on ",[14,5470,5471],{},"wave64"," can strand lanes or force the compiler\u002Fruntime to add masking\u002Fshuffles, often cutting effective throughput. In other words: ",[14,5474,5475],{},"the “same” kernel can be ~2× off the peak simply from the execution width mismatch",", before we even talk about cache behavior or matrix-math usage. AMD dev material underscores how wave64 has different resource\u002Foccupancy characteristics than wave32. ",[163,5478,5481],{"href":5479,"rel":5480},"https:\u002F\u002Fgpuopen.com\u002Flearn\u002Foccupancy-explained\u002F?utm_source=chatgpt.com",[167],"gpuopen.com",[1230,5483,5485],{"id":5484},"performance-isnt-it-compiled",[14,5486,5487],{},"Performance Isn’t “It Compiled”",[18,5489,5490],{},"On paper, these toolchains often say:",[30,5492,5493,5496],{},[33,5494,5495],{},"“Ahead-of-time compilation, not emulation.”",[33,5497,5498],{},"“No inherent overhead vs native paths.”",[18,5500,5501],{},"For some classes of applications (classic CUDA-tuned HPC without mature ROCm ports), that can be reasonably true.",[18,5503,5504,5505,5508],{},"But for modern AI on AMD, performance comes from things ",[630,5506,5507],{},"way"," beyond “the CUDA syntax compiled”:",[30,5510,5511,5517,5523,5529],{},[33,5512,5513,5516],{},[14,5514,5515],{},"Fused kernels"," designed for AMD’s wavefront, caches, and memory system.",[33,5518,5519,5522],{},[14,5520,5521],{},"GEMM tuning"," across BF16\u002FFP16\u002FFP8 with AMD’s library paths and MFMA shapes.",[33,5524,5525,5528],{},[14,5526,5527],{},"FP8 enablement",": quantizing weights\u002Factivations to AMD’s FP8 (E4M3\u002FE5M2), FP8 KV-cache, scaling policies, and packing, wired up in frameworks like vLLM.",[33,5530,5531,5534],{},[14,5532,5533],{},"Tensor parallelism & communication"," tuned for MI***-class topology and ROCm collectives.",[18,5536,5537],{},"These are AMD-first engineering choices. A generic “CUDA front-end → AMD back-end” doesn’t conjure them out of thin air.",[10,5539,5541],{"id":5540},"concrete-example-2-fp8-isnt-plug-and-play",[14,5542,5543],{},"Concrete example #2,  FP8 isn’t plug-and-play",[18,5545,5546],{},[283,5547],{"alt":285,"src":5548},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fcudaexample2.webp",[18,5550,5551,5552,5555,5556,5559,5560,5564,5565,5568,5569,5572,5573,5575,5576],{},"On AMD ",[14,5553,5554],{},"CDNA3"," (MI300-class), FP8 uses ",[14,5557,5558],{},"E4M3\u002FE5M2"," formats and associated scaling\u002Fpacking. ",[163,5561,5384],{"href":5562,"rel":5563},"https:\u002F\u002Frocm.docs.amd.com\u002Fen\u002Flatest\u002Freference\u002Fprecision-support.html?utm_source=chatgpt.com",[167]," To run LLMs efficiently in FP8, you don’t just “compile” your CUDA. You typically ",[14,5566,5567],{},"pre-process or quantize weights"," and enable FP8 KV-cache\u002Factivations via AMD-aware flows, e.g., ",[14,5570,5571],{},"Quark"," quantization tutorials and ",[14,5574,696],{}," FP8 guides show explicit steps and configuration to hit the fast paths. ",[163,5577,5580],{"href":5578,"rel":5579},"https:\u002F\u002Frocm.docs.amd.com\u002Fprojects\u002Fai-developer-hub\u002Fen\u002Flatest\u002Fnotebooks\u002Fgpu_dev_optimize\u002Ffp8_quantization_quark_vllm.html?utm_source=chatgpt.com",[167],"VLLM Docs+3AMD ROCm Documentation+3rocm.blogs.amd.com+3",[18,5582,5583,5584,5587,5588,5591],{},"If a translation stack has to ",[14,5585,5586],{},"emulate"," missing FP8 behavior or falls back to non-optimal packing\u002Fscaling, the “portable” path quickly becomes ",[14,5589,5590],{},"measurably slower"," than AMD-native FP8 enablement.",[10,5593,5595],{"id":5594},"how-we-approach-it-paiton",[14,5596,5597],{},"How We Approach It (Paiton)",[18,5599,5600,5601,46],{},"This is exactly why, at Eliovp, we took the opposite approach with ",[14,5602,68],{},[18,5604,5605],{},"Paiton was and is being built for AMD first, not as a side effect of a CUDA translator. We:",[30,5607,5608,5611,5614,5621,5628],{},[33,5609,5610],{},"work directly on ROCm\u002FHIP,",[33,5612,5613],{},"integrate with vLLM\u002FSGLang rather than replace them,",[33,5615,5616,5617,5620],{},"write and tune ",[14,5618,5619],{},"custom kernels"," for MI***-class hardware,",[33,5622,5623,5624,5627],{},"fuse ops, tune GEMMs, and optimize ",[14,5625,5626],{},"FP8"," data paths,",[33,5629,5630,5631,5634],{},"optimize ",[14,5632,5633],{},"tensor parallelism"," and communication for AMD interconnect and topology.",[18,5636,5637,5638,5641],{},"In our public work, MI300X + Paiton beats newer-gen NVIDIA setups in real LLM inference while you keep your existing engine stack. That’s the point: a compatibility layer tries to make AMD behave like “CUDA-compatible hardware”; ",[14,5639,5640],{},"Paiton leans into what AMD actually is"," and extracts more.",[18,5643,5644,5645,5648],{},"Could a translation stack match that across FP8, MoE, TP, and real traffic? Only by re-implementing the same AMD-specific effort ",[630,5646,5647],{},"inside"," the translator, perpetually.",[10,5650,5652],{"id":5651},"the-ecosystem-risk-bad-first-impressions",[14,5653,5654],{},"The Ecosystem Risk: Bad First Impressions",[18,5656,5657],{},[283,5658],{"alt":285,"src":5659},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fcudaecosystem.webp",[18,5661,5662],{},"Typical pattern:",[114,5664,5665,5668,5675],{},[33,5666,5667],{},"Team with a CUDA codebase tries a “drop-in” tool for AMD.",[33,5669,5670,5671,5674],{},"Runs LLM\u002Fvision workloads; sees ",[14,5672,5673],{},"sub-optimal"," numbers vs NVIDIA baseline.",[33,5676,5677],{},"Internal conclusion: “We tried AMD. It’s slower.”",[18,5679,5680],{},"They rarely check whether:",[30,5682,5683,5686,5689],{},[33,5684,5685],{},"the layer used AMD FP8 fast paths or proper quantization,",[33,5687,5688],{},"ROCm-first kernels would have done better,",[33,5690,5691],{},"framework guidance suggests different AMD-specific flags.",[18,5693,5694,5695,5698],{},"They just see a dashboard. The result: ",[14,5696,5697],{},"“proof” AMD is slower",", even when AMD-first stacks show the opposite.",[10,5700,5702],{"id":5701},"the-maintenance-treadmill",[14,5703,5704],{},"The Maintenance Treadmill",[18,5706,5707],{},[283,5708],{"alt":285,"src":5709},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fcudamaintenancethreadmill-2.webp",[18,5711,5712],{},"To keep pace with AMD-native tooling, a “CUDA-everywhere” layer must constantly:",[30,5714,5715,5723,5730,5738],{},[33,5716,5717,5718],{},"track ROCm releases (new FP8 paths, GEMM lifts, library upgrades), ",[163,5719,5722],{"href":5720,"rel":5721},"https:\u002F\u002Frocm.docs.amd.com\u002Fen\u002Fdocs-7.1.0\u002Fhow-to\u002Frocm-for-ai\u002Finference-optimization\u002Fvllm-optimization.html?utm_source=chatgpt.com",[167],"AMD ROCm Documentation+1",[33,5724,5725,5726],{},"follow new Instinct GPUs and their tuning knobs, ",[163,5727,5722],{"href":5728,"rel":5729},"https:\u002F\u002Frocm.docs.amd.com\u002Fen\u002Flatest\u002Fcompatibility\u002Fcompatibility-matrix.html?utm_source=chatgpt.com",[167],[33,5731,5732,5733],{},"mirror techniques landing in vLLM for AMD (FP8 KV-cache, attention backends, etc.), ",[163,5734,5737],{"href":5735,"rel":5736},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Fstable\u002Ffeatures\u002Fquantization\u002Fquantized_kvcache.html?utm_source=chatgpt.com",[167],"VLLM Docs+1",[33,5739,5740],{},"and stay compatible with evolving CUDA semantics.",[18,5742,5743],{},"That’s a high-burn, reactive position, one step removed from where AMD and partners ship optimizations first.",[10,5745,5747],{"id":5746},"where-these-toolsdomake-sense-and-a-fair-note-on-consumer-gpus",[14,5748,5749,5750,5753],{},"Where These Tools",[630,5751,5752],{},"Do","Make Sense (and a fair note on consumer GPUs)",[30,5755,5756,5759,5762],{},[33,5757,5758],{},"Large legacy CUDA\u002FHPC codebases where you need “it runs!” quickly.",[33,5760,5761],{},"Functionality validation while planning a proper ROCm\u002FHIP path.",[33,5763,5764,5767,5768,5771],{},[14,5765,5766],{},"Consumer\u002Fgaming GPUs:"," RDNA introduced ",[14,5769,5770],{},"native wave32",", aligning with CUDA’s warp width. For desktop users, where ROCm setup can be trickier or officially limited, these layers can be a pragmatic bridge for experimentation.",[18,5773,5774,5775,5778],{},"This is a reasonable on-ramp. It’s just ",[14,5776,5777],{},"not"," how you showcase what AMD Instinct hardware can really do on LLMs.",[10,5780,5782],{"id":5781},"bottom-line",[14,5783,5784],{},"Bottom Line",[18,5786,5787],{},[283,5788],{"alt":285,"src":5789},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fcudabottomline.webp",[18,5791,5792],{},"We’re not saying “never use CUDA-on-AMD compilers or CUDA-to-HIP translators”.",[18,5794,5795,5796,5799],{},"We’re saying ",[14,5797,5798],{},"don’t judge AMD"," based on them.",[18,5801,5802],{},"If you want to see what AMD GPUs can actually do for AI:",[30,5804,5805,5808,5811,5814,5817,5820],{},[33,5806,5807],{},"use AMD-first kernels and libraries,",[33,5809,5810],{},"run ROCm-native tuning,",[33,5812,5813],{},"configure frameworks for AMD FP8 and attention backends,",[33,5815,5816],{},"choose parallelism strategies that fit MI***-class topology,",[33,5818,5819],{},"and work with teams who optimize for AMD by design, not by translation.",[33,5821,5822],{},"Or simply use Paiton, which comes with all necessary optimizations",[18,5824,5825],{},"Otherwise, you’re adding a compatibility tax and blaming the hardware for the bill.AMD didn’t choose to be a CUDA clone, and that’s fine.",[18,5827,5828,5829,5832],{},"Treat it as its ",[14,5830,5831],{},"own platform"," with its own strengths, and it will surprise you.",[18,5834,5835,5836,5838],{},"That’s the thesis behind ",[14,5837,68],{},", and why we keep beating expectations with AMD-first engineering.",{"title":285,"searchDepth":673,"depth":673,"links":5840},[5841,5842,5843,5844,5845,5846,5847,5848,5850],{"id":5281,"depth":676,"text":5284},{"id":5363,"depth":676,"text":5366},{"id":5439,"depth":676,"text":5442},{"id":5540,"depth":676,"text":5543},{"id":5594,"depth":676,"text":5597},{"id":5651,"depth":676,"text":5654},{"id":5701,"depth":676,"text":5704},{"id":5746,"depth":676,"text":5849},"Where These ToolsDoMake Sense (and a fair note on consumer GPUs)",{"id":5781,"depth":676,"text":5784},[688,689,68,1149,191,689,5852,5626,5853,5854,5855,5856,68,5857],"CUDA Translation","GPU Optimization","High Performance Computing","HIP","Kernel Tuning","ROCm","2025-11-12T14:48:37","Every few years, a new solution pops up promising the same dream: On paper, that sounds perfect. Take your existing CUDA applications, swap out the toolchain, and suddenly you’re “portable.” And to be fair: if you’re running research code or trying to get an internal tool to compile on a non-NVIDIA box, that can absolutely ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fchatgpt-image-nov-11-2025-09_16_10-pm-1.webp",{},"https:\u002F\u002Feliovp.com\u002Fwhy-cuda-translation-wont-unlock-amds-real-potential\u002F","\u002Fblog\u002Fwhy-cuda-translation-wont-unlock-amds-real-potential",{"title":5207,"description":5859},"why-cuda-translation-wont-unlock-amds-real-potential","blog\u002Fwhy-cuda-translation-wont-unlock-amds-real-potential","pj7R73ONpkEzkf5Jykr4MpEccQT40ubOxPyyG-4ggRs",{"id":5869,"title":5870,"body":5871,"categories":6593,"date":6599,"description":6600,"extension":699,"image":6601,"meta":6602,"navigation":702,"originalUrl":6603,"path":6604,"seo":6605,"slug":6606,"stem":6607,"__hash__":6608},"blog\u002Fblog\u002Fpaiton-the-simplest-way-to-supercharge-ai-inference.md","Paiton: The Simplest Way to Supercharge AI Inference",{"type":7,"value":5872,"toc":6579},[5873,5876,5879,5882,5885,5888,5895,5900,5903,5909,5915,5922,5925,5928,5938,5944,5947,5950,6040,6045,6051,6054,6061,6067,6169,6174,6187,6193,6199,6202,6205,6292,6297,6302,6308,6311,6314,6320,6390,6395,6400,6406,6412,6415,6422,6427,6432,6438,6441,6444,6447,6453,6464,6480,6483,6486,6493,6499,6502,6517,6524,6527,6533,6536],[18,5874,5875],{},"Let’s be honest, we’re not the marketing type.",[18,5877,5878],{},"We’ve never taken a cent of outside investment, never burned cash on ad campaigns, and never hired a sales army.",[18,5880,5881],{},"We just build things that work.",[18,5883,5884],{},"In today’s world, it seems the companies shouting the loudest often get the spotlight, while the ones doing the actual engineering quietly build the future.",[18,5886,5887],{},"We’re the latter.",[18,5889,5890,5891,5894],{},"Still, after a friendly nudge from someone who ",[630,5892,5893],{},"actually knows marketing",", we were told:",[5350,5896,5897],{},[18,5898,5899],{},"You guys should brag, the results are obvious and you’ve earned it.",[18,5901,5902],{},"So… here we go. 😉",[10,5904,5906],{"id":5905},"the-smarter-way-to-get-faster",[14,5907,5908],{},"The Smarter Way to Get Faster",[18,5910,5911,5912,5914],{},"In a market flooded with “revolutionary” engines and new runtimes, ",[14,5913,68],{}," takes a simpler, smarter path.",[18,5916,5917,5918,5921],{},"It’s not another inference engine, it’s a ",[14,5919,5920],{},"performance amplifier"," for the one you already use.",[18,5923,5924],{},"With Paiton, you don’t need to re-download your models, migrate your stack, or reconfigure your infrastructure.",[18,5926,5927],{},"You just plug it in, and watch your throughput jump.",[18,5929,5930,5931,5933,5934,5937],{},"If you’re using ",[14,5932,696],{}," or ",[14,5935,5936],{},"SGLang",", you’re already compatible.",[10,5939,5941],{"id":5940},"plug-in-power-up",[14,5942,5943],{},"Plug In, Power Up",[18,5945,5946],{},"Paiton slots right into your existing environment due to it being completely engine agnostic.",[18,5948,5949],{},"No new APIs. No retraining. No learning curve.",[298,5951,5952,5981],{},[301,5953,5954],{},[304,5955,5956,5961,5966,5971,5976],{},[307,5957,5958],{},[14,5959,5960],{},"Engine Stack",[307,5962,5963],{},[14,5964,5965],{},"Setup Change",[307,5967,5968],{},[14,5969,5970],{},"Model Support",[307,5972,5973],{},[14,5974,5975],{},"Typical Speedup",[307,5977,5978],{},[14,5979,5980],{},"Compatibility",[316,5982,5983,6008,6027],{},[304,5984,5985,5990,5995,6000,6003],{},[321,5986,5987],{},[14,5988,5989],{},"Paiton + vLLM",[321,5991,5992],{},[14,5993,5994],{},"None",[321,5996,5997],{},[14,5998,5999],{},"Always",[321,6001,6002],{},"+25–40%",[321,6004,6005],{},[14,6006,6007],{},"100%",[304,6009,6010,6013,6016,6019,6022],{},[321,6011,6012],{},"Custom Engine",[321,6014,6015],{},"High",[321,6017,6018],{},"Not Always",[321,6020,6021],{},"+20–30%",[321,6023,6024],{},[14,6025,6026],{},"Limited",[304,6028,6029,6032,6034,6036,6038],{},[321,6030,6031],{},"Standard vLLM",[321,6033,5994],{},[321,6035,5999],{},[321,6037,794],{},[321,6039,6007],{},[18,6041,6042],{},[283,6043],{"alt":285,"src":6044},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fthroughput_b32_dark_v2.jpg",[10,6046,6048],{"id":6047},"real-performance-real-roi",[14,6049,6050],{},"Real Performance. Real ROI.",[18,6052,6053],{},"We focused our energy where it matters, deep kernel fusion, custom GEMMs, and optimized inter-GPU communication.",[18,6055,6056,6057,6060],{},"The result: ",[14,6058,6059],{},"AMD MI300X with Paiton outperforms newer GPUs"," that cost significantly more, while running on your existing stack.",[10,6062,6064],{"id":6063},"performance-per-dollar",[14,6065,6066],{},"Performance per Dollar",[298,6068,6069,6096],{},[301,6070,6071],{},[304,6072,6073,6077,6082,6087,6092],{},[307,6074,6075],{},[14,6076,962],{},[307,6078,6079],{},[14,6080,6081],{},"Framework",[307,6083,6084],{},[14,6085,6086],{},"Tokens\u002Fsec",[307,6088,6089],{},[14,6090,6091],{},"Perf per $ (H200 = 1.0)",[307,6093,6094],{},[14,6095,782],{},[316,6097,6098,6113,6128,6144],{},[304,6099,6100,6103,6105,6108,6111],{},[321,6101,6102],{},"NVIDIA H200",[321,6104,696],{},[321,6106,6107],{},"3,862",[321,6109,6110],{},"1.00",[321,6112,794],{},[304,6114,6115,6117,6119,6122,6125],{},[321,6116,747],{},[321,6118,696],{},[321,6120,6121],{},"4,589",[321,6123,6124],{},"0.82",[321,6126,6127],{},"Slight bump, lower efficiency",[304,6129,6130,6133,6135,6138,6141],{},[321,6131,6132],{},"Stock MI300X",[321,6134,696],{},[321,6136,6137],{},"3,210",[321,6139,6140],{},"1.43",[321,6142,6143],{},"Strong",[304,6145,6146,6151,6154,6159,6164],{},[321,6147,6148],{},[14,6149,6150],{},"AMD MI300X + Paiton",[321,6152,6153],{},"vLLM \u002F SGLang",[321,6155,6156],{},[14,6157,6158],{},"4,613",[321,6160,6161],{},[14,6162,6163],{},"2.07",[321,6165,6166],{},[14,6167,6168],{},"Winner",[18,6170,6171],{},[283,6172],{"alt":285,"src":6173},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fperf_per_dollar_dark_v2.jpg",[18,6175,6176,999,6179,999,6184],{},[630,6177,6178],{},"Real-world efficiency: Paiton MI300X delivers over",[630,6180,6181],{},[14,6182,6183],{},"2×",[630,6185,6186],{},"the performance-per-dollar of NVIDIA H200.",[18,6188,6189,6192],{},[14,6190,6191],{},"Meaning:"," You get 40–50% higher performance at nearly half the cost per token, without changing your stack.",[10,6194,6196],{"id":6195},"keep-your-stack-keep-your-models-just-go-faster",[14,6197,6198],{},"Keep Your Stack. Keep Your Models. Just Go Faster.",[18,6200,6201],{},"Other tools promise performance but force you to rebuild everything.",[18,6203,6204],{},"Paiton simply optimizes what you’re already running.",[298,6206,6207,6229],{},[301,6208,6209],{},[304,6210,6211,6216,6221,6225],{},[307,6212,6213],{},[14,6214,6215],{},"Feature",[307,6217,6218],{},[14,6219,6220],{},"Standard Stack",[307,6222,6223],{},[14,6224,6012],{},[307,6226,6227],{},[14,6228,68],{},[316,6230,6231,6246,6261,6276],{},[304,6232,6233,6236,6239,6242],{},[321,6234,6235],{},"Model File Compatibility",[321,6237,6238],{},"Native",[321,6240,6241],{},"Often Requires Conversion",[321,6243,6244],{},[14,6245,6238],{},[304,6247,6248,6251,6254,6257],{},[321,6249,6250],{},"Engine Migration",[321,6252,6253],{},"Needed",[321,6255,6256],{},"Required",[321,6258,6259],{},[14,6260,5994],{},[304,6262,6263,6266,6269,6271],{},[321,6264,6265],{},"Stack Maintenance",[321,6267,6268],{},"Normal",[321,6270,6015],{},[321,6272,6273],{},[14,6274,6275],{},"Low",[304,6277,6278,6281,6284,6287],{},[321,6279,6280],{},"Immediate ROI",[321,6282,6283],{},"Medium",[321,6285,6286],{},"Slow",[321,6288,6289],{},[14,6290,6291],{},"Instant",[18,6293,6294],{},[283,6295],{"alt":285,"src":6296},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fcomplexity_vs_roi_dark_v2.jpg",[18,6298,6299],{},[630,6300,6301],{},"Paiton sits in the “sweet spot”: lowest complexity, highest ROI.",[10,6303,6305],{"id":6304},"the-business-math",[14,6306,6307],{},"The Business Math",[18,6309,6310],{},"Every millisecond shaved off inference time translates into real savings.",[18,6312,6313],{},"Here’s what that looks like in practice, based on real-world data from our own Qwen3-30B benchmarks:",[10,6315,6317],{"id":6316},"cost-per-1m-tokens-usd",[14,6318,6319],{},"Cost per 1M Tokens (USD)",[298,6321,6322,6341],{},[301,6323,6324],{},[304,6325,6326,6331,6336],{},[307,6327,6328],{},[14,6329,6330],{},"Setup",[307,6332,6333],{},[14,6334,6335],{},"Cost per 1M Tokens",[307,6337,6338],{},[14,6339,6340],{},"Efficiency Gain",[316,6342,6343,6353,6363,6373],{},[304,6344,6345,6347,6350],{},[321,6346,6102],{},[321,6348,6349],{},"$0.186",[321,6351,6352],{},"—",[304,6354,6355,6357,6360],{},[321,6356,747],{},[321,6358,6359],{},"$0.227",[321,6361,6362],{},"-22%",[304,6364,6365,6367,6370],{},[321,6366,6132],{},[321,6368,6369],{},"$0.130",[321,6371,6372],{},"+43%",[304,6374,6375,6380,6385],{},[321,6376,6377],{},[14,6378,6379],{},"Paiton MI300X",[321,6381,6382],{},[14,6383,6384],{},"$0.090",[321,6386,6387],{},[14,6388,6389],{},"+51%",[18,6391,6392],{},[283,6393],{"alt":285,"src":6394},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fcost_per_million_dark_v2.jpg",[18,6396,6397],{},[630,6398,6399],{},"Paiton cuts inference cost per million tokens by nearly half compared to H200.",[18,6401,6402,6405],{},[14,6403,6404],{},"Result:"," Faster inference, lower costs, and immediate ROI.",[10,6407,6409],{"id":6408},"startup-time-matters-too",[14,6410,6411],{},"Startup Time Matters Too",[18,6413,6414],{},"Inference isn’t everything, startup speed counts, especially for scaling large models.",[18,6416,6417,6418,6421],{},"For Llama-3.1-405B-Instruct-FP8-KV, Paiton drastically reduced cold-start time by ",[14,6419,6420],{},"46%"," overall.",[18,6423,6424],{},[283,6425],{"alt":285,"src":6426},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fstartup_breakdown_dark_v2.jpg",[18,6428,6429],{},[630,6430,6431],{},"From model loading to warm-up, Paiton slashed startup latency from 266s to 143s, a 46% reduction.",[10,6433,6435],{"id":6434},"why-it-matters",[14,6436,6437],{},"Why It Matters",[18,6439,6440],{},"There’s a myth that to get better performance, you need a new engine.",[18,6442,6443],{},"We disagree.",[18,6445,6446],{},"**Paiton proves smarter software beats newer hardware, and you can have both.**No downloads. No friction. Just better results.",[10,6448,6450],{"id":6449},"the-bottom-line",[14,6451,6452],{},"The Bottom Line",[18,6454,6455,6456,2197,6458,6460,6461,6463],{},"If you’re running ",[14,6457,696],{},[14,6459,5936],{},", or any AMD-based deployment, ",[14,6462,68],{}," is the easiest upgrade you’ll ever make.",[30,6465,6466,6469,6472,6475],{},[33,6467,6468],{},"No new engine",[33,6470,6471],{},"No model downloads",[33,6473,6474],{},"No workflow disruption",[33,6476,6477],{},[14,6478,6479],{},"Instant acceleration and lower cost",[18,6481,6482],{},"Faster inference isn’t just a benchmark, it’s a business advantage.",[18,6484,6485],{},"And Paiton is built to deliver it.",[18,6487,6488,6489],{},"Learn more at ",[163,6490,6491],{"href":6491,"rel":6492},"https:\u002F\u002Fai.eliovp.com\u002Fpaiton",[167],[10,6494,6496],{"id":6495},"stay-tuned",[14,6497,6498],{},"Stay Tuned",[18,6500,6501],{},"Make sure to keep an eye on us, we’re not done raising eyebrows just yet.",[18,6503,6504,6505,6508,6509,6512,6513,6516],{},"We’ve been deep in the trenches pushing ",[14,6506,6507],{},"FP8 optimization"," to new limits, both ",[14,6510,6511],{},"static and dynamic",", fine-tuned for ",[14,6514,6515],{},"Mixture-of-Experts (MoE)"," models.",[18,6518,6519,6520,6523],{},"The early results? Let’s just say they’re ",[630,6521,6522],{},"borderline unbelievable",",  and all achieved on AMD hardware, without changing your stack or relying on any proprietary APIs.",[18,6525,6526],{},"Stay tuned,  the next drop will redefine what “optimized inference” really means.",[10,6528,6530],{"id":6529},"sources-further-reading",[14,6531,6532],{},"Sources \u002F Further Reading",[18,6534,6535],{},"All performance claims in this article are based on our own internal benchmarking (with evaluation models available) on AMD MI300X systems using vLLM\u002FSGLang with Paiton enabled, as documented below:",[30,6537,6538,6552,6561,6570],{},[33,6539,6540,6541,6544,6545,6547,6548,1061],{},"[",[14,6542,6543],{},"Introduction to Paiton by Eliovp"," (guest post on AMD)",[1671,6546],{},"](",[163,6549,6550],{"href":6550,"rel":6551},"https:\u002F\u002Fwww.amd.com\u002Fen\u002Fblogs\u002F2025\u002Fintroduction-to-paiton-by-eliovp.html?utm_source=chatgpt.com",[167],[33,6553,6554],{},[163,6555,6558],{"href":6556,"rel":6557},"https:\u002F\u002Feliovp.com\u002Fstop-overpaying-paiton-mi300x-moe-beats-h200-b200-on-1m-tokens\u002F?utm_source=chatgpt.com",[167],[14,6559,6560],{},"Stop overpaying: Paiton MI300X MoE beats H200\u002FB200 on 1M tokens",[33,6562,6563],{},[163,6564,6567],{"href":6565,"rel":6566},"https:\u002F\u002Feliovp.com\u002Fmi300x-fp8-data%e2%80%91parallel-benchmarks-8-64-gpus-h200-left-behind-b200-within-reach\u002F?utm_source=chatgpt.com",[167],[14,6568,6569],{},"MI300X FP8 data-parallel benchmarks (8–64 GPUs): H200 left behind, B200 within reach",[33,6571,6572],{},[163,6573,6576],{"href":6574,"rel":6575},"https:\u002F\u002Feliovp.com\u002Fpaiton-dramatically-faster-startup-and-performance-for-llama-3-1-405b\u002F?utm_source=chatgpt.com",[167],[14,6577,6578],{},"Paiton: dramatically faster startup and performance for Llama-3.1-405B",{"title":285,"searchDepth":673,"depth":673,"links":6580},[6581,6582,6583,6584,6585,6586,6587,6588,6589,6590,6591,6592],{"id":5905,"depth":676,"text":5908},{"id":5940,"depth":676,"text":5943},{"id":6047,"depth":676,"text":6050},{"id":6063,"depth":676,"text":6066},{"id":6195,"depth":676,"text":6198},{"id":6304,"depth":676,"text":6307},{"id":6316,"depth":676,"text":6319},{"id":6408,"depth":676,"text":6411},{"id":6434,"depth":676,"text":6437},{"id":6449,"depth":676,"text":6452},{"id":6495,"depth":676,"text":6498},{"id":6529,"depth":676,"text":6532},[688,689,68,6594,6595,191,6596,6597,6598,5856,68,5936,696],"AI Inference","AMD Instinct","Cost Efficiency","High Throughput","Inference Optimization","2025-11-11T10:31:22","Let’s be honest, we’re not the marketing type.We’ve never taken a cent of outside investment, never burned cash on ad campaigns, and never hired a sales army.We just build things that work. In today’s world, it seems the companies shouting the loudest often get the spotlight, while the ones doing the actual engineering quietly build ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fpaiton-powaaah.webp",{},"https:\u002F\u002Feliovp.com\u002Fpaiton-the-simplest-way-to-supercharge-ai-inference\u002F","\u002Fblog\u002Fpaiton-the-simplest-way-to-supercharge-ai-inference",{"title":5870,"description":6600},"paiton-the-simplest-way-to-supercharge-ai-inference","blog\u002Fpaiton-the-simplest-way-to-supercharge-ai-inference","xDIct_GgSFM061ONdX5GIzv_S0soM12px_Z4kc6dggM",{"id":6610,"title":6611,"body":6612,"categories":7269,"date":7275,"description":7276,"extension":699,"image":7277,"meta":7278,"navigation":702,"originalUrl":7279,"path":7280,"seo":7281,"slug":7282,"stem":7283,"__hash__":7284},"blog\u002Fblog\u002Fstop-overpaying-paiton-mi300x-moe-beats-h200-b200-on-1m-tokens.md","Stop Overpaying: Paiton MI300X MoE Beats H200\u002FB200 on $\u002F1M Tokens",{"type":7,"value":6613,"toc":7260},[6614,6620,6626,6629,6636,6639,6642,6648,6651,6754,6758,6762,6765,6950,6956,6962,6965,6971,6975,6978,6982,6985,6990,6993,7005,7009,7105,7110,7115,7130,7136,7139,7165,7171,7174,7194,7198,7202,7209,7212,7219,7223],[18,6615,6616,6619],{},[14,6617,6618],{},"Short summary:"," We benchmarked Paiton with our new MoE support on Qwen\u002FQwen3-30B-A3B-Instruct-2507 to compare inference performance across several setups. Each configuration was run five times per batch size and we report the mean across runs.",[10,6621,6623],{"id":6622},"why-this-benchmark",[14,6624,6625],{},"Why this benchmark",[18,6627,6628],{},"Most published numbers use synthetic prompts or toy datasets. We focused on realistic conversational workloads (we always do) to highlight true latency and throughput behaviour. The model under test is Qwen\u002FQwen3-30B-A3B-Instruct-2507 running with Paiton’s custom-kernel runtime.",[18,6630,6631,6632,6635],{},"Meanwhile, billions are being funneled into shiny NVIDIA racks, yet here we are, ",[14,6633,6634],{},"crushing"," them with cheaper, “old-gen” AMD MI300X.",[18,6637,6638],{},"If your goal is outcomes per dollar, not logo per rack unit, you’re literally paying a premium to go slower.",[18,6640,6641],{},"While others rush to buy the newest GPUs, we’re focused on unlocking the full potential of both current and previous generations. GPU vendors often push new hardware before the previous generation is fully optimized and we’re here to change that.",[10,6643,6645],{"id":6644},"methodology",[14,6646,6647],{},"Methodology",[18,6649,6650],{},"To ensure the validity and applicability of our findings, we adhered to a meticulous benchmarking methodology:",[30,6652,6653,6659,6665,6671,6748],{},[33,6654,6655,6658],{},[14,6656,6657],{},"Model Under Test:"," We selected Qwen\u002FQwen3-30B-A3B-Instruct-2507, a representative large MoE model, to ensure our results are relevant to contemporary LLM deployments.",[33,6660,6661,6664],{},[14,6662,6663],{},"Dataset:"," Instead of synthetic data, we utilized realistic conversational traces from the ShareGPT-dataset. This choice is critical for accurately assessing performance in scenarios that mimic actual user interactions.",[33,6666,6667,6670],{},[14,6668,6669],{},"Output Length:"," To maintain consistency and reflect typical conversational turn lengths, the output length was capped at 256 tokens for all inference runs.",[33,6672,6673,6676],{},[14,6674,6675],{},"Hardware & Software Configurations:",[30,6677,6678,6703,6726],{},[33,6679,6680,6683],{},[14,6681,6682],{},"AMD MI300X (Paiton Optimized):",[30,6684,6685,6691,6697],{},[33,6686,6687,6690],{},[14,6688,6689],{},"ROCm Versions:"," Tested with both 6.4.1 and 7.0.0 to evaluate the impact of different software stacks.",[33,6692,6693,6696],{},[14,6694,6695],{},"VRAM:"," 192 GB HBM, providing substantial memory capacity for large models.",[33,6698,6699,6702],{},[14,6700,6701],{},"Software Stack:"," Paiton’s custom-kernel runtime with MoE support, alongside stock vLLM v0.10.0 and other competitor stacks for a holistic comparison.",[33,6704,6705,6708],{},[14,6706,6707],{},"NVIDIA H200 (Competitor Reference):",[30,6709,6710,6716,6721],{},[33,6711,6712,6715],{},[14,6713,6714],{},"CUDA Version:"," CUDA 13, representing the latest NVIDIA software environment.",[33,6717,6718,6720],{},[14,6719,6695],{}," 141 GB HBM.",[33,6722,6723,6725],{},[14,6724,6701],{}," vLLM 0.10.2 and competitor stacks.",[33,6727,6728,6731],{},[14,6729,6730],{},"NVIDIA B200 (Competitor Reference):",[30,6732,6733,6738,6743],{},[33,6734,6735,6737],{},[14,6736,6714],{}," CUDA 13.",[33,6739,6740,6742],{},[14,6741,6695],{}," 180 GB HBM.",[33,6744,6745,6747],{},[14,6746,6701],{}," vLLM 0.10.2 built from source and competitor stacks.",[33,6749,6750,6753],{},[14,6751,6752],{},"Benchmark Procedure:"," For each batch size in the set {1, 2, 4, 8, 16, 24, 32, 64, 128, 256}, inference jobs were executed five times end-to-end. Throughput (tokens\u002Fsec) and latency were meticulously recorded, with the mean across runs reported to mitigate statistical variance. Crucially, tokenization and generation settings were kept identical across all setups to ensure a fair comparison.",[10,6755,6756],{"id":753},[14,6757,754],{},[1230,6759,6761],{"id":6760},"throughput-and-cost-efficiency-analysis","Throughput and Cost-Efficiency Analysis",[18,6763,6764],{},"The following table presents the mean throughput (tokens\u002Fsec) observed for each configuration across varying batch sizes:",[298,6766,6767,6796],{},[301,6768,6769],{},[304,6770,6771,6776,6781,6786,6791],{},[307,6772,6773],{},[14,6774,6775],{},"Batch Size",[307,6777,6778],{},[14,6779,6780],{},"Paiton (MI300X) ROCm 6.4.1 \u002F vLLM 0.9.0",[307,6782,6783],{},[14,6784,6785],{},"MI300X ROCm 7.0 \u002F vLLM 0.10.0",[307,6787,6788],{},[14,6789,6790],{},"NVIDIA H200 CUDA 13 \u002F vLLM 0.10.2",[307,6792,6793],{},[14,6794,6795],{},"NVIDIA B200 CUDA 13 \u002F vLLM 0.10.2",[316,6797,6798,6815,6832,6849,6866,6883,6899,6916,6933],{},[304,6799,6800,6803,6806,6809,6812],{},[321,6801,6802],{},"1",[321,6804,6805],{},"189.37",[321,6807,6808],{},"162.24",[321,6810,6811],{},"189.51",[321,6813,6814],{},"180.11",[304,6816,6817,6820,6823,6826,6829],{},[321,6818,6819],{},"2",[321,6821,6822],{},"347.77",[321,6824,6825],{},"299.53",[321,6827,6828],{},"331.47",[321,6830,6831],{},"339.00",[304,6833,6834,6837,6840,6843,6846],{},[321,6835,6836],{},"4",[321,6838,6839],{},"580.07",[321,6841,6842],{},"496.88",[321,6844,6845],{},"551.72",[321,6847,6848],{},"610.16",[304,6850,6851,6854,6857,6860,6863],{},[321,6852,6853],{},"8",[321,6855,6856],{},"1,397.40",[321,6858,6859],{},"1,122.12",[321,6861,6862],{},"1,277.69",[321,6864,6865],{},"1,457.11",[304,6867,6868,6871,6874,6877,6880],{},[321,6869,6870],{},"16",[321,6872,6873],{},"2,855.09",[321,6875,6876],{},"1,989.90",[321,6878,6879],{},"2,222.61",[321,6881,6882],{},"2,840.19",[304,6884,6885,6887,6890,6893,6896],{},[321,6886,349],{},[321,6888,6889],{},"4,613.32",[321,6891,6892],{},"3,209.90",[321,6894,6895],{},"3,862.38",[321,6897,6898],{},"4,588.88",[304,6900,6901,6904,6907,6910,6913],{},[321,6902,6903],{},"64",[321,6905,6906],{},"7,234.05",[321,6908,6909],{},"5,573.63",[321,6911,6912],{},"6,687.79",[321,6914,6915],{},"8,368.82",[304,6917,6918,6921,6924,6927,6930],{},[321,6919,6920],{},"128",[321,6922,6923],{},"9,554.87",[321,6925,6926],{},"8,717.82",[321,6928,6929],{},"10,489.65",[321,6931,6932],{},"13,884.28",[304,6934,6935,6938,6941,6944,6947],{},[321,6936,6937],{},"256",[321,6939,6940],{},"14,672.35",[321,6942,6943],{},"12,587.27",[321,6945,6946],{},"16,129.35",[321,6948,6949],{},"21,980.31",[18,6951,6952],{},[283,6953],{"alt":285,"src":6954,"title":6955},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fimage-2-3.jpg","Chart",[1230,6957,6959],{"id":6958},"observations",[14,6960,6961],{},"Observations",[18,6963,6964],{},"While raw throughput numbers show competitive performance across the board, a deeper analysis, particularly concerning cost, reveals significant differentiators. Paiton, with its MoE support, demonstrates robust and smoothly scaling throughput, maintaining predictable latency even at higher batch sizes. This stands in contrast to some competitor stacks, which exhibit greater variance and potentially lower peak efficiency.",[10,6966,6968],{"id":6967},"token-economics",[14,6969,6970],{},"Token Economics",[1230,6972,6974],{"id":6973},"on-demand-cost-analysis","On-Demand Cost Analysis",[18,6976,6977],{},"To provide a truly meaningful comparison, we analyzed the cost per 1 million tokens, a critical metric for production deployments. This analysis considers the cheapest credible on-demand $\u002FGPU-hr pricing available for each GPU family, combined with our measured batch-size 32 throughput. Batch size 32 was chosen as it represents a common operating point for many conversational MoE deployments.",[1230,6979,6981],{"id":6980},"transparent-formula","Transparent formula",[18,6983,6984],{},"The cost per 1 million tokens is calculated using the following formula:",[18,6986,6987],{},[14,6988,6989],{},"$ per 1M tokens = (1,000,000 × R) \u002F (T × 3600)",[18,6991,6992],{},"Where:",[30,6994,6995,7000],{},[33,6996,6997],{},[14,6998,6999],{},"R = on-demand $\u002Fhr price per GPU",[33,7001,7002],{},[14,7003,7004],{},"T = tokens\u002Fsec (per GPU)",[1230,7006,7008],{"id":7007},"on-demand-results-cheapest-credible-listings","On-demand results (cheapest credible listings)",[298,7010,7011,7039],{},[301,7012,7013],{},[304,7014,7015,7019,7024,7029,7034],{},[307,7016,7017],{},[14,7018,772],{},[307,7020,7021],{},[14,7022,7023],{},"Throughput (tokens\u002Fsec)",[307,7025,7026],{},[14,7027,7028],{},"$\u002FGPU\u002Fhr",[307,7030,7031],{},[14,7032,7033],{},"Tokens per $",[307,7035,7036],{},[14,7037,7038],{},"$ per 1M tokens",[316,7040,7041,7057,7073,7089],{},[304,7042,7043,7047,7049,7052,7055],{},[321,7044,7045],{},[14,7046,6379],{},[321,7048,6889],{},[321,7050,7051],{},"$1.50",[321,7053,7054],{},"11,071,968",[321,7056,6384],{},[304,7058,7059,7064,7066,7068,7071],{},[321,7060,7061],{},[14,7062,7063],{},"Stock AMD MI300X",[321,7065,6892],{},[321,7067,7051],{},[321,7069,7070],{},"7,703,760",[321,7072,6369],{},[304,7074,7075,7079,7081,7084,7087],{},[321,7076,7077],{},[14,7078,6102],{},[321,7080,6895],{},[321,7082,7083],{},"$2.59",[321,7085,7086],{},"5,368,559",[321,7088,6349],{},[304,7090,7091,7095,7097,7100,7103],{},[321,7092,7093],{},[14,7094,747],{},[321,7096,6898],{},[321,7098,7099],{},"$3.75",[321,7101,7102],{},"4,405,325",[321,7104,6359],{},[18,7106,7107],{},[283,7108],{"alt":285,"src":7109,"title":6955},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fimage-2-4.jpg",[18,7111,7112],{},[630,7113,7114],{},"Cheapest on-demand pricing with batch-size 32 throughput → Paiton MI300X leads on $\u002F1M tokens.",[18,7116,7117,7120,7121,7124,7125,7129],{},[14,7118,7119],{},"Executive readout:"," The data clearly indicates that Paiton MI300X achieves approximately ",[14,7122,7123],{},"$0.090 per 1 million tokens",", a significant reduction compared to NVIDIA H200 (",[7126,7127,7128],"del",{},"$0.186) and B200 (","$0.227). This substantial cost differential represents tangible return on investment, not a marginal improvement.",[10,7131,7133],{"id":7132},"why-paiton-wins",[14,7134,7135],{},"Why Paiton wins",[18,7137,7138],{},"Paiton’s compelling performance and cost-efficiency can be attributed to several key architectural and software optimizations:",[30,7140,7141,7147,7153,7159],{},[33,7142,7143,7146],{},[14,7144,7145],{},"Kernel-Level MoE Optimization:"," Paiton’s custom kernels are designed to maximize arithmetic intensity and memory locality. By bypassing generic library overheads, these kernels achieve a more efficient execution path for MoE models, directly translating to higher performance.",[33,7148,7149,7152],{},[14,7150,7151],{},"Strategic HBM Leverage on MI300X:"," The AMD MI300X’s generous 192 GB of High Bandwidth Memory (HBM) is strategically utilized by Paiton. This ample memory capacity ensures that expert weights and key-value (KV) caches remain “hot” in memory, drastically reducing the need for costly memory swaps. This is a critical factor in achieving sustained high performance and, consequently, lower operational costs.",[33,7154,7155,7158],{},[14,7156,7157],{},"Reduced Kernel Calls:"," Paiton fuses multiple operations into larger, expert-aware kernels, which means far fewer kernel launches at runtime. By cutting the number of kernel launches, Paiton reduces latency, improves GPU occupancy, and increases arithmetic intensity. All of which translate to higher throughput and more stable performance compared to stacks that execute many small kernels.",[33,7160,7161,7164],{},[14,7162,7163],{},"ROCm Version Agnosticism:"," Paiton’s custom kernels are designed to be largely independent of specific ROCm version changes. This provides greater stability and flexibility for deployment, as performance remains consistent even with updates to the underlying ROCm software stack.",[10,7166,7168],{"id":7167},"reproducibility-pricing-transparency",[14,7169,7170],{},"Reproducibility & Pricing Transparency",[18,7172,7173],{},"We are committed to transparency and reproducibility.",[30,7175,7176,7182,7188],{},[33,7177,7178,7181],{},[14,7179,7180],{},"Throughput\u002FLatency Data:"," All raw data is included within this post.",[33,7183,7184,7187],{},[14,7185,7186],{},"Formulas:"," The formulas used for cost calculations are explicitly stated.",[33,7189,7190,7193],{},[14,7191,7192],{},"Pricing:"," On-demand pricing reflects the lowest credible public listings available at the time of this benchmark.",[10,7195,7196],{"id":640},[14,7197,643],{},[1230,7199,7201],{"id":7200},"the-new-cost-per-token-baseline","The New Cost-Per-Token Baseline",[18,7203,7204,7205,7208],{},"For organizations deploying large Mixture-of-Experts models in production, Paiton on AMD MI300X emerges as the undisputed ",[14,7206,7207],{},"cost-per-token baseline",". The benchmark results unequivocally confirm that Paiton leaves other solutions behind when serving models like Qwen3-30B-A3B-Instruct-2507 at scale. This advantage holds true for both on-demand cloud deployments and on-premises “owned iron” infrastructures. The alignment of superior performance with significant cost savings makes Paiton an indispensable solution for achieving optimal return on investment in LLM inference.",[18,7210,7211],{},"Furthermore, we are continuously optimizing our solutions and anticipate even more impressive results in the near future. Stay tuned for upcoming updates, including detailed FP8 results, which we will be posting very soon.",[18,7213,7214,7218],{},[163,7215,7217],{"href":6491,"rel":7216},[167],"Reach out and let’s talk"," :)",[18,7220,7221],{},[14,7222,4559],{},[114,7224,7225,7232,7239,7246,7253],{},[33,7226,7227],{},[163,7228,7231],{"href":7229,"rel":7230},"https:\u002F\u002Fwww.supermicro.com\u002Fen\u002Fproducts\u002Fsystem\u002Fgpu\u002F8u\u002Fas%20-8125gs-tnmr2",[167],"Supermicro GPU System AS-8125GS-TNMR2",[33,7233,7234],{},[163,7235,7238],{"href":7236,"rel":7237},"https:\u002F\u002Fwww.amd.com\u002Fen\u002Fproducts\u002Faccelerators\u002Finstinct\u002Fmi300\u002Fmi300x.html",[167],"AMD Instinct MI300X",[33,7240,7241],{},[163,7242,7245],{"href":7243,"rel":7244},"https:\u002F\u002Fgithub.com\u002FROCm\u002Fvllm",[167],"ROCm\u002Fvllm GitHub",[33,7247,7248],{},[163,7249,7252],{"href":7250,"rel":7251},"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm",[167],"vllm-project\u002Fvllm GitHub",[33,7254,7255],{},[163,7256,7259],{"href":7257,"rel":7258},"https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen3-30B-A3B-Instruct-2507",[167],"Hugging Face Qwen3-30B-A3B-Instruct-2507",{"title":285,"searchDepth":673,"depth":673,"links":7261},[7262,7263,7264,7265,7266,7267,7268],{"id":6622,"depth":676,"text":6625},{"id":6644,"depth":676,"text":6647},{"id":753,"depth":676,"text":754},{"id":6967,"depth":676,"text":6970},{"id":7132,"depth":676,"text":7135},{"id":7167,"depth":676,"text":7170},{"id":640,"depth":676,"text":643},[688,689,68,7270,191,7271,6598,7272,7273,747,6102,68,7274],"AI Benchmarks","Cost per Token","Mixture of Experts","MoE","Qwen3","2025-09-26T13:36:18","Short summary: We benchmarked Paiton with our new MoE support on Qwen\u002FQwen3-30B-A3B-Instruct-2507 to compare inference performance across several setups. Each configuration was run five times per batch size and we report the mean across runs. Why this benchmark Most published numbers use synthetic prompts or toy datasets. We focused on realistic conversational workloads (we always ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fhulkvshulkpaitonwins.webp",{},"https:\u002F\u002Feliovp.com\u002Fstop-overpaying-paiton-mi300x-moe-beats-h200-b200-on-1m-tokens\u002F","\u002Fblog\u002Fstop-overpaying-paiton-mi300x-moe-beats-h200-b200-on-1m-tokens",{"title":6611,"description":7276},"stop-overpaying-paiton-mi300x-moe-beats-h200-b200-on-1m-tokens","blog\u002Fstop-overpaying-paiton-mi300x-moe-beats-h200-b200-on-1m-tokens","mKv4S0bKNdRP1lD3zL8p1jNXavP2fKsHml4agsAXlQs",{"id":7286,"title":7287,"body":7288,"categories":8170,"date":8177,"description":8178,"extension":699,"image":8179,"meta":8180,"navigation":702,"originalUrl":8181,"path":8182,"seo":8183,"slug":8184,"stem":8185,"__hash__":8186},"blog\u002Fblog\u002Fagentic-ai-but-make-it-local-from-inbox-to-insight-to-action-en.md","Agentic AI, But Make It Local: From Inbox to Insight to Action",{"type":7,"value":7289,"toc":8145},[7290,7297,7300,7307,7313,7316,7327,7333,7377,7383,7389,7421,7427,7468,7474,7530,7536,7568,7574,7581,7598,7601,7607,7643,7646,7652,7678,7686,7693,7700,7706,7712,7715,7723,7734,7741,7747,7750,7761,7767,7809,7815,7821,7852,7858,7898,7904,7957,7963,7991,7997,8013,8030,8033,8039,8072,8075,8081,8105,8113,8124,8131,8136,8142],[18,7291,7292,7293,1061],{},"(",[163,7294,7296],{"href":7295},"#agentic_ai_nl","Nederlandse versie",[18,7298,7299],{},"We’ve built production-ready, local-first agentic AI that plugs into your existing email stack, auto-creates tickets, classifies messages, extracts multi-question threads, reads PDFs, spots invoices\u002Fquotes, analyzes images (yes, damage detection), and pushes structured reports into your systems, no dependency on OpenAI, Google, or Microsoft unless you want it. Tailor-made models trained on your data, on your hardware, inside your compliance perimeter. Net-net: faster cycle times, lower cost-to-serve, stronger control.",[18,7301,7302],{},[163,7303,7306],{"href":7304,"rel":7305},"https:\u002F\u002Fwww.youtube.com\u002Fembed\u002FIwQ3MsbXJ9g?feature=oembed",[167],"demo agentic ai   english",[10,7308,7310],{"id":7309},"why-this-why-now",[14,7311,7312],{},"Why this, why now",[18,7314,7315],{},"Most “AI” launched over the last 18 months has been a thin UI over a vendor API. Great for prototyping. Risky for production. You hand over data, get usage-based billing surprises, and hope the model behaves.",[18,7317,7318,7319,7322,7323,7326],{},"Our approach is different: ",[14,7320,7321],{},"agentic AI that lives where your data lives",". On-prem or private cloud, trained on your corpus, instrumented end-to-end, and wired into the apps you already use. Think ",[14,7324,7325],{},"operational leverage"," without the compliance headaches.",[10,7328,7330],{"id":7329},"what-the-agent-actually-does-today",[14,7331,7332],{},"What the agent actually does (today)",[30,7334,7335,7345,7351,7357,7367],{},[33,7336,7337,7340,7341,7344],{},[14,7338,7339],{},"Email ingestion + triage",": Connects to your mail provider via standard APIs. Auto-creates tickets, tags intents, routes to the right queue, pulls out ",[630,7342,7343],{},"multiple"," questions from one email, and drafts structured follow-ups.",[33,7346,7347,7350],{},[14,7348,7349],{},"Document intelligence",": Reads PDFs and office docs; detects invoices, quotes, contracts; extracts line items, totals, dates, counterparties; flags anomalies.",[33,7352,7353,7356],{},[14,7354,7355],{},"Vision in the loop",": Processes attached images; detects damage or non-conformities; links findings to the relevant case\u002Fticket and generates a templated report with evidence.",[33,7358,7359,7362,7363,7366],{},[14,7360,7361],{},"Everything local",": Runs with ",[14,7364,7365],{},"no external AI APIs"," by default. If you want Gemini\u002FCopilot\u002FChatGPT in the mix, we can BYO connectors and policy-gate them.",[33,7368,7369,7372,7373,7376],{},[14,7370,7371],{},"Customer-specific models",": We fine-tune and\u002For distill ",[14,7374,7375],{},"tailor-made models"," on your data, your terminology, your workflows. Your moat, not ours.",[10,7378,7380],{"id":7379},"real-world-use-cases-the-fun-part",[14,7381,7382],{},"Real-world use cases (the fun part)",[1230,7384,7386],{"id":7385},"horizontal-works-in-almost-any-org",[14,7387,7388],{},"Horizontal (works in almost any org)",[30,7390,7391,7397,7403,7409,7415],{},[33,7392,7393,7396],{},[14,7394,7395],{},"Level-1 support triage"," → classify, summarize, suggest resolution steps, auto-respond when policy allows; escalate with a crisp state-of-play.",[33,7398,7399,7402],{},[14,7400,7401],{},"Accounts Payable"," → detect invoices from email\u002FPDF, extract fields, 3-way match hints, push to ERP; flag duplicates and date\u002Famount anomalies.",[33,7404,7405,7408],{},[14,7406,7407],{},"Sales ops"," → parse inbound RFQs, extract requirements, map to SKUs, draft quotes or route to the right AE with a discovery checklist.",[33,7410,7411,7414],{},[14,7412,7413],{},"Procurement"," → auto-classify supplier emails, track delivery promises vs. PO, generate nudges on slips or compliance gaps.",[33,7416,7417,7420],{},[14,7418,7419],{},"ITSM"," → create tickets from any inbox, enrich with logs\u002Fattachments, propose runbook steps; close-loop summaries for audit.",[1230,7422,7424],{"id":7423},"industry-flavored",[14,7425,7426],{},"Industry-flavored",[30,7428,7429,7435,7441,7447,7453,7462],{},[33,7430,7431,7434],{},[14,7432,7433],{},"Insurance"," → claims intake from email\u002Fphotos; damage detection on images; pre-fill claim forms; detect missing evidence; draft adjuster notes.",[33,7436,7437,7440],{},[14,7438,7439],{},"Construction & field services"," → site photos → defect\u002Fdamage detection; daily report generation; change-order deltas; safety checklist extraction.",[33,7442,7443,7446],{},[14,7444,7445],{},"Manufacturing"," → quality non-conformance from line photos; auto-create CAPA tickets; supplier doc (CoA\u002FCoC) parsing with exception flags.",[33,7448,7449,7452],{},[14,7450,7451],{},"Logistics"," → BOL\u002Fpacking list extraction; exception emails → auto-case creation; POD verification from images; SLA breach alerts.",[33,7454,7455,999,7458,7461],{},[14,7456,7457],{},"Healthcare admin",[630,7459,7460],{},"(non-diagnostic)"," → referral\u002Fintake document parsing, authorization form extraction, appointment coordination mail triage.",[33,7463,7464,7467],{},[14,7465,7466],{},"Legal\u002FCompliance"," → matter intake classification, clause extraction from attachments, PII detection\u002Fredaction suggestions, audit-grade trails.",[10,7469,7471],{"id":7470},"why-local-first-agentic-ai-wins-for-operators-and-cisos",[14,7472,7473],{},"Why local-first agentic AI wins (for operators and CISOs)",[30,7475,7476,7482,7492,7498,7508,7518,7524],{},[33,7477,7478,7481],{},[14,7479,7480],{},"Data governance by design",": Keep PHI\u002FPII\u002Fcommercials inside your perimeter. No third-party model sees your data unless you explicitly opt in.",[33,7483,7484,7487,7488,7491],{},[14,7485,7486],{},"Deterministic cost structure",": Run on your hardware or dedicated instances; ",[14,7489,7490],{},"predictable TCO"," vs. API shock pricing.",[33,7493,7494,7497],{},[14,7495,7496],{},"Latency + throughput",": Move tokens, not internet round-trips. Lower p95 latency, higher concurrency, happier SLAs.",[33,7499,7500,7503,7504,7507],{},[14,7501,7502],{},"Customization depth",": Fine-tune on your knowledge base, email tone, escalation thresholds; build ",[14,7505,7506],{},"company-native"," behaviors, not generic.",[33,7509,7510,7513,7514,7517],{},[14,7511,7512],{},"Observability & auditability",": Full event logs, prompts, tool calls, and outputs. ",[14,7515,7516],{},"Reproducible"," decisions with human-in-the-loop when required.",[33,7519,7520,7523],{},[14,7521,7522],{},"Air-gapped optionality",": Operate disconnected for sensitive environments; sync when policy allows.",[33,7525,7526,7529],{},[14,7527,7528],{},"Vendor risk mitigation",": No singular dependency. Pluggable model layer with fallbacks.",[10,7531,7533],{"id":7532},"under-the-hood-at-a-high-level",[14,7534,7535],{},"Under the hood (at a high level)",[30,7537,7538,7544,7550,7556,7562],{},[33,7539,7540,7543],{},[14,7541,7542],{},"Orchestrator",": Event-driven agent runtime with tool-use (email, ticketing, OCR\u002Fvision, DB\u002FRAG, report templates), policies, and guardrails.",[33,7545,7546,7549],{},[14,7547,7548],{},"Models",": Local LLMs for language; local VLMs for vision. Fine-tuned adapters (LoRA\u002FPEFT) per customer domain; RAG over your data sources.",[33,7551,7552,7555],{},[14,7553,7554],{},"Connectors",": IMAP\u002FGraph\u002FGmail for mail; Jira\u002FServiceNow\u002FYouTrack\u002FZendesk for tickets; SAP\u002FNetSuite\u002FOdoo for ERP; S3\u002FSharePoint\u002FFile shares for docs.",[33,7557,7558,7561],{},[14,7559,7560],{},"Safety & governance",": PII detection, redaction options, policy gates, confidence thresholds, fallback to human.",[33,7563,7564,7567],{},[14,7565,7566],{},"MLOps",": Versioned models & prompts, offline evals with real workloads, canary releases, drift monitors, rollbacks.",[10,7569,7571],{"id":7570},"what-agentic-means-in-practice",[14,7572,7573],{},"What “agentic” means in practice",[18,7575,7576,7577,7580],{},"This isn’t just “autocomplete for emails.” The agent plans steps, calls tools, checks itself, and ",[14,7578,7579],{},"takes action"," within policy bounds. Example:",[114,7582,7583,7586,7589,7592,7595],{},[33,7584,7585],{},"Ingest inbound email with two requests + PDF invoice + 3 photos",[33,7587,7588],{},"Parse entities from the PDF; detect damage from photos",[33,7590,7591],{},"Open\u002Fappend to the right ticket; populate fields + attach evidence",[33,7593,7594],{},"Draft a reply with answers to both questions, pending confirmations highlighted",[33,7596,7597],{},"If confidence \u003C policy threshold, route to human with a one-screen brief",[18,7599,7600],{},"Zero swivel-chair. Maximum throughput.",[10,7602,7604],{"id":7603},"kpis-our-customers-track",[14,7605,7606],{},"KPIs our customers track",[30,7608,7609,7615,7621,7627,7632,7638],{},[33,7610,7611,7614],{},[14,7612,7613],{},"MTTR \u002F First Response Time"," ↓",[33,7616,7617,7620],{},[14,7618,7619],{},"Ticket deflection"," (auto-resolved) ↑",[33,7622,7623,7626],{},[14,7624,7625],{},"Case handling capacity per agent"," ↑",[33,7628,7629,7626],{},[14,7630,7631],{},"Invoice touchless rate",[33,7633,7634,7637],{},[14,7635,7636],{},"Exception accuracy"," (precision\u002Frecall on flags) ↑",[33,7639,7640,7614],{},[14,7641,7642],{},"Audit readiness time",[18,7644,7645],{},"We’ll help you baseline, then run an A\u002FB or time-boxed pilot so the deltas are crystal clear.",[10,7647,7649],{"id":7648},"implementation-playbook-no-drama",[14,7650,7651],{},"Implementation playbook (no drama)",[114,7653,7654,7660,7666,7672],{},[33,7655,7656,7659],{},[14,7657,7658],{},"Discovery (1–2 weeks)",": Map systems, data, policies, high-value workflows. Identify redlines.",[33,7661,7662,7665],{},[14,7663,7664],{},"Pilot (4–6 weeks)",": One or two golden paths (e.g., L1 support + AP). Local deployment, model adaptation, HIL review.",[33,7667,7668,7671],{},[14,7669,7670],{},"Scale-out",": Expand connectors, raise autonomy thresholds, add departments.",[33,7673,7674,7677],{},[14,7675,7676],{},"Operate",": Metrics, drift checks, periodic re-training on new data, light prompt\u002Fmodel maintenance.",[10,7679,7681,7682,7685],{"id":7680},"faq-the-spicy-takeswhy-not-just-use-copilotgeminichatgptgreat-for-knowledge-work-in-the-open-for-repeatable-high-stakes-workflows-on-sensitive-data-local-first-yields-better-control-lower-latency-and-a-saner-cost-curve-we-can-still-wire-those-services-in-with-strict-policy-gates-if-theres-a-clear-roi","**FAQ (the spicy takes)Why not just use Copilot\u002FGemini\u002FChatGPT?**Great for knowledge work in the open. For ",[14,7683,7684],{},"repeatable, high-stakes workflows on sensitive data",", local-first yields better control, lower latency, and a saner cost curve. We can still wire those services in, with strict policy gates, if there’s a clear ROI.",[18,7687,7688,7689,7692],{},"**Are local models “good enough”?**For targeted domains with the right data + adapters: absolutely. We optimize for your tasks, not leaderboard vanity. The win comes from ",[14,7690,7691],{},"domain fit + tool use",", not model size alone.",[18,7694,7695,7696,7699],{},"**Will this replace my team?**No. It ",[14,7697,7698],{},"removes grunt work"," and boosts throughput. Your experts handle exceptions and edge cases; the agent handles the rest.",[288,7701,7703],{"id":7702},"call-to-action",[14,7704,7705],{},"Call to action",[18,7707,7708,7709,46],{},"If your inbox is your backlog and your PDFs run your business, it’s time to operationalize AI, ",[14,7710,7711],{},"on your turf",[18,7713,7714],{},"Want the video, architecture deep-dive, and a pilot outline tailored to your stack? Ping us. We’ll bring the demo, you bring the gnarly emails.",[7716,7717,7719,7722],"h1",{"id":7718},"agentic-ai-maar-dan-lokaal-van-inbox-naar-inzicht-naar-actie-nl",[14,7720,7721],{},"Agentic AI, maar dan Lokaal: van Inbox naar Inzicht naar Actie"," (NL)",[18,7724,7725,7726,7729,7730,7733],{},"We hebben productieklare, ",[14,7727,7728],{},"lokaal-eerst"," AI-agenten gebouwd die in je bestaande e-mail stack pluggen, automatisch tickets aanmaken, berichten classificeren, meerdere vragen uit één mail halen, PDF’s lezen, facturen\u002Foffertes herkennen, afbeeldingen analyseren (ja, schadedetectie) en gestructureerde rapporten terug in je systemen pushen, ",[14,7731,7732],{},"zonder"," afhankelijkheid van OpenAI, Google of Microsoft, tenzij jij dat expliciet wilt. Modellen op maat, getraind op jouw data, op jouw hardware, binnen jouw compliance grenzen. Net-net: snellere doorlooptijd, lagere cost-to-serve, maximale controle.",[18,7735,7736],{},[163,7737,7740],{"href":7738,"rel":7739},"https:\u002F\u002Fwww.youtube.com\u002Fembed\u002FL9Ivpf9dWQ4?feature=oembed",[167],"Agentic AI Demo",[10,7742,7744],{"id":7743},"waarom-dit-waarom-nu",[14,7745,7746],{},"Waarom dit, waarom nu",[18,7748,7749],{},"De meeste “AI” van de afgelopen 18 maanden is een dunne UI over een vendor-API. Prima voor een proof-of-concept. Risicovol voor productie. Je levert data in, krijgt gebruiksurprises en hoopt dat het model zich gedraagt.",[18,7751,7752,7753,7756,7757,7760],{},"Onze aanpak is anders: ",[14,7754,7755],{},"AI-agenten die draaien waar jouw data draait",". On-prem of in je private cloud, getraind op jouw corpus, volledig instrumented en gekoppeld aan de tools die je al gebruikt. Denk: ",[14,7758,7759],{},"operationele leverage"," zonder de compliance-hoofdpijn.",[10,7762,7764],{"id":7763},"wat-de-agent-vandaag-al-doet",[14,7765,7766],{},"Wat de agent vandaag al doet",[30,7768,7769,7779,7784,7789,7799],{},[33,7770,7771,7774,7775,7778],{},[14,7772,7773],{},"E-mailinname + triage",": Verbindt via standaard-API’s met je mailprovider. Maakt automatisch tickets aan, tagt intents, routeert naar de juiste queue, haalt ",[630,7776,7777],{},"meerdere"," vragen uit één mail en stelt gestructureerde follow-ups voor.",[33,7780,7781,7783],{},[14,7782,7349],{},": Leest PDF’s en Office-docs; detecteert facturen, offertes en contracten; extraheert regelitems, totalen, data en partijen; markeert anomalieën.",[33,7785,7786,7788],{},[14,7787,7355],{},": Verwerkt bijgevoegde afbeeldingen; detecteert schade of non-conformiteiten; koppelt bevindingen aan het juiste ticket en genereert een getemplate rapport met bewijs.",[33,7790,7791,7794,7795,7798],{},[14,7792,7793],{},"Alles lokaal",": ",[14,7796,7797],{},"Geen externe AI-API’s"," by default. Wil je Gemini\u002FCopilot\u002FChatGPT erbij? Kan, met strikte policy-gates.",[33,7800,7801,7804,7805,7808],{},[14,7802,7803],{},"Klant-specifieke modellen",": We fine-tunen en\u002Fof distillen ",[14,7806,7807],{},"maatwerkmodellen"," op jouw data, jargon en workflows. Jouw moat, niet de onze.",[10,7810,7812],{"id":7811},"real-life-use-cases-de-leuke-dingen",[14,7813,7814],{},"Real-life use cases (de leuke dingen)",[1230,7816,7818],{"id":7817},"horizontaal-werkt-in-bijna-elke-organisatie",[14,7819,7820],{},"Horizontaal (werkt in bijna elke organisatie)",[30,7822,7823,7829,7835,7841,7847],{},[33,7824,7825,7828],{},[14,7826,7827],{},"L1 support triage"," → classificeren, samenvatten, oplossingsstappen voorstellen, auto-reply waar beleid het toelaat; escaleren met een haarscherp statusoverzicht.",[33,7830,7831,7834],{},[14,7832,7833],{},"Crediteuren (AP)"," → facturen detecteren uit mail\u002FPDF, velden extraheren, 3-way-match-hints, push naar ERP; dubbele facturen en datum\u002Fbedrag-afwijkingen vlaggen.",[33,7836,7837,7840],{},[14,7838,7839],{},"Sales operations"," → RFQ’s parsen, requirements extraheren, mappen naar SKU’s, conceptoffertes opstellen of naar de juiste AE routeren met een discovery-checklist.",[33,7842,7843,7846],{},[14,7844,7845],{},"Inkoop"," → leveranciersmails auto-classificeren, leverbeloftes vs. PO volgen, nudges genereren bij slip\u002Fcompliance-gap.",[33,7848,7849,7851],{},[14,7850,7419],{}," → tickets aanmaken vanuit inboxen, verrijken met logs\u002Fbijlagen, runbook-stappen voorstellen; sluit-samenvattingen voor audit.",[1230,7853,7855],{"id":7854},"branche-specifiek",[14,7856,7857],{},"Branche-specifiek",[30,7859,7860,7866,7872,7878,7884,7893],{},[33,7861,7862,7865],{},[14,7863,7864],{},"Verzekeringen"," → schademeldingen uit mail\u002Ffoto’s; schadedetectie; claimformulieren pre-fillen; ontbrekend bewijs detecteren; conceptrapport voor de schade-expert.",[33,7867,7868,7871],{},[14,7869,7870],{},"Bouw & field services"," → werffoto’s → defect\u002Fschade-detectie; dagrapporten genereren; meer-\u002Fminderwerk-deltas; veiligheidschecklists extraheren.",[33,7873,7874,7877],{},[14,7875,7876],{},"Maakindustrie"," → kwaliteits-NC uit lijnfoto’s; CAPA-tickets auto-creëren; leveranciersdocumenten (CoA\u002FCoC) parsen met uitzonderingsvlaggen.",[33,7879,7880,7883],{},[14,7881,7882],{},"Logistiek"," → CMR\u002FBOL\u002Fpacking list extractie; exception-mails → cases; POD-verificatie op foto; SLA-breukalerts.",[33,7885,7886,999,7889,7892],{},[14,7887,7888],{},"Zorgadministratie",[630,7890,7891],{},"(niet-diagnostisch)"," → verwijzingen\u002Fintake-documenten parsen, machtigingsformulieren extraheren, afspraken-triage uit mail.",[33,7894,7895,7897],{},[14,7896,7466],{}," → zaak-intake classificatie, clausule-extractie uit bijlagen, PII-detectie\u002Fanon-suggesties, audit-trails.",[10,7899,7901],{"id":7900},"waarom-lokaal-eerst-ai-agenten-winnen-voor-operators-én-cisos",[14,7902,7903],{},"Waarom lokaal-eerst AI-agenten winnen (voor operators én CISO’s)",[30,7905,7906,7912,7922,7927,7936,7945,7951],{},[33,7907,7908,7911],{},[14,7909,7910],{},"Datagovernance by design",": Houd PHI\u002FPII\u002Fcommercials binnen je eigen perimeter. Geen derde partij ziet je data tenzij je het expliciet toestaat.",[33,7913,7914,7917,7918,7921],{},[14,7915,7916],{},"Deterministische kosten",": Draai op je eigen hardware of dedicated instances; ",[14,7919,7920],{},"voorspelbare TCO"," in plaats van API-schokprijzen.",[33,7923,7924,7926],{},[14,7925,7496],{},": Tokens lokaal bewegen, geen internet-roundtrips. Lagere p95, hogere concurrency, sterkere SLA’s.",[33,7928,7929,7932,7933,7935],{},[14,7930,7931],{},"Diepe maatwerk-fit",": Fine-tune op je kennisbank, tone-of-voice, escalatiemeters; bouw ",[14,7934,7506],{}," gedrag i.p.v. generiek.",[33,7937,7938,7940,7941,7944],{},[14,7939,7512],{},": Volledige eventlogs, prompts, toolcalls en outputs. ",[14,7942,7943],{},"Reproduceerbare"," beslissingen met human-in-the-loop waar nodig.",[33,7946,7947,7950],{},[14,7948,7949],{},"Air-gapped optioneel",": Draai desnoods volledig disconnected; synchroniseer wanneer beleid het toelaat.",[33,7952,7953,7956],{},[14,7954,7955],{},"Vendor-risico mitigatie",": Geen single point of failure. Pluggable modellayer met fallbacks.",[10,7958,7960],{"id":7959},"onder-de-motorkap-hoog-over",[14,7961,7962],{},"Onder de motorkap (hoog over)",[30,7964,7965,7970,7976,7981,7986],{},[33,7966,7967,7969],{},[14,7968,7542],{},": Event-gedreven agent-runtime met tool-use (mail, ticketing, OCR\u002Fvision, DB\u002FRAG, rapporttemplates), policies en guardrails.",[33,7971,7972,7975],{},[14,7973,7974],{},"Modellen",": Lokale LLM’s voor taal; lokale VLM’s voor visie. Klant-adapters (LoRA\u002FPEFT) per domein; RAG over jouw bronnen.",[33,7977,7978,7980],{},[14,7979,7554],{},": IMAP\u002FGraph\u002FGmail voor e-mail; Jira\u002FServiceNow\u002FYouTrack\u002FZendesk voor tickets; SAP\u002FNetSuite\u002FOdoo voor ERP; S3\u002FSharePoint\u002Ffileshares voor documenten.",[33,7982,7983,7985],{},[14,7984,7560],{},": PII-detectie, redactie-opties, policy-gates, confidence-thresholds, fallback naar human.",[33,7987,7988,7990],{},[14,7989,7566],{},": Versiebeheer van modellen & prompts, offline evaluaties met echte workloads, canary releases, drift-monitoring, rollbacks.",[10,7992,7994],{"id":7993},"wat-agentisch-in-de-praktijk-betekent",[14,7995,7996],{},"Wat “agentisch” in de praktijk betekent",[18,7998,7999,8000,2197,8003,2197,8006,2201,8009,8012],{},"Dit is niet “autocomplete voor e-mail”. De agent ",[14,8001,8002],{},"plant",[14,8004,8005],{},"roept tools aan",[14,8007,8008],{},"controleert zichzelf",[14,8010,8011],{},"neemt actie"," binnen beleidsgrenzen. Voorbeeld:",[114,8014,8015,8018,8021,8024,8027],{},[33,8016,8017],{},"Inkomende mail met twee vragen + PDF-factuur + 3 foto’s",[33,8019,8020],{},"Entiteiten uit de PDF extraheren; schade op foto’s detecteren",[33,8022,8023],{},"Het juiste ticket openen\u002Faanvullen; velden vullen + bewijs toevoegen",[33,8025,8026],{},"Antwoord op beide vragen opstellen, met open punten duidelijk gemarkeerd",[33,8028,8029],{},"Bij confidence \u003C policy-drempel naar een medewerker met een one-screen briefing",[18,8031,8032],{},"Geen swivel-chair. Maximale throughput.",[10,8034,8036],{"id":8035},"kpis-die-klanten-meten",[14,8037,8038],{},"KPI’s die klanten meten",[30,8040,8041,8045,8051,8056,8061,8067],{},[33,8042,8043,7614],{},[14,8044,7613],{},[33,8046,8047,8050],{},[14,8048,8049],{},"Ticket-deflectie"," (auto-opgelost) ↑",[33,8052,8053,7626],{},[14,8054,8055],{},"Cases per agent",[33,8057,8058,7626],{},[14,8059,8060],{},"Touchless-rate facturen",[33,8062,8063,8066],{},[14,8064,8065],{},"Exception-nauwkeurigheid"," (precision\u002Frecall) ↑",[33,8068,8069,7614],{},[14,8070,8071],{},"Tijd naar audit-gereed",[18,8073,8074],{},"We helpen je eerst te baseline’en en draaien dan een A\u002FB of time-boxed pilot zodat de deltas glashelder zijn.",[10,8076,8078],{"id":8077},"implementatie-playbook-zonder-drama",[14,8079,8080],{},"Implementatie-playbook (zonder drama)",[114,8082,8083,8089,8095,8100],{},[33,8084,8085,8088],{},[14,8086,8087],{},"Discovery (1–2 weken)",": Systemen, data, policies en high-value workflows mappen. Red lines bepalen.",[33,8090,8091,8094],{},[14,8092,8093],{},"Pilot (4–6 weken)",": Eén of twee golden paths (bv. L1 support + AP). Lokale deployment, model-adaptatie, HIL-review.",[33,8096,8097,8099],{},[14,8098,7670],{},": Connectors uitbreiden, autonomie-drempels opschroeven, extra afdelingen aansluiten.",[33,8101,8102,8104],{},[14,8103,7676],{},": Metrics, drift-checks, periodieke retraining op nieuwe data, lichte prompt\u002Fmodel-maintenance.",[10,8106,8108,8109,8112],{"id":8107},"faq-met-een-beetje-pitwaarom-geen-copilotgeminichatgpttop-voor-open-knowledge-work-voor-repeteerbare-kritieke-workflows-op-gevoelige-data-levert-lokaal-eerst-betere-controle-lagere-latency-en-een-gezondere-kostenlijn-we-kunnen-die-diensten-alsnog-inprikken-onder-strikte-policies-als-de-roi-er-is","**FAQ (met een beetje pit)Waarom geen Copilot\u002FGemini\u002FChatGPT?**Top voor open knowledge work. Voor ",[14,8110,8111],{},"repeteerbare, kritieke workflows op gevoelige data"," levert lokaal-eerst betere controle, lagere latency en een gezondere kostenlijn. We kunnen die diensten alsnog inprikken, onder strikte policies, als de ROI er is.",[18,8114,8115,8116,8119,8120,8123],{},"**Zijn lokale modellen “goed genoeg”?**Voor gerichte domeinen met de juiste data + adapters: absoluut. We optimaliseren voor ",[14,8117,8118],{},"jouw taken",", niet voor leaderboard-records. De winst komt uit ",[14,8121,8122],{},"domeinfit + tool-use",", niet alleen uit modelgrootte.",[18,8125,8126,8127,8130],{},"**Vervangt dit mijn team?**Nee. Het ",[14,8128,8129],{},"verwijdert repetitief werk"," en verhoogt throughput. Jouw experts pakken uitzonderingen en edge cases; de agent doet de rest.",[288,8132,8134],{"id":8133},"call-to-action-1",[14,8135,7705],{},[18,8137,8138,8139,46],{},"Is je inbox je backlog en draaien PDF’s je bedrijf? Dan is het tijd om AI te operationaliseren, ",[14,8140,8141],{},"op jouw terrein",[18,8143,8144],{},"Wil je de video, architectuur-deep-dive en een pilotvoorstel afgestemd op jouw stack? Neem contact op. Wij brengen de demo, jij de taaie mails.",{"title":285,"searchDepth":673,"depth":673,"links":8146},[8147,8148,8149,8150,8151,8152,8153,8154,8155,8157,8169],{"id":7309,"depth":676,"text":7312},{"id":7329,"depth":676,"text":7332},{"id":7379,"depth":676,"text":7382},{"id":7470,"depth":676,"text":7473},{"id":7532,"depth":676,"text":7535},{"id":7570,"depth":676,"text":7573},{"id":7603,"depth":676,"text":7606},{"id":7648,"depth":676,"text":7651},{"id":7680,"depth":676,"text":8156},"**FAQ (the spicy takes)Why not just use Copilot\u002FGemini\u002FChatGPT?**Great for knowledge work in the open. For repeatable, high-stakes workflows on sensitive data, local-first yields better control, lower latency, and a saner cost curve. We can still wire those services in, with strict policy gates, if there’s a clear ROI.",{"id":7702,"depth":673,"text":7705,"children":8158},[8159,8160,8161,8162,8163,8164,8165,8166,8167],{"id":7743,"depth":676,"text":7746},{"id":7763,"depth":676,"text":7766},{"id":7811,"depth":676,"text":7814},{"id":7900,"depth":676,"text":7903},{"id":7959,"depth":676,"text":7962},{"id":7993,"depth":676,"text":7996},{"id":8035,"depth":676,"text":8038},{"id":8077,"depth":676,"text":8080},{"id":8107,"depth":676,"text":8168},"**FAQ (met een beetje pit)Waarom geen Copilot\u002FGemini\u002FChatGPT?**Top voor open knowledge work. Voor repeteerbare, kritieke workflows op gevoelige data levert lokaal-eerst betere controle, lagere latency en een gezondere kostenlijn. We kunnen die diensten alsnog inprikken, onder strikte policies, als de ROI er is.",{"id":8133,"depth":673,"text":7705},[688,689,3971,1149,3972,8171,8172,8173,8174,3977,3979,969,8175,8176],"Damage Detection","Document Processing","Email Automation","Invoice Extraction","Ticket Automation","Workflow Automation","2025-09-16T13:09:00","(Nederlandse versie) We’ve built production-ready, local-first agentic AI that plugs into your existing email stack, auto-creates tickets, classifies messages, extracts multi-question threads, reads PDFs, spots invoices\u002Fquotes, analyzes images (yes, damage detection), and pushes structured reports into your systems, no dependency on OpenAI, Google, or Microsoft unless you want it. Tailor-made models trained on your data, ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Ffrontfotoblog.webp",{},"https:\u002F\u002Feliovp.com\u002Fagentic-ai-but-make-it-local-from-inbox-to-insight-to-action-en\u002F","\u002Fblog\u002Fagentic-ai-but-make-it-local-from-inbox-to-insight-to-action-en",{"title":7287,"description":8178},"agentic-ai-but-make-it-local-from-inbox-to-insight-to-action-en","blog\u002Fagentic-ai-but-make-it-local-from-inbox-to-insight-to-action-en","fafNzcVek1xW5zLHU6a_irrPGDK2LpvQs2pqR6epebk",{"id":8188,"title":8189,"body":8190,"categories":9657,"date":9659,"description":9660,"extension":699,"image":9661,"meta":9662,"navigation":702,"originalUrl":9663,"path":9664,"seo":9665,"slug":9666,"stem":9667,"__hash__":9668},"blog\u002Fblog\u002Fmi300x-fp8-data%e2%80%91parallel-benchmarks-8-64-gpus-h200-left-behind-b200-within-reach.md","MI300X FP8 Data‑Parallel Benchmarks (8–64 GPUs): H200 Left Behind, B200 Within Reach",{"type":7,"value":8191,"toc":9645},[8192,8195,8204,8207,8210,8215,8259,8262,8267,8278,8284,8287,8421,8424,8430,8433,8440,8447,8450,8459,8466,8469,8475,8478,8485,8488,8494,8500,8503,8511,8514,8521,8527,8534,8540,8543,8545,8550,8553,8559,8565,8567,8573,8579,8582,8588,8595,8598,8601,8606,8609,8614,8617,8623,8627,8928,8933,8937,9123,9126,9139,9144,9147,9150,9232,9276,9281,9316,9322,9325,9437,9442,9447,9468,9474,9477,9480,9483,9490,9507,9513,9539,9543,9546,9549,9552,9555,9561,9564,9573,9576,9581,9585],[18,8193,8194],{},"At ElioVP, we’re all about pushing AI inference past the limits, and packaging every squeeze of performance into a plug‑and‑play runtime.",[18,8196,8197,8198,8203],{},"Remember ",[163,8199,8202],{"href":8200,"rel":8201},"https:\u002F\u002Feliovp.com\u002Fpaiton-fp8-beats-nvidias-h200-on-amds-mi300x\u002F",[167],"our last blog",", where Paiton’s FP8 pipeline on AMD’s MI300X completely outclassed NVIDIA’s H200? Well, buckle up, because we’ve gone back to the drawing board.",[18,8205,8206],{},"This time, we’re loading Llama-3.1-8B-Instruct-FP8-KV, the leaner, meaner FP8‑quantized Llama variant, into not 8 GPUs but 64 virtual GPUs carved out of a single MI300X server.",[18,8208,8209],{},"Powered by vLLM and Paiton’s kernel magic, we expected modest gains in multi‑tenant scaling…what we got instead was an unexpected, yet amazing surprise and a near dead‑heat with Nvidia’s B200’s.",[18,8211,8212],{},[14,8213,8214],{},"Why did we do this?",[30,8216,8217,8223,8229,8235,8241,8247,8253],{},[33,8218,8219,8222],{},[14,8220,8221],{},"Maximize utilization",": Slice the silicon so every tenant only pays for, and uses, exactly the VRAM and compute they need.",[33,8224,8225,8228],{},[14,8226,8227],{},"Elastic multi‑tenancy",": Spin up isolated vGPUs in seconds, eliminating noisy‑neighbor slowdowns and siloed resource contention.",[33,8230,8231,8234],{},[14,8232,8233],{},"Granular SLAs",": Tailor QoS per slice, ultra‑low latency for chatbots, bulk throughput for batch jobs, without juggling hardware.",[33,8236,8237,8240],{},[14,8238,8239],{},"Cost‑efficient scaling",": Right‑size your compute footprint (and your budget) by renting mini‑GPUs instead of the whole chip.",[33,8242,8243,8246],{},[14,8244,8245],{},"Rapid CI\u002FCD provisioning",": Integrate GPU slices into your pipeline for instant A\u002FB tests, blue\u002Fgreen rollouts, and regression benchmarks.",[33,8248,8249,8252],{},[14,8250,8251],{},"Fault isolation",": Contain OOMs and driver hiccups at the slice level, so one bad job doesn’t take down the entire server.",[33,8254,8255,8258],{},[14,8256,8257],{},"Future‑proof flexibility",": Re‑slice on the fly to match new model footprints or quant formats, no forklift upgrades required.",[18,8260,8261],{},"With these building blocks in place, we set out to see how far Paiton could stretch inference on a partitioned MI300X, and the numbers? Let’s just say they’ll make you sit up and take notice.",[18,8263,8264],{},[14,8265,8266],{},"Goals",[30,8268,8269,8272,8275],{},[33,8270,8271],{},"Evaluate the inference scalability of Paiton on MI300X when using GPU partitioning.",[33,8273,8274],{},"Measure latency and throughput of Llama 3.1 8B in FP8 format using vLLM.",[33,8276,8277],{},"Validate memory efficiency and kernel fusion benefits of plug-and-play Paiton models.",[10,8279,8281],{"id":8280},"benchmarking-testbed-methodology",[14,8282,8283],{},"Benchmarking Testbed & Methodology",[18,8285,8286],{},"Our benchmarking method follows a clear set of rules and steps. This makes sure our tests are open and reproducible.",[30,8288,8289,8305,8342,8352,8368,8384,8389],{},[33,8290,8291,237,8294],{},[14,8292,8293],{},"Hardware Configuration",[30,8295,8296,8299,8302],{},[33,8297,8298],{},"8 x AMD MI300x",[33,8300,8301],{},"8 x Nvidia H200",[33,8303,8304],{},"8 x Nvidia B200",[33,8306,8307,237,8310],{},[14,8308,8309],{},"Inference Library",[30,8311,8312,8319,8326,8334],{},[33,8313,8314,8315],{},"AMD MI300x (Paiton): ",[163,8316,8318],{"href":7243,"rel":8317},[167],"vLLM v0.9.0",[33,8320,8321,8322],{},"AMD MI300x (AITER): ",[163,8323,8325],{"href":7243,"rel":8324},[167],"v0.9.2",[33,8327,8328,8329,8333],{},"NVIDIA H200: ",[163,8330,8332],{"href":7250,"rel":8331},[167],"v0.10.0","  (V1 mode)",[33,8335,8336,8337,8341],{},"NVIDIA B200: ",[163,8338,8340],{"href":7250,"rel":8339},[167],"v0.10.1","  (V1 mode) (Had to build from source to support the B200 arch)",[33,8343,8344,999,8347],{},[14,8345,8346],{},"Language Model:",[163,8348,8351],{"href":8349,"rel":8350},"https:\u002F\u002Fhuggingface.co\u002Famd\u002FLlama-3.1-8B-Instruct-FP8-KV",[167],"Llama-3.1-8B-Instruct-FP8-KV",[33,8353,8354,237,8357],{},[14,8355,8356],{},"Driver Stack",[30,8358,8359,8362,8365],{},[33,8360,8361],{},"AMD MI300x: ROCm 6.4.2",[33,8363,8364],{},"NVIDIA H200: CUDA 12.8.1",[33,8366,8367],{},"NVIDIA B200: CUDA 12.8.1",[33,8369,8370,8373],{},[14,8371,8372],{},"Framework:",[30,8374,8375,8378,8381],{},[33,8376,8377],{},"AMD MI300x: Torch 2.7.1+rocm6.3",[33,8379,8380],{},"NVIDIA H200: Torch 2.7.1+cu128",[33,8382,8383],{},"NVIDIA B200: Torch 2.9.0.dev+cu128",[33,8385,8386,8388],{},[14,8387,6775],{},": 1024",[33,8390,8391,8394,8395],{},[14,8392,8393],{},"Measurement Protocol",": Each benchmark was run 10 times, and the numbers we report are overall averages. This helps reduce the effect of temporary system changes. Our careful measurement steps include:\n",[30,8396,8397,8403,8409,8415],{},[33,8398,8399,8402],{},[14,8400,8401],{},"Startup Times",": Important for checking how long it takes to load the model and get the system ready.",[33,8404,8405,8408],{},[14,8406,8407],{},"Cold-Start TTFT (Time to First Token)",": Measures how long it takes from a new request until the first generated token appears. This is key for how quickly interactive applications respond.",[33,8410,8411,8414],{},[14,8412,8413],{},"Steady-State TTFT",": Checks the TTFT after the system has been running steadily, showing typical performance under constant use.",[33,8416,8417,8420],{},[14,8418,8419],{},"End-to-End Latency Metrics",": Gives a full picture of the time it takes for a complete inference request, from sending input to getting the final output.",[18,8422,8423],{},"This detailed method provides a strong way to check the specific performance details of Paiton in busy, partitioned GPU environments.",[10,8425,8427],{"id":8426},"data-parallelism-without-partitioning",[14,8428,8429],{},"Data Parallelism Without Partitioning",[18,8431,8432],{},"Our first approach was to try to utilize vllm’s built in “–data-parallel-size” option, we quickly realized that this was not going to work out of the box and would require some serious modification. So instead, we took a different approach.",[18,8434,8435,8436,1061],{},"To run the benchmarks across 8 containers using vLLM, we first followed the official NGINX load balancing guide (",[163,8437,8438],{"href":8438,"rel":8439},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Fstable\u002Fdeployment\u002Fnginx.html",[167],[114,8441,8442],{},[33,8443,8444],{},[14,8445,8446],{},"NGINX Configuration",[18,8448,8449],{},"Here is the load balancing configuration we used in \u002Fetc\u002Fnginx\u002Fnginx.conf:",[8451,8452,8457],"pre",{"className":8453,"code":8455,"language":8456,"meta":285},[8454],"language-text","upstream backend {\n    least_conn;\n    server vllm0:8000 max_fails=3 fail_timeout=10000s;\n    server vllm1:8000 max_fails=3 fail_timeout=10000s;\n    server vllm2:8000 max_fails=3 fail_timeout=10000s;\n    server vllm3:8000 max_fails=3 fail_timeout=10000s;\n    server vllm4:8000 max_fails=3 fail_timeout=10000s;\n    server vllm5:8000 max_fails=3 fail_timeout=10000s;\n    server vllm6:8000 max_fails=3 fail_timeout=10000s;\n    server vllm7:8000 max_fails=3 fail_timeout=10000s;\n}\n\nserver {\n    listen 80;\n    location \u002F {\n        proxy_pass http:\u002F\u002Fbackend;\n        proxy_set_header Host $host;\n        proxy_set_header X-Real-IP $remote_addr;\n        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n        proxy_set_header X-Forwarded-Proto $scheme;\n    }\n}\n","text",[223,8458,8455],{"__ignoreMap":285},[114,8460,8461],{"start":673},[33,8462,8463],{},[14,8464,8465],{},"Launching the Docker Containers",[18,8467,8468],{},"We used the following script to launch 8 containers using incremental device and port numbers:",[8451,8470,8473],{"className":8471,"code":8472,"language":8456,"meta":285},[8454],"#!\u002Fbin\u002Fbash\n\necho \"Starting vLLM containers with incremental configuration…\"\n\nfor i in {0..7}; do\n    device_num=$((128 + (i * 8)))\n    device_path=\"\u002Fdev\u002Fdri\u002FrenderD${device_num}\"\n    port=$((8080 + i))\n    container_name=\"vllm${i}\"\n\n    echo \"Starting container ${container_name} on port ${port} with device ${device_path}…\"\n\n    docker run -itd \\\n        –ipc host \\\n        -v \u002Fdata:\u002Fdata \\\n        –network vllm_nginx \\\n        -e VLLM_ROCM_USE_AITER=True \\\n        -e HF_HOME=root\u002F.cache\u002Fhuggingface \\\n        -e HF_HUB_CACHE=\u002Froot\u002F.cache\u002Fhuggingface\u002Fhub \\\n        –device=\u002Fdev\u002Fkfd \\\n        –device=${device_path} \\\n        –group-add video \\\n        -p ${port}:8000 \\\n        –name ${container_name} \\\n        rocm\u002Fvllm:latest \\\n        vllm serve \\\n        amd\u002FLlama-3.1-8B-Instruct-FP8-KV \\\n        –num-scheduler-steps 10 \\\n        –kv-cache-dtype fp8 \\\n        –max-model-len 4096\n\n    if [ $? -eq 0 ]; then\n        echo \"✓ Container ${container_name} started successfully\"\n    else\n        echo \"✗ Failed to start container ${container_name}\"\n    fi\n\n    echo \"—\"\ndone\n\necho \"All containers started. Summary:\"\necho \"Containers: vllm0 through vllm7\"\necho \"Ports: 8081 through 8088\"\necho \"Devices: renderD128 through renderD184 (in steps of 8)\"\n",[223,8474,8472],{"__ignoreMap":285},[18,8476,8477],{},"This line, device_num=$((128 + (i * 8))), was necessary because of leftover render device entries in \u002Fdev\u002Fdri\u002F from previous GPU partitioning. Even after resetting the partitions, the device numbers did not reset to their original state. As a result, we had to offset each device path to correctly reference the available render nodes.",[114,8479,8480],{"start":676},[33,8481,8482],{},[14,8483,8484],{},"Benchmarking",[18,8486,8487],{},"Finally, we ran the following command to benchmark across all containers:",[8451,8489,8492],{"className":8490,"code":8491,"language":8456,"meta":285},[8454],"for i in {1..10}; do\n    echo \"=== Running benchmark iteration $i\u002F10 ===\"\n    python3 ~\u002Fvllm\u002Fbenchmarks\u002Fbenchmark_serving.py \\\n      –backend vllm \\\n      –model amd\u002FLlama-3.1-8B-Instruct-FP8-KV \\\n      –dataset-name sharegpt \\\n      –dataset-path ~\u002Fvllm\u002FShareGPT_V3_unfiltered_cleaned_split.json \\\n      –num-prompts 1024 \\\n      –random-range-ratio 1.0 \\\n      –percentile-metrics ttft,tpot,itl,e2el \\\n      –sharegpt-output-len 256\n    echo \"=== Completed iteration $i\u002F10 ===\"\n    echo\ndone\n",[223,8493,8491],{"__ignoreMap":285},[10,8495,8497],{"id":8496},"data-parallelism-with-partitioning",[14,8498,8499],{},"Data Parallelism With Partitioning",[18,8501,8502],{},"The most important first step was to partitionize our GPUs.",[18,8504,8505,8506,46],{},"This was very straightforward and easy to do following ",[163,8507,8510],{"href":8508,"rel":8509},"https:\u002F\u002Finstinct.docs.amd.com\u002Fprojects\u002Famdgpu-docs\u002Fen\u002Flatest\u002Fgpu-partitioning\u002Findex.html",[167],"AMD’s official documentation",[18,8512,8513],{},"Steps:",[114,8515,8516],{},[33,8517,8518],{},[14,8519,8520],{},"Set the compute partitions.",[8451,8522,8525],{"className":8523,"code":8524,"language":8456,"meta":285},[8454],"sudo amd-smi set –gpu all –compute-partition CPX\n",[223,8526,8524],{"__ignoreMap":285},[114,8528,8529],{"start":673},[33,8530,8531],{},[14,8532,8533],{},"Set the memory partitions.",[8451,8535,8538],{"className":8536,"code":8537,"language":8456,"meta":285},[8454],"sudo amd-smi set –memory-partition NPS4\n",[223,8539,8537],{"__ignoreMap":285},[18,8541,8542],{},"Wait a few seconds and, done!",[18,8544,6404],{},[18,8546,8547],{},[283,8548],{"alt":285,"src":8549},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Frocmoverview.webp",[18,8551,8552],{},"Ready to go!",[18,8554,8555,8556,1061],{},"As mentioned in the previous section, to run the benchmarks across multiple containers using vLLM, we first followed the official NGINX load balancing guide (",[163,8557,8438],{"href":8438,"rel":8558},[167],[114,8560,8561],{},[33,8562,8563],{},[14,8564,8446],{},[18,8566,8449],{},[8451,8568,8571],{"className":8569,"code":8570,"language":8456,"meta":285},[8454],"upstream backend {\n    least_conn;\n    server vllm0:8000 max_fails=3 fail_timeout=10000s;\n    .\n    .\n    .\n    server vllm63:8000 max_fails=3 fail_timeout=10000s;\n}\n\nserver {\n    listen 80;\n    location \u002F {\n        proxy_pass http:\u002F\u002Fbackend;\n        proxy_set_header Host $host;\n        proxy_set_header X-Real-IP $remote_addr;\n        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n        proxy_set_header X-Forwarded-Proto $scheme;\n    }\n}\n",[223,8572,8570],{"__ignoreMap":285},[114,8574,8575],{"start":676},[33,8576,8577],{},[14,8578,8465],{},[18,8580,8581],{},"We used the following script to launch 64 containers using incremental device and port numbers:",[8451,8583,8586],{"className":8584,"code":8585,"language":8456,"meta":285},[8454],"#!\u002Fbin\u002Fbash\n\n# Script to run vLLM containers with incremental device, port, and name changes\n# Runs 64 containers with device=\u002Fdev\u002Fdri\u002FrenderD128 increasing in steps of 64\n# Port starting at 8081 and increasing by 1 each time\n# Container name starting at vllm0 and increasing by 1 each time\n\necho \"Starting vLLM containers with incremental configuration…\"\n\nfor i in {0..63}; do\n    # Calculate device number (renderD128, renderD192, renderD256, etc.)\n    device_num=$((128 + i))\n    device_path=\"\u002Fdev\u002Fdri\u002FrenderD${device_num}\"\n\n    # Calculate port (8081, 8082, 8083, etc.)\n    port=$((8081 + i))\n\n    # Calculate container name (vllm0, vllm1, vllm2, etc.)\n    container_name=\"vllm${i}\"\n\n    echo \"Starting container ${container_name} on port ${port} with device ${device_path}…\"\n    docker run -itd \\\n        –ipc host \\\n        -v \u002Fdata:\u002Fdata \\\n        –network vllm_nginx \\\n        -e VLLM_ROCM_USE_AITER=True \\\n        -e HF_HOME=root\u002F.cache\u002Fhuggingface \\\n        -e HF_HUB_CACHE=\u002Froot\u002F.cache\u002Fhuggingface\u002Fhub \\\n        –device=\u002Fdev\u002Fkfd \\\n        –device=${device_path} \\\n        –group-add video \\\n        -p ${port}:8000 \\\n        –name ${container_name} \\\n        rocm\u002Fvllm:latest \\\n        vllm serve \\\n        \u002Fdata\u002F.cache\u002Fhuggingface\u002Fhub\u002Fmodels–amd–Llama-3.1-8B-Instruct-FP8-KV\u002Fsnapshots\u002Ffa42f9a9105c545755fea25cf69f49ac8c8b40e1\u002F \\\n        –num-scheduler-steps 10 \\\n        –kv-cache-dtype fp8 \\\n        –max-model-len 4096\n\n    # Check if container started successfully\n    if [ $? -eq 0 ]; then\n        echo \"✓ Container ${container_name} started successfully\"\n    else\n        echo \"✗ Failed to start container ${container_name}\"\n    fi\n\n    echo \"—\"\ndone\n\necho \"All containers started. Summary:\"\necho \"Containers: vllm0 through vllm63\"\necho \"Ports: 8081 through 8144\"\necho \"Devices: renderD128 through renderD4160 (in steps of 64)\"\necho \"\"\necho \"To check container status: docker ps\"\necho \"To view logs: docker logs <container_name>\"\n",[223,8587,8585],{"__ignoreMap":285},[114,8589,8591],{"start":8590},4,[33,8592,8593],{},[14,8594,8484],{},[18,8596,8597],{},"Same script as in the previous section.",[18,8599,8600],{},"Action view :)",[18,8602,8603],{},[283,8604],{"alt":285,"src":8605},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Flh7-rt.googleusercontent.com-dc3c57fbfc36.gif",[18,8607,8608],{},"Paiton MI300x",[18,8610,8611],{},[283,8612],{"alt":285,"src":8613},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Flh7-rt.googleusercontent.com-d1bd449add79.gif",[18,8615,8616],{},"Stock MI300x",[10,8618,8620],{"id":8619},"benchmark-results",[14,8621,8622],{},"Benchmark Results",[1230,8624,8626],{"id":8625},"no-partitions-8-gpus","No Partitions – 8 GPUs",[298,8628,8629,8670],{},[301,8630,8631],{},[304,8632,8633,8637,8642,8647,8652,8656,8661,8665],{},[307,8634,8635],{},[14,8636,325],{},[307,8638,8639],{},[14,8640,8641],{},"Paiton ∆",[307,8643,8644],{},[14,8645,8646],{},"Stock",[307,8648,8649],{},[14,8650,8651],{},"∆ vs Stock",[307,8653,8654],{},[14,8655,693],{},[307,8657,8658],{},[14,8659,8660],{},"∆ vs H200",[307,8662,8663],{},[14,8664,955],{},[307,8666,8667],{},[14,8668,8669],{},"∆ vs B200",[316,8671,8672,8704,8736,8768,8800,8832,8864,8896],{},[304,8673,8674,8679,8682,8685,8690,8693,8698,8701],{},[321,8675,8676],{},[14,8677,8678],{},"Benchmark duration (s) ↓",[321,8680,8681],{},"4.812",[321,8683,8684],{},"11.029",[321,8686,8687],{},[14,8688,8689],{},"+129.20%",[321,8691,8692],{},"11.84",[321,8694,8695],{},[14,8696,8697],{},"+146.05%",[321,8699,8700],{},"4.59",[321,8702,8703],{},"-4.61%",[304,8705,8706,8711,8714,8717,8722,8725,8730,8733],{},[321,8707,8708],{},[14,8709,8710],{},"Request throughput (req\u002Fs) ↑",[321,8712,8713],{},"213.55",[321,8715,8716],{},"94.308",[321,8718,8719],{},[14,8720,8721],{},"+126.44%",[321,8723,8724],{},"83.22",[321,8726,8727],{},[14,8728,8729],{},"+156.61%",[321,8731,8732],{},"225.99",[321,8734,8735],{},"–5.50%",[304,8737,8738,8743,8746,8749,8754,8757,8762,8765],{},[321,8739,8740],{},[14,8741,8742],{},"Output token throughput (tok\u002Fs) ↑",[321,8744,8745],{},"53851.639",[321,8747,8748],{},"23809.63",[321,8750,8751],{},[14,8752,8753],{},"+126.18%",[321,8755,8756],{},"20940.86",[321,8758,8759],{},[14,8760,8761],{},"+157.16%",[321,8763,8764],{},"56989.26",[321,8766,8767],{},"-5.52%",[304,8769,8770,8775,8778,8781,8786,8789,8794,8797],{},[321,8771,8772],{},[14,8773,8774],{},"Total Token throughput (tok\u002Fs) ↑",[321,8776,8777],{},"101941.667",[321,8779,8780],{},"45047.076",[321,8782,8783],{},[14,8784,8785],{},"+126.30%",[321,8787,8788],{},"39674.51",[321,8790,8791],{},[14,8792,8793],{},"+156.94%",[321,8795,8796],{},"107827.34",[321,8798,8799],{},"-5.46%",[304,8801,8802,8807,8810,8813,8818,8821,8826,8829],{},[321,8803,8804],{},[14,8805,8806],{},"Mean TTFT (ms) ↓",[321,8808,8809],{},"543.799",[321,8811,8812],{},"4252.513",[321,8814,8815],{},[14,8816,8817],{},"+682.47%",[321,8819,8820],{},"3027.49",[321,8822,8823],{},[14,8824,8825],{},"+456.96%",[321,8827,8828],{},"1245.55",[321,8830,8831],{},"+129.05%",[304,8833,8834,8839,8842,8845,8850,8853,8858,8861],{},[321,8835,8836],{},[14,8837,8838],{},"Mean TPOT (ms) ↓",[321,8840,8841],{},"15.075",[321,8843,8844],{},"16.872",[321,8846,8847],{},[14,8848,8849],{},"+11.92%",[321,8851,8852],{},"26.70",[321,8854,8855],{},[14,8856,8857],{},"+77.02%",[321,8859,8860],{},"10.27",[321,8862,8863],{},"-31.87%",[304,8865,8866,8871,8874,8877,8882,8885,8890,8893],{},[321,8867,8868],{},[14,8869,8870],{},"Mean ITL (ms) ↓",[321,8872,8873],{},"15.025",[321,8875,8876],{},"16.509",[321,8878,8879],{},[14,8880,8881],{},"+9.88%",[321,8883,8884],{},"71.11",[321,8886,8887],{},[14,8888,8889],{},"+373.37%",[321,8891,8892],{},"32.62",[321,8894,8895],{},"+117.10%",[304,8897,8898,8903,8906,8909,8914,8917,8922,8925],{},[321,8899,8900],{},[14,8901,8902],{},"Mean E2EL (ms) ↓",[321,8904,8905],{},"4317.43",[321,8907,8908],{},"8403.948",[321,8910,8911],{},[14,8912,8913],{},"+94.65%",[321,8915,8916],{},"9705.94",[321,8918,8919],{},[14,8920,8921],{},"+124.79%",[321,8923,8924],{},"3818.69",[321,8926,8927],{},"-11.51%",[18,8929,8930],{},[283,8931],{"alt":285,"src":8932,"title":6955},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Faverage-throughput-2-2.webp",[1230,8934,8936],{"id":8935},"partitions-64-vgpus","Partitions – 64 vGPUs",[298,8938,8939,8967],{},[301,8940,8941],{},[304,8942,8943,8947,8952,8956,8961,8964],{},[307,8944,8945],{},[14,8946,325],{},[307,8948,8949,8951],{},[14,8950,8641],{},"*",[307,8953,8954],{},[14,8955,8646],{},[307,8957,8958],{},[14,8959,8960],{},"∆vs Stock(Ratio)",[307,8962,8963],{},"**H200",[307,8965,8966],{},"**∆vs H200",[316,8968,8969,8989,9009,9029,9048,9067,9086,9104],{},[304,8970,8971,8976,8979,8982,8985,8987],{},[321,8972,8973],{},[14,8974,8975],{},"Benchmark duration (s)",[321,8977,8978],{},"7.875",[321,8980,8981],{},"17.294",[321,8983,8984],{},"2.20",[321,8986],{},[321,8988],{},[304,8990,8991,8996,8999,9002,9005,9007],{},[321,8992,8993],{},[14,8994,8995],{},"Request throughput (req\u002Fs)",[321,8997,8998],{},"130.234",[321,9000,9001],{},"59.727",[321,9003,9004],{},"2.18",[321,9006],{},[321,9008],{},[304,9010,9011,9016,9019,9022,9025,9027],{},[321,9012,9013],{},[14,9014,9015],{},"Output token throughput (tok\u002Fs)",[321,9017,9018],{},"33339.931",[321,9020,9021],{},"15047.115",[321,9023,9024],{},"2.22",[321,9026],{},[321,9028],{},[304,9030,9031,9036,9039,9042,9044,9046],{},[321,9032,9033],{},[14,9034,9035],{},"Total Token throughput (tok\u002Fs)",[321,9037,9038],{},"62667.914",[321,9040,9041],{},"28497.62",[321,9043,8984],{},[321,9045],{},[321,9047],{},[304,9049,9050,9054,9057,9060,9063,9065],{},[321,9051,9052],{},[14,9053,414],{},[321,9055,9056],{},"1082.885",[321,9058,9059],{},"6255.879",[321,9061,9062],{},"5.78",[321,9064],{},[321,9066],{},[304,9068,9069,9073,9076,9079,9082,9084],{},[321,9070,9071],{},[14,9072,441],{},[321,9074,9075],{},"20.99",[321,9077,9078],{},"31.289",[321,9080,9081],{},"1.49",[321,9083],{},[321,9085],{},[304,9087,9088,9092,9094,9097,9100,9102],{},[321,9089,9090],{},[14,9091,469],{},[321,9093,9075],{},[321,9095,9096],{},"31.13",[321,9098,9099],{},"1.48",[321,9101],{},[321,9103],{},[304,9105,9106,9110,9113,9116,9119,9121],{},[321,9107,9108],{},[14,9109,497],{},[321,9111,9112],{},"6435.477",[321,9114,9115],{},"14067.724",[321,9117,9118],{},"2.19",[321,9120],{},[321,9122],{},[18,9124,9125],{},"**Note: We are working on improving these numbers even more.",[18,9127,9128,9129,9138],{},"**Note2: Not possible with Nvidia, or at least very difficult (",[630,9130,9131],{},[163,9132,9135],{"href":9133,"rel":9134},"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\u002Fissues\u002F6551#issuecomment-2237624342",[167],[630,9136,9137],{},"complicated",")*",[18,9140,9141],{},[283,9142],{"alt":285,"src":9143,"title":6955},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Faverage-throughput-2-2-1.webp",[18,9145,9146],{},"Now let’s look at this from an ROI-driven perspective.",[18,9148,9149],{},"If we haven’t impressed you so far, we’re pretty sure this will. We use the MI300X server as a reference\u002Fbaseline to compare its cost factor and throughput against the H200 and B200.",[298,9151,9152,9172],{},[301,9153,9154],{},[304,9155,9156,9161,9166,9169],{},[307,9157,9158],{},[14,9159,9160],{},"Architecture",[307,9162,9163],{},[14,9164,9165],{},"Cost Factor vs Paiton",[307,9167,9168],{},"**Throughput Cost-Eff ***",[307,9170,9171],{},"**Latency Cost-Eff",[316,9173,9174,9188,9203,9218],{},[304,9175,9176,9181,9184,9186],{},[321,9177,9178],{},[14,9179,9180],{},"MI300X+Paiton",[321,9182,9183],{},"Ref",[321,9185,9183],{},[321,9187,9183],{},[304,9189,9190,9194,9197,9200],{},[321,9191,9192],{},[14,9193,8616],{},[321,9195,9196],{},"1x",[321,9198,9199],{},"+126.31%",[321,9201,9202],{},"+94.57%",[304,9204,9205,9209,9212,9215],{},[321,9206,9207],{},[14,9208,693],{},[321,9210,9211],{},"1.375x",[321,9213,9214],{},"+253.30%",[321,9216,9217],{},"+209.07%",[304,9219,9220,9224,9227,9230],{},[321,9221,9222],{},[14,9223,955],{},[321,9225,9226],{},"2x",[321,9228,9229],{},"+89.18%",[321,9231,8857],{},[5350,9233,9234,9257],{},[18,9235,9236,999,9239,999,9244,9247,9248,999,9251,9256],{},[630,9237,9238],{},"*Throughput Cost-Efficiency: = %",[630,9240,9241],{},[14,9242,9243],{},"more",[630,9245,9246],{},"total‑token","** ",[630,9249,9250],{},"throughput",[630,9252,9253],{},[14,9254,9255],{},"per dollar"," *vs each platform.",[18,9258,9259,9260,999,9265,999,9268,9247,9273],{},"**Latency Cost-Efficiency: = %* ",[630,9261,9262],{},[14,9263,9264],{},"better",[630,9266,9267],{},"end‑to‑end",[630,9269,9270,9255],{},[14,9271,9272],{},"latency",[630,9274,9275],{},"vs each platform.",[18,9277,9278],{},[14,9279,9280],{},"What this tells you",[30,9282,9283,9293,9306],{},[33,9284,9285,9288,9289,9292],{},[14,9286,9287],{},"Paiton delivers 2.5 × the throughput per $"," over an H200 and ",[14,9290,9291],{},"+126 %"," over stock.",[33,9294,9295,9298,9299,266,9302,9305],{},[14,9296,9297],{},"Latency per $"," is ",[14,9300,9301],{},"3.1 × better than the H200",[14,9303,9304],{},"+94 % over stock",", solid ROI on every millisecond shaved.",[33,9307,9308,9309,9311,9312,9315],{},"The ",[14,9310,955],{}," gap is real, but remember it costs ",[14,9313,9314],{},"twice"," as much, Paiton still wins on cost‑efficiency across the board.",[10,9317,9319],{"id":9318},"cost-per-million-tokens",[14,9320,9321],{},"Cost per Million tokens",[18,9323,9324],{},"If we use available renting prices for the different systems, we could calculate the relative cost per 1M tokens:",[298,9326,9327,9360],{},[301,9328,9329],{},[304,9330,9331,9335,9340,9345,9350,9355],{},[307,9332,9333],{},[14,9334,9160],{},[307,9336,9337],{},[14,9338,9339],{},"Throughput (tok\u002Fs)",[307,9341,9342],{},[14,9343,9344],{},"GPU Count",[307,9346,9347],{},[14,9348,9349],{},"Approx. hourly cost",[307,9351,9352],{},[14,9353,9354],{},"Inference Cost \u002F 1M Tokens",[307,9356,9357],{},[14,9358,9359],{},"Relative Cost",[316,9361,9362,9382,9400,9419],{},[304,9363,9364,9368,9370,9373,9376,9379],{},[321,9365,9366],{},[14,9367,9180],{},[321,9369,8777],{},[321,9371,9372],{},"~8",[321,9374,9375],{},"$20.50",[321,9377,9378],{},"$0.06",[321,9380,9381],{},"REF",[304,9383,9384,9388,9390,9392,9394,9397],{},[321,9385,9386],{},[14,9387,8616],{},[321,9389,8780],{},[321,9391,9372],{},[321,9393,9375],{},[321,9395,9396],{},"$0.13",[321,9398,9399],{},"2.26× ↑",[304,9401,9402,9406,9408,9410,9413,9416],{},[321,9403,9404],{},[14,9405,693],{},[321,9407,8788],{},[321,9409,9372],{},[321,9411,9412],{},"$28.20",[321,9414,9415],{},"$0.20",[321,9417,9418],{},"3.54× ↑",[304,9420,9421,9425,9427,9429,9432,9434],{},[321,9422,9423],{},[14,9424,955],{},[321,9426,8796],{},[321,9428,9372],{},[321,9430,9431],{},"$48.60",[321,9433,9396],{},[321,9435,9436],{},"2.24× ↑",[18,9438,9439],{},[283,9440],{"alt":285,"src":9441},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fimage-2-1.jpg",[18,9443,9444],{},[14,9445,9446],{},"Insights:",[30,9448,9449,9454,9459],{},[33,9450,9451,9453],{},[14,9452,68],{}," delivers 2.26× cost savings compared to unoptimized MI300X.",[33,9455,9456,9458],{},[14,9457,693],{}," costs 3.54× more than MI300X+Paiton per 1M tokens.",[33,9460,9461,9463,9464,9467],{},[14,9462,955],{}," is the most expensive, costing over ",[14,9465,9466],{},"2.24× more"," than the optimized MI300X setup.",[10,9469,9471],{"id":9470},"big-win-for-amd",[14,9472,9473],{},"Big win for AMD",[18,9475,9476],{},"Trying to figure out how MIG worked on Nvidia with vLLM was like trying to find a perfect gift for your spouse, it was exhausting. Eventually we ran into a NCCL error which seemed unsolvable and that was the last straw.",[18,9478,9479],{},"While MIG allows virtual partitioning on supported NVIDIA GPUs, as previously mentioned we encountered significant limitations when attempting to use it in conjunction with vLLM for data-parallel workloads. Specifically, vLLM was unable to properly leverage MIG slices for distributed inference.",[18,9481,9482],{},"In contrast, AMD’s architecture enabled straightforward partitioning and containerized deployment of vLLM instances without any issues. This streamlined setup, along with ROCm’s compatibility, made AMD far better suited for true multi-tenancy out of the box.",[18,9484,9485,9486,9489],{},"This represents a ",[14,9487,9488],{},"major win for AMD",", particularly for enterprises aiming to deploy isolated inference workloads across shared hardware without too much friction or compromise.",[5350,9491,9492],{},[5350,9493,9494],{},[5350,9495,9496,9501,9504],{},[18,9497,9498],{},[14,9499,9500],{},"Having methodically outpaced Intel in performance, AMD is now strategically poised to challenge NVIDIA’s leadership, an evolution we’re proud to drive.",[18,9502,9503],{},"***Kian Mohadjerin",[18,9505,9506],{},"Head of AI, Eliovp BV*",[10,9508,9510],{"id":9509},"key-results",[14,9511,9512],{},"Key Results",[30,9514,9515,9521,9527,9533],{},[33,9516,9517,9520],{},[14,9518,9519],{},"Throughput scaling"," was near-linear up to 64 partitions, thanks to Paiton’s minimized memory overhead and fast kernel dispatch.",[33,9522,9523,9526],{},[14,9524,9525],{},"Latency"," remained stable across parallel sessions, demonstrating the strength of Paiton’s per-GPU scheduling and shared memory optimizations.",[33,9528,9529,9532],{},[14,9530,9531],{},"Memory usage per partition"," was significantly lower compared to standard vLLM or runtimes, enabling high-density deployment.",[33,9534,9535,9538],{},[14,9536,9537],{},"Cost per Million Tokens"," was reduced by over 2× compared to high-end systems like the B200, showcasing Paiton’s ability to deliver industry-leading efficiency even on more affordable AMD hardware.",[10,9540,9541],{"id":640},[14,9542,643],{},[18,9544,9545],{},"This experiment highlights Paiton’s ability to unlock the full potential of modern hardware like the MI300X through advanced packaging and optimization techniques. Running Llama 3.1 8B FP8 across 64 GPU partitions showcases how inference workloads can be massively parallelized without sacrificing too much performance or usability.",[18,9547,9548],{},"Imagine the potential of Paiton paired with AMD’s upcoming MI355X. With even more memory bandwidth, compute, and architectural improvements on the horizon, the synergy between next-gen hardware and the Paiton runtime could redefine the state of high-performance AI serving.",[18,9550,9551],{},"Stay tuned for future updates as we expand Paiton’s capabilities.",[18,9553,9554],{},"Don’t believe our results? Neither did we, so test Paiton for yourself and request an evaluation model.",[10,9556,9558],{"id":9557},"pricing",[14,9559,9560],{},"Pricing",[18,9562,9563],{},"If you’re curious about pricing with Paiton, our formula is quite simple:",[298,9565,9566],{},[301,9567,9568],{},[304,9569,9570],{},[307,9571,9572],{},"50% of x% costs saved per 1M tokens",[18,9574,9575],{},"The cost saved is measured by looking at the customer’s current throughput compared to the throughput using Paiton.",[18,9577,9578,7218],{},[163,9579,7217],{"href":6491,"rel":9580},[167],[10,9582,9583],{"id":4556},[14,9584,4559],{},[114,9586,9587,9592,9597,9603,9608,9613,9619,9625,9631,9638],{},[33,9588,9589],{},[163,9590,7231],{"href":7229,"rel":9591},[167],[33,9593,9594],{},[163,9595,7238],{"href":7236,"rel":9596},[167],[33,9598,9599],{},[163,9600,9602],{"href":8200,"rel":9601},[167],"Paiton FP8 beats Nvidia’s H200 on AMD’s MI300X",[33,9604,9605],{},[163,9606,7245],{"href":7243,"rel":9607},[167],[33,9609,9610],{},[163,9611,7252],{"href":7250,"rel":9612},[167],[33,9614,9615],{},[163,9616,9618],{"href":8349,"rel":9617},[167],"Hugging Face AMD Llama-3.1-8B-Instruct-FP8-KV",[33,9620,9621],{},[163,9622,9624],{"href":8438,"rel":9623},[167],"vLLM Nginx Deployment",[33,9626,9627],{},[163,9628,9630],{"href":8508,"rel":9629},[167],"AMD GPU Partitioning Documentation",[33,9632,9633],{},[163,9634,9637],{"href":9635,"rel":9636},"https:\u002F\u002Frocm.blogs.amd.com\u002Fsoftware-tools-optimization\u002Fcompute-memory-modes\u002FREADME.html",[167],"ROCm Compute Memory Modes",[33,9639,9640],{},[163,9641,9644],{"href":9642,"rel":9643},"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\u002Fissues\u002F6551",[167],"vLLM GitHub Issue #6551",{"title":285,"searchDepth":673,"depth":673,"links":9646},[9647,9648,9649,9650,9651,9652,9653,9654,9655,9656],{"id":8280,"depth":676,"text":8283},{"id":8426,"depth":676,"text":8429},{"id":8496,"depth":676,"text":8499},{"id":8619,"depth":676,"text":8622},{"id":9318,"depth":676,"text":9321},{"id":9470,"depth":676,"text":9473},{"id":9509,"depth":676,"text":9512},{"id":640,"depth":676,"text":643},{"id":9557,"depth":676,"text":9560},{"id":4556,"depth":676,"text":4559},[688,689,68,1149,9658,690,955,693,694,965,966,68,696],"AI","2025-07-31T13:32:57","At ElioVP, we’re all about pushing AI inference past the limits, and packaging every squeeze of performance into a plug‑and‑play runtime.  Remember our last blog, where Paiton’s FP8 pipeline on AMD’s MI300X completely outclassed NVIDIA’s H200? Well, buckle up, because we’ve gone back to the drawing board. This time, we’re loading Llama-3.1-8B-Instruct-FP8-KV, the leaner, meaner ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fimage-2-1.webp",{},"https:\u002F\u002Feliovp.com\u002Fmi300x-fp8-data%e2%80%91parallel-benchmarks-8-64-gpus-h200-left-behind-b200-within-reach\u002F","\u002Fblog\u002Fmi300x-fp8-datapercente2percent80percent91parallel-benchmarks-8-64-gpus-h200-left-behind-b200-within-reach",{"title":8189,"description":9660},"mi300x-fp8-data%e2%80%91parallel-benchmarks-8-64-gpus-h200-left-behind-b200-within-reach","blog\u002Fmi300x-fp8-data%e2%80%91parallel-benchmarks-8-64-gpus-h200-left-behind-b200-within-reach","0C-aaj579FFi-Wj_Jbqje930NbNY8UccYoiX4QjkWEY",{"id":9670,"title":9671,"body":9672,"categories":9877,"date":9878,"description":9879,"extension":699,"image":9880,"meta":9881,"navigation":702,"originalUrl":9882,"path":9883,"seo":9884,"slug":9885,"stem":9886,"__hash__":9887},"blog\u002Fblog\u002Fapplicable-ai-for-businesses.md","Applicable AI for Businesses",{"type":7,"value":9673,"toc":9868},[9674,9677,9680,9683,9689,9692,9695,9698,9701,9707,9710,9713,9739,9745,9748,9765,9768,9774,9781,9787,9790,9810,9816,9819,9822,9828,9831,9857,9860],[18,9675,9676],{},"Here at Eliovp, we continuously innovate when it comes to building practical solutions. If there’s one core strength, it’s our team’s ability to think outside the box.",[18,9678,9679],{},"One key area of focus for us is developing applicable AI solutions, everyday usable AI implementations tailored specifically for our clients’ needs.",[18,9681,9682],{},"In this blog, we’ll explore our core concepts, innovative ideas, and efficient workflows for creating these powerful AI-driven solutions.",[10,9684,9686],{"id":9685},"what-is-applicable-ai",[14,9687,9688],{},"What is Applicable AI?",[18,9690,9691],{},"Today, countless applications leverage AI, from basic tools like chatbots and image generation to advanced agents assisting in managing emails or enhancing developer productivity. At Eliovp, we focus on building AI solutions that significantly enhance business processes and customer experiences.",[18,9693,9694],{},"We design tailored solutions, integrating both AI and traditional machine learning techniques to drive meaningful improvements, whether automating customer support based on historical data or ensuring consistent product quality.",[18,9696,9697],{},"Crucially, we don’t merely deliver off-the-shelf solutions. Instead, we collaborate closely with your team, actively engaging in identifying opportunities, defining innovative solutions, and integrating them seamlessly into your existing workflows.",[18,9699,9700],{},"The ideal applicable AI operates quietly in the background, subtly enhancing productivity without disrupting daily operations, ensuring a seamless integration and enhanced user experience.",[10,9702,9704],{"id":9703},"our-focus-on-data-sovereignty-and-local-infrastructure",[14,9705,9706],{},"Our Focus on Data Sovereignty and Local Infrastructure",[18,9708,9709],{},"At Eliovp, we strongly believe in empowering businesses to maintain control over their data. All our AI solutions run entirely within our clients’ private networks, eliminating external dependencies and risks associated with data exposure.",[18,9711,9712],{},"We offer complete end-to-end solutions:",[30,9714,9715,9721,9727,9733],{},[33,9716,9717,9720],{},[14,9718,9719],{},"In-house Model Training:"," Custom-trained models tailored to your specific datasets.",[33,9722,9723,9726],{},[14,9724,9725],{},"Localized Vector Databases and MCP (Model Context Protocol):"," Secure, private connections to your internal services.",[33,9728,9729,9732],{},[14,9730,9731],{},"On-Site Inferencing:"," Optimized, high-performance models leveraging our proprietary optimization suite, Paiton, on AMD hardware.",[33,9734,9735,9738],{},[14,9736,9737],{},"Expert Support Team:"," A dedicated, experienced team to manage, deploy, and support your AI infrastructure.",[10,9740,9742],{"id":9741},"real-world-example-ai-powered-damage-detection-for-logistics",[14,9743,9744],{},"Real-World Example: AI-Powered Damage Detection for Logistics",[18,9746,9747],{},"One standout implementation of our applicable AI is an advanced damage detection system designed for freight and logistics. Our innovative AI agent automatically inspects trucks as they pass through a scanning arch, capturing images and instantly identifying:",[30,9749,9750,9753,9756,9759,9762],{},[33,9751,9752],{},"Rust",[33,9754,9755],{},"Dents",[33,9757,9758],{},"Glue residue",[33,9760,9761],{},"Missing ADR labels",[33,9763,9764],{},"And much more …",[18,9766,9767],{},"This system significantly streamlines operations, minimizes human error, and instantly generates comprehensive damage reports.",[10,9769,9771],{"id":9770},"watch-our-solution-in-action",[14,9772,9773],{},"Watch our solution in action:",[18,9775,9776],{},[163,9777,9780],{"href":9778,"rel":9779},"https:\u002F\u002Fwww.youtube.com\u002Fembed\u002FtnN0oXSBvgs?feature=oembed",[167],"Eliovp Agentic AI auto damage report container",[10,9782,9784],{"id":9783},"how-we-work",[14,9785,9786],{},"How We Work",[18,9788,9789],{},"At Eliovp, transparency and effective communication guide every project:",[30,9791,9792,9798,9804],{},[33,9793,9794,9797],{},[14,9795,9796],{},"Collaborative Approach:"," Close collaboration ensures solutions match your unique needs.",[33,9799,9800,9803],{},[14,9801,9802],{},"Continuous Feedback Loop:"," Regular, early deployments collect valuable insights, refining the solution iteratively to deliver maximum value.",[33,9805,9806,9809],{},[14,9807,9808],{},"Customized Tools:"," Tailored tools that seamlessly integrate into existing operational flows, minimizing disruption and maximizing productivity.",[10,9811,9813],{"id":9812},"results-of-our-approach",[14,9814,9815],{},"Results of Our Approach",[18,9817,9818],{},"Our methodology delivers immediate, tangible improvements shortly after project initiation. Through incremental enhancements and regular feedback, we consistently achieve higher client engagement and satisfaction.",[18,9820,9821],{},"By optimizing workflows, we enhance productivity, reduce errors, and elevate the quality of outcomes, transforming everyday operations into streamlined, efficient processes.",[10,9823,9825],{"id":9824},"why-choose-eliovp",[14,9826,9827],{},"Why Choose Eliovp?",[18,9829,9830],{},"When partnering with Eliovp, you benefit from:",[30,9832,9833,9839,9845,9851],{},[33,9834,9835,9838],{},[14,9836,9837],{},"Expertise:"," A dedicated team, each member possessing over 10 years of software development experience, including specialized domain knowledge (e.g., biomedical engineering).",[33,9840,9841,9844],{},[14,9842,9843],{},"Tailored Solutions:"," Custom-developed AI designed specifically around your operational needs.",[33,9846,9847,9850],{},[14,9848,9849],{},"Data Integrity:"," Comprehensive data sovereignty, ensuring your data never leaves your network.",[33,9852,9853,9856],{},[14,9854,9855],{},"Independent Infrastructure:"," Our self-sufficient approach leverages advanced AMD hardware without reliance on third-party APIs or external providers.",[18,9858,9859],{},"Ready to see how Eliovp’s applicable AI can transform your business operations?",[18,9861,9862,9867],{},[163,9863,9866],{"href":9864,"rel":9865},"https:\u002F\u002Fai.eliovp.com\u002F",[167],"Reach out today"," and meet our innovative team!",{"title":285,"searchDepth":673,"depth":673,"links":9869},[9870,9871,9872,9873,9874,9875,9876],{"id":9685,"depth":676,"text":9688},{"id":9703,"depth":676,"text":9706},{"id":9741,"depth":676,"text":9744},{"id":9770,"depth":676,"text":9773},{"id":9783,"depth":676,"text":9786},{"id":9812,"depth":676,"text":9815},{"id":9824,"depth":676,"text":9827},[688,689,3971,3972,8171,8172,8173,8174,3977,3979,969,8175,8176],"2025-07-09T21:35:30","Here at Eliovp, we continuously innovate when it comes to building practical solutions. If there’s one core strength, it’s our team’s ability to think outside the box. One key area of focus for us is developing applicable AI solutions, everyday usable AI implementations tailored specifically for our clients’ needs. In this blog, we’ll explore our ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fscherm_afbeelding-2025-07-09-om-23.30.17.webp",{},"https:\u002F\u002Feliovp.com\u002Fapplicable-ai-for-businesses\u002F","\u002Fblog\u002Fapplicable-ai-for-businesses",{"title":9671,"description":9879},"applicable-ai-for-businesses","blog\u002Fapplicable-ai-for-businesses","1v8RCVaLtUYYwPHobdmgEvwHSlxnBWZs4AMbF66Qsmw",{"id":9889,"title":9890,"body":9891,"categories":10009,"date":10010,"description":10011,"extension":699,"image":285,"meta":10012,"navigation":702,"originalUrl":10013,"path":10014,"seo":10015,"slug":10016,"stem":10017,"__hash__":10018},"blog\u002Fblog\u002Fintroducing-paitons-free-evaluation-models.md","Introducing Paiton’s Free Evaluation Models",{"type":7,"value":9892,"toc":10007},[9893,9898,9901,9908,9913,9916,9919,9930,9936,9941,9944,9964,9967,9972,9992,9995,10004],[18,9894,9895],{},[14,9896,9897],{},"Introduction",[18,9899,9900],{},"AI is rapidly transforming every industry, but running large models efficiently remains a major technical and financial challenge. At ElioVP, we specialize in optimizing for AMD accelerators, helping organizations unlock the full potential of their hardware.",[18,9902,9903,9904,9907],{},"Today, we’re excited to announce a new offering: ",[14,9905,9906],{},"free evaluation models"," that let you test our cutting-edge optimizations on your own workloads before committing. Our evaluation packages give you a firsthand look at the performance gains you can expect with Paiton.",[18,9909,9910],{},[14,9911,9912],{},"What Is Paiton?",[18,9914,9915],{},"Paiton is an AI model optimization framework built specifically for AMD architectures. We take your existing models like LLaMA, Stable Diffusion, CLIP, and more, then compile them into custom, highly-performant shared libraries using our own kernels and optimization passes.",[18,9917,9918],{},"This results in:",[30,9920,9921,9924,9927],{},[33,9922,9923],{},"Faster inference times",[33,9925,9926],{},"Reduced power consumption",[33,9928,9929],{},"Better hardware utilization",[18,9931,9932,9933,9935],{},"Our solution works with ",[14,9934,696],{},", making integration seamless.",[18,9937,9938],{},[14,9939,9940],{},"Introducing Evaluation Models",[18,9942,9943],{},"We’re now offering pre-compiled evaluation models so potential customers can experience Paiton’s performance boosts firsthand. These include:",[30,9945,9946,9952,9958],{},[33,9947,9948,9951],{},[14,9949,9950],{},"Text models"," like meta-llama\u002FMeta-Llama-3-8B",[33,9953,9954,9957],{},[14,9955,9956],{},"Vision models"," like CLIP and ResNet",[33,9959,9960,9963],{},[14,9961,9962],{},"Image generation models"," like Stable Diffusion XL",[18,9965,9966],{},"Evaluation models are available for various AMD GPUs, including MI300X, MI250 and even MI210. We provide simple Python interfaces and example scripts to get you started.",[18,9968,9969],{},[14,9970,9971],{},"Why Try a Paiton Evaluation Model?",[30,9973,9974,9980,9986],{},[33,9975,9976,9979],{},[14,9977,9978],{},"No commitment",": Test before you buy",[33,9981,9982,9985],{},[14,9983,9984],{},"Real-world benchmarking",": Use your own prompts and data",[33,9987,9988,9991],{},[14,9989,9990],{},"Side-by-side comparison",": Measure against your current setup",[18,9993,9994],{},"If you like what you see, we can work together to compile and optimize your custom models or offer you the existing models, optimized with Paiton, permanently.",[18,9996,9997,9998,10003],{},"Interested in trying out an evaluation model? Just reach out to us ",[163,9999,10002],{"href":10000,"rel":10001},"https:\u002F\u002Feliovp.com\u002Fcontact\u002F",[167],"here",", and we’ll help you select the right package for your hardware and workload.",[18,10005,10006],{},"AI is only as good as the infrastructure it runs on. With Paiton, you don’t need to overhaul your stack to achieve state-of-the-art performance on AMD GPUs. Our new evaluation offering gives you a low-risk way to explore what’s possible.",{"title":285,"searchDepth":673,"depth":673,"links":10008},[],[688,689,68],"2025-07-07T11:26:13","Introduction AI is rapidly transforming every industry, but running large models efficiently remains a major technical and financial challenge. At ElioVP, we specialize in optimizing for AMD accelerators, helping organizations unlock the full potential of their hardware. Today, we’re excited to announce a new offering: free evaluation models that let you test our cutting-edge optimizations ...",{},"https:\u002F\u002Feliovp.com\u002Fintroducing-paitons-free-evaluation-models\u002F","\u002Fblog\u002Fintroducing-paitons-free-evaluation-models",{"title":9890,"description":10011},"introducing-paitons-free-evaluation-models","blog\u002Fintroducing-paitons-free-evaluation-models","oqFEHCvzWpvoMzlaGMS_C_32BEQLBTz72IFW_NdYbT4",{"id":10020,"title":10021,"body":10022,"categories":10467,"date":10472,"description":10473,"extension":699,"image":10474,"meta":10475,"navigation":702,"originalUrl":10476,"path":10477,"seo":10478,"slug":10479,"stem":10480,"__hash__":10481},"blog\u002Fblog\u002Fpaiton-dramatically-faster-startup-and-performance-for-llama-3-1-405b.md","Paiton: Dramatically Faster Startup and Performance for Llama-3.1-405B",{"type":7,"value":10023,"toc":10456},[10024,10030,10044,10050,10053,10060,10064,10067,10096,10102,10108,10111,10131,10134,10140,10143,10163,10169,10273,10283,10289,10292,10312,10318,10412,10415,10421,10424,10430,10433,10453],[18,10025,10026,10027,10029],{},"With ",[14,10028,68],{},", we’re not merely pursuing peak inference speeds, we’re fundamentally reshaping the entire lifecycle of large language model (LLM) deployment. Our latest endeavor pairs AMD’s cutting-edge MI300X GPUs with the colossal Llama-3.1-405B-Instruct-FP8-KV model, achieving groundbreaking milestones:",[114,10031,10032,10038],{},[33,10033,10034,10037],{},[14,10035,10036],{},"Instant-On Startup",": Significantly reducing cold-start delays for massive LLM deployments.",[33,10039,10040,10043],{},[14,10041,10042],{},"Advanced Tensor Parallelism (TP)",": Dramatically enhancing inference throughput and slashing latency through sophisticated TP optimizations.",[10,10045,10047],{"id":10046},"visual-demonstration-startup-speed-showcase",[14,10048,10049],{},"Visual Demonstration: Startup Speed Showcase",[18,10051,10052],{},"We’re excited to share a visual demonstration of Paiton’s revolutionary startup performance. Watch below how Paiton transforms a typically sluggish startup process into an agile, responsive experience. After startup we showcase the first inference run as well to show you the whole picture, from startup to first request.",[18,10054,10055],{},[163,10056,10059],{"href":10057,"rel":10058},"https:\u002F\u002Fwww.youtube.com\u002Fembed\u002FtTgMARD3i_0?feature=oembed",[167],"Paiton on Llama 405B vs Aiter",[10,10061,10062],{"id":8280},[14,10063,8283],{},[18,10065,10066],{},"Ensuring transparency and reproducibility, our benchmarking approach includes detailed specifications:",[30,10068,10069,10074,10079,10085,10090],{},[33,10070,10071,10073],{},[14,10072,8309],{},": vLLM v0.9.0 with amd\u002FLlama-3.1-405B-Instruct-FP8-KV",[33,10075,10076,10078],{},[14,10077,962],{},": 8 × AMD MI300X GPUs (192 GB total HBM3 memory)",[33,10080,10081,10084],{},[14,10082,10083],{},"Software Stack",": ROCm 6.3.1 on Ubuntu 22.04 (notably still utilizing an older driver stack)",[33,10086,10087,10089],{},[14,10088,6775],{},": 32, representative of realistic, interactive AI workloads",[33,10091,10092,10095],{},[14,10093,10094],{},"Measurements",": Comprehensive averages over 10 runs, meticulously covering startup times, cold-start and steady-state TTFT, and end-to-end latency metrics",[18,10097,10098,10101],{},[14,10099,10100],{},"Key Highlight",": Paiton consistently delivers stable and reliable performance, eliminating variability common in other inference solutions.",[10,10103,10105],{"id":10104},"llm-startup-the-critical-bottleneck",[14,10106,10107],{},"LLM Startup: The Critical Bottleneck",[18,10109,10110],{},"Deploying large-scale LLMs like Llama-3.1-405B-Instruct-FP8-KV is a significant engineering challenge. Startup delays commonly arise from:",[30,10112,10113,10119,10125],{},[33,10114,10115,10118],{},[14,10116,10117],{},"Model Weight Loading",": Transferring massive sets of parameters from storage to GPU memory.",[33,10120,10121,10124],{},[14,10122,10123],{},"Graph Compilation",": Transforming high-level model definitions into optimized execution plans.",[33,10126,10127,10130],{},[14,10128,10129],{},"Initial Warm-up",": Performing preliminary inferences to reach peak operational efficiency.",[18,10132,10133],{},"These delays directly impact scalability, developer productivity, operational cost-efficiency, and end-user experience.",[10,10135,10137],{"id":10136},"instant-on-startup-paitons-strategic-advantage",[14,10138,10139],{},"Instant-On Startup: Paiton’s Strategic Advantage",[18,10141,10142],{},"Paiton uniquely harnesses AMD’s GPU architecture combined with proprietary optimizations to substantially reduce startup times:",[30,10144,10145,10151,10157],{},[33,10146,10147,10150],{},[14,10148,10149],{},"Hyper-Optimized Weight Loading",": Leveraging AMD’s ultra-fast HBM3 memory with parallel data transfers.",[33,10152,10153,10156],{},[14,10154,10155],{},"Accelerated Graph Compilation",": Custom routines that entirely eliminate compilation wait times.",[33,10158,10159,10162],{},[14,10160,10161],{},"Intelligent Warm-up",": Advanced priming strategies guaranteeing immediate and sustained responsiveness.",[10,10164,10166],{"id":10165},"startup-comparison-llama-31-405b-instruct-fp8-kv",[14,10167,10168],{},"Startup Comparison: Llama-3.1-405B-Instruct-FP8-KV",[298,10170,10171,10195],{},[301,10172,10173],{},[304,10174,10175,10180,10185,10190],{},[307,10176,10177],{},[14,10178,10179],{},"Stage",[307,10181,10182],{},[14,10183,10184],{},"Standard vLLM (sec)",[307,10186,10187],{},[14,10188,10189],{},"AMD + Paiton (sec)",[307,10191,10192],{},[14,10193,10194],{},"Improvement",[316,10196,10197,10211,10225,10238,10251],{},[304,10198,10199,10202,10205,10208],{},[321,10200,10201],{},"Model Weight Load",[321,10203,10204],{},"71.24",[321,10206,10207],{},"64.28",[321,10209,10210],{},"9.7% Faster",[304,10212,10213,10216,10219,10222],{},[321,10214,10215],{},"Memory Profiling",[321,10217,10218],{},"69.97",[321,10220,10221],{},"38.63",[321,10223,10224],{},"44.8% Faster",[304,10226,10227,10229,10232,10235],{},[321,10228,10123],{},[321,10230,10231],{},"27",[321,10233,10234],{},"0.00",[321,10236,10237],{},"100% Faster",[304,10239,10240,10242,10245,10248],{},[321,10241,10129],{},[321,10243,10244],{},"98.37",[321,10246,10247],{},"40.59",[321,10249,10250],{},"58.7% Faster",[304,10252,10253,10258,10263,10268],{},[321,10254,10255],{},[14,10256,10257],{},"Total Startup",[321,10259,10260],{},[14,10261,10262],{},"266.58",[321,10264,10265],{},[14,10266,10267],{},"143.50",[321,10269,10270],{},[14,10271,10272],{},"46.2% Faster",[18,10274,10275,10278,10279,10282],{},[14,10276,10277],{},"Real-world Impact",": Deploy a fully operational 405B parameter LLM in less than ",[14,10280,10281],{},"2.4 minutes",", significantly outperforming traditional deployment methods.",[10,10284,10286],{"id":10285},"deep-dive-into-tensor-parallelism-the-paiton-edge",[14,10287,10288],{},"Deep Dive into Tensor Parallelism: The Paiton Edge",[18,10290,10291],{},"Tensor Parallelism is vital for harnessing the power of multi-GPU configurations. At Paiton, we’ve invested extensive effort in deeply optimized kernel development and an enhanced communication layer specifically tailored for AMD’s MI300X GPUs. Our proprietary approach to TP provides unparalleled performance:",[30,10293,10294,10300,10306],{},[33,10295,10296,10299],{},[14,10297,10298],{},"Highly Optimized Kernels",": Precision-crafted to maximize GPU compute efficiency and reduce intra-node latency.",[33,10301,10302,10305],{},[14,10303,10304],{},"Advanced Communication Layer",": Significantly streamlined inter-GPU communication, drastically reducing overhead.",[33,10307,10308,10311],{},[14,10309,10310],{},"Scalable Architecture",": Consistent, predictable scaling even in complex multi-GPU deployments.",[10,10313,10315],{"id":10314},"sustained-performance-gains-with-paitons-tp",[14,10316,10317],{},"Sustained Performance Gains with Paiton’s TP",[298,10319,10320,10338],{},[301,10321,10322],{},[304,10323,10324,10328,10333],{},[307,10325,10326],{},[14,10327,325],{},[307,10329,10330],{},[14,10331,10332],{},"Paiton Avg",[307,10334,10335],{},[14,10336,10337],{},"Improvement vs Aiter",[316,10339,10340,10352,10364,10376,10388,10400],{},[304,10341,10342,10344,10347],{},[321,10343,373],{},[321,10345,10346],{},"2.35",[321,10348,10349],{},[14,10350,10351],{},"+20.5% Faster",[304,10353,10354,10356,10359],{},[321,10355,387],{},[321,10357,10358],{},"462.52",[321,10360,10361],{},[14,10362,10363],{},"+13.4% Faster",[304,10365,10366,10368,10371],{},[321,10367,401],{},[321,10369,10370],{},"1009.91",[321,10372,10373],{},[14,10374,10375],{},"+17.3% Faster",[304,10377,10378,10380,10383],{},[321,10379,414],{},[321,10381,10382],{},"2581.94",[321,10384,10385],{},[14,10386,10387],{},"42.8% Faster",[304,10389,10390,10392,10395],{},[321,10391,497],{},[321,10393,10394],{},"11051.53",[321,10396,10397],{},[14,10398,10399],{},"24.3% Faster",[304,10401,10402,10404,10407],{},[321,10403,469],{},[321,10405,10406],{},"43.28",[321,10408,10409],{},[14,10410,10411],{},"10.6% Faster",[18,10413,10414],{},"Paiton’s combination of rapid deployment capabilities and superior runtime performance uniquely positions AMD-based infrastructure for enterprise-grade AI deployments.",[10,10416,10418],{"id":10417},"shaping-the-future-of-llm-deployment",[14,10419,10420],{},"Shaping the Future of LLM Deployment",[18,10422,10423],{},"Our significant breakthroughs with Llama-3.1-405B-Instruct-FP8-KV represent a transformative shift, providing unprecedented agility, efficiency, and scalability for deploying large-scale AI workloads.",[10,10425,10427],{"id":10426},"transparency-our-commitment-to-authenticity",[14,10428,10429],{},"Transparency: Our Commitment to Authenticity",[18,10431,10432],{},"At Paiton, we pride ourselves on real, measurable results. Our journey is distinct:",[30,10434,10435,10438,10441,10444,10447,10450],{},[33,10436,10437],{},"Lean & Independent: Just 3 engineers, entirely self-funded with zero external investments or support.",[33,10439,10440],{},"Self-Reliance: No financial, technical, or promotional assistance from AMD or other external entities; fully self-financed investment in our MI300X hardware.",[33,10442,10443],{},"Proven Expertise: Previously shipped over 250,000 GPUs, thousands of AMD EPYC CPUs, and numerous AI servers, successfully spinning up large-scale AI clusters worldwide.",[33,10445,10446],{},"Original Innovation: Unlike many startups leveraging open-source software or superficial wrappers, we build everything, including deep kernel optimizations, from scratch.",[33,10448,10449],{},"Direct Message to AMD: While AMD’s Aiter library is commendable, our compact team achieves consistently superior performance, demonstrating efficiency and innovation unmatched even by larger, funded teams.",[33,10451,10452],{},"Results Over Hype: We never shout before we deliver. Unlike others that secure millions in funding with limited tangible outcomes, we achieve groundbreaking results first and let those results speak for themselves.",[18,10454,10455],{},"Together, let’s redefine what’s achievable with cutting-edge AI technology and AMD hardware.",{"title":285,"searchDepth":673,"depth":673,"links":10457},[10458,10459,10460,10461,10462,10463,10464,10465,10466],{"id":10046,"depth":676,"text":10049},{"id":8280,"depth":676,"text":8283},{"id":10104,"depth":676,"text":10107},{"id":10136,"depth":676,"text":10139},{"id":10165,"depth":676,"text":10168},{"id":10285,"depth":676,"text":10288},{"id":10314,"depth":676,"text":10317},{"id":10417,"depth":676,"text":10420},{"id":10426,"depth":676,"text":10429},[688,689,68,1149,6594,191,10468,5626,10123,10469,10470,68,10471,618],"Cold Start","Llama 3.1 405B","LLM Optimization","Startup Latency","2025-06-12T20:15:23","With Paiton, we’re not merely pursuing peak inference speeds, we’re fundamentally reshaping the entire lifecycle of large language model (LLM) deployment. Our latest endeavor pairs AMD’s cutting-edge MI300X GPUs with the colossal Llama-3.1-405B-Instruct-FP8-KV model, achieving groundbreaking milestones: Visual Demonstration: Startup Speed Showcase We’re excited to share a visual demonstration of Paiton’s revolutionary startup performance. Watch ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fservingscreenshot.webp",{},"https:\u002F\u002Feliovp.com\u002Fpaiton-dramatically-faster-startup-and-performance-for-llama-3-1-405b\u002F","\u002Fblog\u002Fpaiton-dramatically-faster-startup-and-performance-for-llama-3-1-405b",{"title":10021,"description":10473},"paiton-dramatically-faster-startup-and-performance-for-llama-3-1-405b","blog\u002Fpaiton-dramatically-faster-startup-and-performance-for-llama-3-1-405b","Bi6bFPHCRe_7kk06bcwSwvQZO7Uc8dj6KLauYNmhLXw",{"id":10483,"title":10484,"body":10485,"categories":11011,"date":11018,"description":11019,"extension":699,"image":11020,"meta":11021,"navigation":702,"originalUrl":8200,"path":11022,"seo":11023,"slug":11024,"stem":11025,"__hash__":11026},"blog\u002Fblog\u002Fpaiton-fp8-beats-nvidias-h200-on-amds-mi300x.md","Paiton FP8 Beats NVIDIA’s H200 on AMD’s MI300X",{"type":7,"value":10486,"toc":11005},[10487,10490,10493,10496,10502,10505,10508,10544,10550,10553,10556,10562,10565,10568,10923,10929,10934,10941,10947,10985,10988,10991,10994,10997],[18,10488,10489],{},"The world of AI is moving at an unprecedented pace, and efficient inference is key to deploying powerful models in real-world applications. At Eliovp, we’ve consistently pushed the boundaries of AI performance, as highlighted in our previous blogs showcasing significant inference speedups when benchmarking with fp16\u002Fbf16. Now, we’re thrilled to announce a further significant leap forward: Paiton now achieves superior inference performance on AMD MI300X GPUs with FP8, outperforming NVIDIA’s H200. While both AMD and NVIDIA support FP8, our meticulously optimized kernels deliver these breakthrough results.",[18,10491,10492],{},"For too long, the narrative around AI hardware has been dominated by a single player. However, AMD’s MI300 series is a game-changer, offering incredible compute power. The challenge, as always, lies in extracting every ounce of that power. That’s where Paiton comes in.",[18,10494,10495],{},"Our core mission at Paiton is to optimize existing AI models by fine-tuning the kernels used for inference, with a specific focus on AMD architectures. We dive deep into the underlying operations, rewriting, fusing, and refining them to perfectly align with the unique strengths of AMD’s hardware. The result? Dramatically faster inference speeds and more efficient resource utilization.",[10,10497,10499],{"id":10498},"testbed-methodology",[14,10500,10501],{},"Testbed & Methodology",[18,10503,10504],{},"To ensure transparency and reproducibility, here’s our benchmarking setup:",[18,10506,10507],{},"As always, we made sure to use the best possible setup to keep things fair.",[30,10509,10510,10521,10527,10532,10538],{},[33,10511,10512,10515,10516],{},[14,10513,10514],{},"Inference Library:"," vLLM v0.9.0 + ",[163,10517,10520],{"href":10518,"rel":10519},"https:\u002F\u002Fhuggingface.co\u002Famd\u002FLlama-3.1-70B-Instruct-FP8-KV",[167],"amd\u002FLlama-3.1-70B-Instruct-FP8-KV",[33,10522,10523,10526],{},[14,10524,10525],{},"Hardware:"," AMD MI300X (192 GB HBM3) vs. NVIDIA H200 (141GB HBM3e)",[33,10528,10529,10531],{},[14,10530,6701],{}," ROCm 6.3.1 (we know we can get even better results when upgrading to 6.4) on Ubuntu 22.04; CUDA 12.x on Ubuntu 22.04",[33,10533,10534,10537],{},[14,10535,10536],{},"Measurements:"," Averaged over 10 inferences, capturing both cold-start and steady-state Time to First Token (TTFT) and end-to-end latency (API call → final token).",[33,10539,10540,10543],{},[14,10541,10542],{},"Batch Sizes:"," 1–128 to reflect interactive AI loads (1–64) vs. batch-processing extremes (128).",[10,10545,10547],{"id":10546},"the-fp8-advantage-the-era-of-efficiency",[14,10548,10549],{},"The FP8 Advantage: The Era of Efficiency",[18,10551,10552],{},"The introduction of FP8 (8-bit floating point) precision has been a monumental step towards more efficient AI inference for the industry as a whole. By reducing the memory footprint and computational intensity, FP8 allows for larger models to be deployed or existing models to run at significantly higher throughputs while utilizing less memory. While FP8 support is available across modern AI hardware platforms from both AMD and NVIDIA, harnessing its full potential requires highly optimized software implementations.",[18,10554,10555],{},"We’ve been working tirelessly to integrate and optimize our kernels for FP8 on AMD. We’re incredibly proud to announce that our recent benchmarks demonstrate a clear victory for AMD MI300X equipped with Paiton’s optimized kernels.",[10,10557,10559],{"id":10558},"paiton-on-amd-mi300x-setting-a-new-standard",[14,10560,10561],{},"Paiton on AMD MI300X: Setting a New Standard",[18,10563,10564],{},"Our internal testing shows that with Paiton’s optimized kernels, the AMD MI300X GPU achieves demonstrably better results compared to the NVIDIA H200 GPU when running the amd\u002FLlama-3.1-70B-Instruct-FP8-KV model for inference in FP8 precision.",[18,10566,10567],{},"Here are the benchmark results comparing NVIDIA H200 and AMD MI300X performance with Paiton’s optimized kernels:",[298,10569,10570,10604],{},[301,10571,10572],{},[304,10573,10574,10577,10580,10583,10586,10589,10592,10595,10598,10601],{},[307,10575,10576],{},"Batch",[307,10578,10579],{},"H200 (Req\u002Fs)",[307,10581,10582],{},"MI300X + Paiton (Req\u002Fs)",[307,10584,10585],{},"H200 (Tok\u002Fs)",[307,10587,10588],{},"MI300X + Paiton (Tok\u002Fs)",[307,10590,10591],{},"H200 E2E (ms)",[307,10593,10594],{},"MI300X E2E (ms)",[307,10596,10597],{},"H200 TTFT (ms)",[307,10599,10600],{},"MI300X TTFT (ms)",[307,10602,10603],{},"Overall Winner",[316,10605,10606,10646,10686,10726,10766,10806,10846,10885],{},[304,10607,10608,10610,10613,10618,10621,10626,10629,10634,10639,10642],{},[321,10609,6802],{},[321,10611,10612],{},"0.16",[321,10614,10615],{},[14,10616,10617],{},"0.17",[321,10619,10620],{},"39.9",[321,10622,10623],{},[14,10624,10625],{},"43.4",[321,10627,10628],{},"640.7",[321,10630,10631],{},[14,10632,10633],{},"589.5",[321,10635,10636],{},[14,10637,10638],{},"136.3",[321,10640,10641],{},"138.6",[321,10643,10644],{},[14,10645,68],{},[304,10647,10648,10650,10653,10658,10661,10666,10669,10674,10677,10682],{},[321,10649,6819],{},[321,10651,10652],{},"0.31",[321,10654,10655],{},[14,10656,10657],{},"0.34",[321,10659,10660],{},"78.5",[321,10662,10663],{},[14,10664,10665],{},"87.0",[321,10667,10668],{},"518.5",[321,10670,10671],{},[14,10672,10673],{},"477.7",[321,10675,10676],{},"133.6",[321,10678,10679],{},[14,10680,10681],{},"122.4",[321,10683,10684],{},[14,10685,68],{},[304,10687,10688,10690,10693,10698,10701,10706,10709,10714,10717,10722],{},[321,10689,6836],{},[321,10691,10692],{},"0.60",[321,10694,10695],{},[14,10696,10697],{},"0.67",[321,10699,10700],{},"129.6",[321,10702,10703],{},[14,10704,10705],{},"172.0",[321,10707,10708],{},"562.2",[321,10710,10711],{},[14,10712,10713],{},"441.6",[321,10715,10716],{},"134.3",[321,10718,10719],{},[14,10720,10721],{},"132.6",[321,10723,10724],{},[14,10725,68],{},[304,10727,10728,10730,10733,10738,10741,10746,10749,10754,10759,10762],{},[321,10729,6853],{},[321,10731,10732],{},"1.18",[321,10734,10735],{},[14,10736,10737],{},"1.31",[321,10739,10740],{},"225.0",[321,10742,10743],{},[14,10744,10745],{},"252.1",[321,10747,10748],{},"587.6",[321,10750,10751],{},[14,10752,10753],{},"493.3",[321,10755,10756],{},[14,10757,10758],{},"269.9",[321,10760,10761],{},"280.9",[321,10763,10764],{},[14,10765,68],{},[304,10767,10768,10770,10773,10778,10781,10786,10789,10794,10799,10802],{},[321,10769,6870],{},[321,10771,10772],{},"2.21",[321,10774,10775],{},[14,10776,10777],{},"2.45",[321,10779,10780],{},"343.1",[321,10782,10783],{},[14,10784,10785],{},"470.1",[321,10787,10788],{},"560.6",[321,10790,10791],{},[14,10792,10793],{},"502.5",[321,10795,10796],{},[14,10797,10798],{},"461.3",[321,10800,10801],{},"513.8",[321,10803,10804],{},[14,10805,68],{},[304,10807,10808,10810,10813,10818,10821,10826,10829,10834,10837,10842],{},[321,10809,349],{},[321,10811,10812],{},"3.93",[321,10814,10815],{},[14,10816,10817],{},"4.07",[321,10819,10820],{},"591.8",[321,10822,10823],{},[14,10824,10825],{},"814.5",[321,10827,10828],{},"659.1",[321,10830,10831],{},[14,10832,10833],{},"640.6",[321,10835,10836],{},"1188.7",[321,10838,10839],{},[14,10840,10841],{},"1187.5",[321,10843,10844],{},[14,10845,68],{},[304,10847,10848,10850,10855,10858,10863,10866,10871,10874,10877,10882],{},[321,10849,6903],{},[321,10851,10852],{},[14,10853,10854],{},"6.64",[321,10856,10857],{},"6.24",[321,10859,10860],{},[14,10861,10862],{},"1297.4",[321,10864,10865],{},"1179.7",[321,10867,10868],{},[14,10869,10870],{},"789.3",[321,10872,10873],{},"812.2",[321,10875,10876],{},"2362.3",[321,10878,10879],{},[14,10880,10881],{},"1880.3",[321,10883,10884],{},"Draw",[304,10886,10887,10889,10894,10897,10902,10905,10910,10913,10916,10921],{},[321,10888,6920],{},[321,10890,10891],{},[14,10892,10893],{},"9.75",[321,10895,10896],{},"9.01",[321,10898,10899],{},[14,10900,10901],{},"1821.9",[321,10903,10904],{},"1655.8",[321,10906,10907],{},[14,10908,10909],{},"607.7",[321,10911,10912],{},"635.5",[321,10914,10915],{},"3633.3",[321,10917,10918],{},[14,10919,10920],{},"3502.0",[321,10922,693],{},[18,10924,10925,10928],{},[14,10926,10927],{},"Note:"," In preliminary tests, Paiton’s FP8 kernels also outperformed AMD’s own Aiter engine on MI300X.",[18,10930,10931],{},[283,10932],{"alt":285,"src":10933},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fh200vsmi300xfp8.webp",[18,10935,10936,10937,10940],{},"The qualitative results are undeniable: ",[14,10938,10939],{},"Paiton enables the AMD MI300X to process Llama-3.1-70B-Instruct-FP8-KV queries with higher throughput and lower latency than its NVIDIA counterpart across small to mid-range batch sizes, and often achieves better Time to First Token (TTFT)."," This is a direct testament to the power of targeted kernel optimization. While NVIDIA H200 shows strong performance at the highest batch sizes, our focus on optimizing for the most common real-world scenarios (smaller, more frequent requests, typical of interactive AI applications) demonstrates Paiton’s significant and impactful advantage in practical deployments. This means faster responses for end-users and more efficient utilization of your AMD hardware in common operational profiles.",[10,10942,10944],{"id":10943},"why-this-matters-for-you",[14,10945,10946],{},"Why This Matters for You",[30,10948,10949,10955,10965,10975],{},[33,10950,10951,10954],{},[14,10952,10953],{},"Unleash AMD’s True Potential:"," Investing in cutting-edge hardware like AMD MI300X GPUs should mean unlocking their full capabilities, not leaving performance on the table due to generic software. Paiton ensures you’re extracting every possible compute cycle, allowing your AMD infrastructure to reach its maximum potential for AI inference workloads. Our fine-tuned kernels mean your GPUs are working at peak efficiency, minimizing idle time and maximizing useful computation.",[33,10956,10957,10960,10961,10964],{},[14,10958,10959],{},"Cost-Effective AI Deployment:"," Higher inference efficiency directly translates into tangible cost savings. When your models run faster and more efficiently, you can process more requests, or complete existing workloads in less time. This directly impacts your operational expenses, whether you’re managing an on-premise data center or utilizing cloud-based GPU instances. Over time, these optimizations can lead to ",[630,10962,10963],{},"substantial reductions in your total cost of ownership"," for AI infrastructure.",[33,10966,10967,10970,10971,10974],{},[14,10968,10969],{},"Future-Proof Your Deployments:"," The landscape of AI models is constantly evolving, with a clear trend towards larger, more complex architectures and the adoption of lower precision formats like FP8. As these models grow and their computational demands intensify, highly optimized inference solutions will become not just beneficial, but absolutely critical for maintaining performance and scalability. ",[630,10972,10973],{},"Paiton’s continuous development"," ensures your AI deployments remain at the forefront, ready to handle the next generation of models and precision requirements with confidence.",[33,10976,10977,10980,10981,10984],{},[14,10978,10979],{},"Broader Hardware Choice:"," Our work with Paiton empowers developers and organizations to confidently choose AMD for their demanding AI inference needs. By demonstrating ",[630,10982,10983],{},"superior performance on AMD MI210, MI250X, MI300X, MI325X",", MI355X, we’re fostering a more competitive and innovative hardware ecosystem, giving you more flexibility and options in designing your AI infrastructure. This competition drives innovation across the industry, ultimately benefiting all users.",[18,10986,10987],{},"The era of one-size-fits-all AI optimization is rapidly drawing to a close. Paiton is at the forefront, leading the charge in delivering specialized, high-performance solutions meticulously crafted for AMD GPUs. With our latest enhancements, including the breakthrough FP8 optimizations, we’re not just optimizing; we’re actively redefining what’s possible for AI inference on AMD hardware.",[18,10989,10990],{},"Stay tuned for more in-depth technical breakdowns and further benchmark releases as we continue to push the boundaries of AI performance, and during these FP8 benchmarks we uncovered additional optimization techniques that we’ll showcase in upcoming deep-dive posts! We’ll soon be sharing exciting results on Tensor Parallelism (TP), where Paiton continues to demonstrate superior performance and scalability, even surpassing AMD’s own AITER engine in our internal tests. If you’re running AI models on AMD hardware, it’s time to experience the Paiton difference and see how our kernel optimizations can elevate your AI inference capabilities.",[18,10992,10993],{},"If you’re running AI models on AMD hardware, it’s time to experience the Paiton difference and see how our kernel optimizations can elevate your AI inference capabilities.",[18,10995,10996],{},"Reach out for a demo and elevate your AI inference performance today!",[18,10998,10999],{},[14,11000,11001,11002],{},"– ",[630,11003,11004],{},"The Paiton Team –",{"title":285,"searchDepth":673,"depth":673,"links":11006},[11007,11008,11009,11010],{"id":10498,"depth":676,"text":10501},{"id":10546,"depth":676,"text":10549},{"id":10558,"depth":676,"text":10561},{"id":10943,"depth":676,"text":10946},[688,689,68,1149,191,11012,3976,961,11013,11014,11015,10469,11016,11017],"Cold Start Optimization","GPU Performance","Inference Latency","Large Language Models","Model Serving","vLLM Optimization","2025-06-08T19:12:40","The world of AI is moving at an unprecedented pace, and efficient inference is key to deploying powerful models in real-world applications. At Eliovp, we’ve consistently pushed the boundaries of AI performance, as highlighted in our previous blogs showcasing significant inference speedups when benchmarking with fp16\u002Fbf16. Now, we’re thrilled to announce a further significant leap ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fblognewfp8.webp",{},"\u002Fblog\u002Fpaiton-fp8-beats-nvidias-h200-on-amds-mi300x",{"title":10484,"description":11019},"paiton-fp8-beats-nvidias-h200-on-amds-mi300x","blog\u002Fpaiton-fp8-beats-nvidias-h200-on-amds-mi300x","REUe74Eew0-O5Nix7z61cZQd-SmQ5PgvmE23jN7AofQ",{"id":11028,"title":11029,"body":11030,"categories":11753,"date":11756,"description":11757,"extension":699,"image":11758,"meta":11759,"navigation":702,"originalUrl":11760,"path":11761,"seo":11762,"slug":11763,"stem":11764,"__hash__":11765},"blog\u002Fblog\u002Fmi300x-vs-h200-vs-rx-7900-xtx-vs-tenstorrent-n300s-with-vllm.md","MI300X vs H200 vs RX 7900 XTX vs Tenstorrent n300s with vLLM",{"type":7,"value":11031,"toc":11741},[11032,11035,11038,11050,11056,11059,11062,11068,11178,11183,11189,11196,11204,11207,11210,11216,11359,11371,11376,11389,11420,11426,11501,11506,11516,11519,11522,11527,11532,11546,11551,11564,11569,11583,11588,11617,11669,11674,11681,11691,11697,11703,11712,11718,11721,11735,11738],[18,11033,11034],{},"As large language models (LLMs) become a foundational part of modern applications, picking the right server for deployment is more important than ever. Whether you’re an enterprise scaling up inference, a startup optimizing for cost, or a researcher pushing throughput boundaries. This blog compares two high-profile server setups and two not so high-profile setups which are usually not used as servers in a DC, each with unique GPU\u002Faccelerator hardware and using vLLM.",[18,11036,11037],{},"We’ll compare:",[114,11039,11040,11042,11044,11047],{},[33,11041,191],{},[33,11043,6102],{},[33,11045,11046],{},"Tenstorrent n300s",[33,11048,11049],{},"AMD RX 7900 XTX",[18,11051,11052,11053,46],{},"While comparing the RX 7900 XTX to data center-class GPUs may seem unfair, it’s important to recognize its versatility. As a multi-purpose GPU capable of both high-end gaming and AI development, it offers a practical advantage: you can use the same system for development and leisure, making it a compelling option for individual developers, small teams, or data centers that want to ",[14,11054,11055],{},"dynamically switch between gaming and AI workloads based on demand",[18,11057,11058],{},"That said, if we’re being honest, we’re including the RX 7900 XTX primarily because we already have it available and were curious to see how it stacks up against dedicated data center hardware.",[18,11060,11061],{},"The other odd one in this race is the Tenstorrent system, it’s been showing great potential and promise and it caught my eye about 3 years ago. We were very keen to test this system against its giant competitors because it would add some variety and excitement to the current two main players in AI (sorry Intel).",[10,11063,11065],{"id":11064},"gpu-specs-as-of-q2-2025",[14,11066,11067],{},"GPU Specs (as of Q2 2025)",[298,11069,11070,11103],{},[301,11071,11072],{},[304,11073,11074,11079,11083,11088,11093,11098],{},[307,11075,11076],{},[14,11077,11078],{},"Accelerator",[307,11080,11081],{},[14,11082,9160],{},[307,11084,11085],{},[14,11086,11087],{},"VRAM",[307,11089,11090],{},[14,11091,11092],{},"Memory Type",[307,11094,11095],{},[14,11096,11097],{},"TDP",[307,11099,11100],{},[14,11101,11102],{},"Approx Price (USD)",[316,11104,11105,11123,11142,11161],{},[304,11106,11107,11109,11111,11114,11117,11120],{},[321,11108,191],{},[321,11110,5554],{},[321,11112,11113],{},"192 GB",[321,11115,11116],{},"HBM3",[321,11118,11119],{},"750W",[321,11121,11122],{},"~$15,000",[304,11124,11125,11127,11130,11133,11136,11139],{},[321,11126,6102],{},[321,11128,11129],{},"Hopper",[321,11131,11132],{},"141 GB",[321,11134,11135],{},"HBM3e",[321,11137,11138],{},"700W",[321,11140,11141],{},"~$30,000",[304,11143,11144,11146,11149,11152,11155,11158],{},[321,11145,11049],{},[321,11147,11148],{},"RDNA3",[321,11150,11151],{},"24 GB",[321,11153,11154],{},"GDDR6",[321,11156,11157],{},"355W",[321,11159,11160],{},"~$1,000",[304,11162,11163,11165,11168,11170,11172,11175],{},[321,11164,11046],{},[321,11166,11167],{},"Custom RISC-V",[321,11169,11151],{},[321,11171,11154],{},[321,11173,11174],{},"300W",[321,11176,11177],{},"~$1,399",[18,11179,11180],{},[630,11181,11182],{},"Note: These values reflect typical specs for each GPU\u002Faccelerator. Actual performance can vary based on system integration and workload characteristics. (e.g., CPU, motherboard, cooling). The focus here is on the GPU\u002Faccelerator as the main differentiator.",[10,11184,11186],{"id":11185},"power-measurement-method",[14,11187,11188],{},"Power Measurement Method",[18,11190,11191,11192,11195],{},"This comparison was challenging because accurate per-GPU cost per million tokens for MI300X and H200 is difficult to estimate since individual GPU prices are not publicly available. Therefore, we calculated cost per million tokens using the full system price and approximate power consumption. This approach makes more sense, in real deployments, system-level costs (power, hardware, infrastructure) contribute to operational expenses beyond just the GPU. To account for the full system, we multiplied the measured token throughput and single GPU power consumption by the number of GPUs in the server and then added that to the total system power consumption, effectively spreading system cost and power across all GPUs under the assumption of ",[14,11193,11194],{},"full utilization",". Note that these values represent idle or estimated power consumption only and do not account for additional components or increased draw under full system utilization.",[18,11197,11198,11199,46],{},"The idle power consumption for the H200 system was not directly measured, but an estimate was obtained from an article, which suggests that the system typically consumes around 2200W when idle. For more details, refer to the ",[163,11200,11203],{"href":11201,"rel":11202},"https:\u002F\u002Fwww.servethehome.com\u002Fsupermicro-sys-821ge-tnhr-8x-nvidia-h200-gpu-air-cooled-bluefield-intel-xeon-astera-broadcom\u002F5\u002F#:~:text=Supermicro%20SYS%2D821GE%2DTNHR%20Power%20Delivery&text=Generally%20with%20this%20system%2C%20we,around%2010kW%20at%20its%20peak",[167],"ServeTheHome article",[18,11205,11206],{},"The idle power consumption of the RX 7900 system could not be directly measured and was instead estimated to be around 400W.",[18,11208,11209],{},"The power consumption values for other systems were measured using the ipmitool tool.",[10,11211,11213],{"id":11212},"server-configurations-and-pricing",[14,11214,11215],{},"Server Configurations and Pricing",[298,11217,11218,11262],{},[301,11219,11220],{},[304,11221,11222,11227,11232,11237,11242,11247,11252,11257],{},[307,11223,11224],{},[14,11225,11226],{},"Server",[307,11228,11229],{},[14,11230,11231],{},"CPU",[307,11233,11234],{},[14,11235,11236],{},"RAM",[307,11238,11239],{},[14,11240,11241],{},"Storage",[307,11243,11244],{},[14,11245,11246],{},"Cooling",[307,11248,11249],{},[14,11250,11251],{},"# of GPUs",[307,11253,11254],{},[14,11255,11256],{},"Idle System Power (Est.)",[307,11258,11259],{},[14,11260,11261],{},"System Price (USD)",[316,11263,11264,11289,11311,11335],{},[304,11265,11266,11269,11272,11275,11278,11281,11283,11286],{},[321,11267,11268],{},"A+ Server 8125GS-TNMR2",[321,11270,11271],{},"2x EPYC 9654",[321,11273,11274],{},"1536GB",[321,11276,11277],{},"4TB NVMe",[321,11279,11280],{},"Air",[321,11282,6853],{},[321,11284,11285],{},"~2400W",[321,11287,11288],{},"~$260,564",[304,11290,11291,11294,11297,11299,11301,11303,11305,11308],{},[321,11292,11293],{},"SuperServer 821GE-TNHR",[321,11295,11296],{},"2x Intel Xeon Platinum 8468H",[321,11298,11274],{},[321,11300,11277],{},[321,11302,11280],{},[321,11304,6853],{},[321,11306,11307],{},"~2200W",[321,11309,11310],{},"~$307,336",[304,11312,11313,11316,11319,11322,11325,11327,11329,11332],{},[321,11314,11315],{},"DIY AMD Workstation",[321,11317,11318],{},"Ryzen 9 7950X",[321,11320,11321],{},"64GB",[321,11323,11324],{},"2TB NVMe",[321,11326,11280],{},[321,11328,6819],{},[321,11330,11331],{},"~400W",[321,11333,11334],{},"~$3,500",[304,11336,11337,11340,11343,11346,11348,11351,11353,11356],{},[321,11338,11339],{},"Tenstorrent Loudbox",[321,11341,11342],{},"2x Intel® Xeon® Silver 4309Y",[321,11344,11345],{},"512GB",[321,11347,11277],{},[321,11349,11350],{},"Passive",[321,11352,6836],{},[321,11354,11355],{},"~700W",[321,11357,11358],{},"$12,000",[18,11360,11361,999,11364],{},[630,11362,11363],{},"Note: Configurations are representative. Real-world builds may vary depending on components, vendors, and integration costs. The prices here are found via",[163,11365,11368],{"href":11366,"rel":11367},"http:\u002F\u002Fwww.thinkmate.com",[167],[630,11369,11370],{},"www.thinkmate.com",[10,11372,11373],{"id":727},[14,11374,11375],{},"Benchmark Setup",[18,11377,11378,999,11381,999,11386],{},[630,11379,11380],{},"Note: All benchmarks were run using",[630,11382,11383],{},[14,11384,11385],{},"a single GPU or accelerator card per system",[630,11387,11388],{},"to ensure a fair comparison across different hardware classes.",[30,11390,11391,11396,11402,11408,11414],{},[33,11392,11393,11395],{},[14,11394,8372],{}," vLLM (paged attention + continuous batching)",[33,11397,11398,11401],{},[14,11399,11400],{},"Model:"," meta-llama\u002FMeta-Llama-3-8B-Instruct",[33,11403,11404,11407],{},[14,11405,11406],{},"Workload:"," Concurrent prompts, batch size 32, fixed output length of 256 tokens",[33,11409,11410,11413],{},[14,11411,11412],{},"Dataset",": ShareGPT",[33,11415,11416,11419],{},[14,11417,11418],{},"Metrics:"," Tokens\u002Fsec (throughput) and cost-performance",[10,11421,11423],{"id":11422},"vllm-benchmark-results-batch-size-32-only",[14,11424,11425],{},"vLLM Benchmark Results (Batch Size 32 Only)",[298,11427,11428,11450],{},[301,11429,11430],{},[304,11431,11432,11436,11440,11445],{},[307,11433,11434],{},[14,11435,11226],{},[307,11437,11438],{},[14,11439,11251],{},[307,11441,11442],{},[14,11443,11444],{},"Tokens\u002Fsec per GPU",[307,11446,11447],{},[14,11448,11449],{},"System Tokens\u002Fsec",[316,11451,11452,11464,11476,11489],{},[304,11453,11454,11456,11458,11461],{},[321,11455,191],{},[321,11457,6853],{},[321,11459,11460],{},"7003.10",[321,11462,11463],{},"56,024.8",[304,11465,11466,11468,11470,11473],{},[321,11467,6102],{},[321,11469,6853],{},[321,11471,11472],{},"8192.08",[321,11474,11475],{},"65,536.64",[304,11477,11478,11481,11483,11486],{},[321,11479,11480],{},"AMD RX 7900 XTX(*)",[321,11482,6819],{},[321,11484,11485],{},"1113.59",[321,11487,11488],{},"2227.18",[304,11490,11491,11493,11495,11498],{},[321,11492,11339],{},[321,11494,6836],{},[321,11496,11497],{},"1314.0",[321,11499,11500],{},"5256.0",[18,11502,11503],{},[630,11504,11505],{},"Note: We encountered out-of-memory (OOM) errors when the model context length was 131072 with the AMD RX 7900 XTX. It was lowered to 22048, which is a significant change.",[10,11507,11509,999,11513],{"id":11508},"cost-per-million-tokensnote-for-the-simplicity-of-comparison-we-are-using-a-3-year-lifespan-26280-hours-power-costs-are-calculated-with-a-rate-of-010-per-kwh",[14,11510,9537,11511],{},[630,11512,10927],{},[630,11514,11515],{},"For the simplicity of comparison, we are using a 3-year lifespan (26280 hours). Power costs are calculated with a rate of $0.10 per kWh.",[18,11517,11518],{},"The calculation used to get the cost per 1M tokens can be seen and reviewed here:",[18,11520,11521],{},"Assuming a 3 year depreciation and full system+GPU utilization",[18,11523,11524],{},[14,11525,11526],{},"AMD MI300X Server:",[18,11528,11529],{},[630,11530,11531],{},"Assuming 2400W idle power consumption and 8 GPUs fully utilized at 750W, the total would be 8.4kW",[114,11533,11534,11537,11540,11543],{},[33,11535,11536],{},"System Tokens\u002Fsec: 7003.10 tokens\u002Fsec\u002FGPU * 8 GPUs = 56,024.8 tokens\u002Fsec",[33,11538,11539],{},"Tokens per hour: 56,024.8 tokens\u002Fsec * 3600 sec\u002Fhour = 201,689,280 tokens\u002Fhour",[33,11541,11542],{},"Cost per hour: $260,564 \u002F 26280 hours + 8.4kW * $0.10\u002FkWh = $10.76",[33,11544,11545],{},"Cost per 1M tokens: $10.76 \u002F (201,689,280 \u002F 1,000,000) = $0.053 per 1M tokens",[18,11547,11548],{},[14,11549,11550],{},"AMD MI300X Server + Paiton:",[114,11552,11553,11556,11559,11561],{},[33,11554,11555],{},"System Tokens\u002Fsec: 7637.12 tokens\u002Fsec\u002FGPU * 8 GPUs = 61,096.96 tokens\u002Fsec",[33,11557,11558],{},"Tokens per hour: 61,096.96 tokens\u002Fsec * 3600 sec\u002Fhour = 219,949,056 tokens\u002Fhour",[33,11560,11542],{},[33,11562,11563],{},"Cost per 1M tokens: $10.76 \u002F (219,949,056 \u002F 1,000,000) = $0.049 per 1M tokens",[18,11565,11566],{},[14,11567,11568],{},"Nvidia H200 Server:",[114,11570,11571,11574,11577,11580],{},[33,11572,11573],{},"System Tokens\u002Fsec: 8192.08 tokens\u002Fsec\u002FGPU * 8 GPUs = 65,536.64 tokens\u002Fsec",[33,11575,11576],{},"Tokens per hour: 65,536.64 tokens\u002Fsec * 3600 sec\u002Fhour = 235,931,904 tokens\u002Fhour",[33,11578,11579],{},"Cost per hour: $307,336 \u002F 26280 hours + 7.8kW * $0.10\u002FkWh = $12.48",[33,11581,11582],{},"Cost per 1M tokens: $12.48 \u002F (235,931,904 \u002F 1,000,000) = $0.053 per 1M tokens",[18,11584,11585],{},[14,11586,11587],{},"AMD RX 7900 XTX Workstation:",[114,11589,11590,11593,11596,11599,11605,11608,11611,11614],{},[33,11591,11592],{},"System Tokens\u002Fsec: 1113.59 tokens\u002Fsec\u002FGPU * 2 GPUs = 2227.18 tokens\u002Fsec",[33,11594,11595],{},"Tokens per hour: 2227.18 tokens\u002Fsec * 3600 sec\u002Fhour = 8,018,153 tokens\u002Fhour",[33,11597,11598],{},"Cost per hour: $3,500 \u002F 26280 hours + 1.054kW * $0.10\u002FkWh = $0.24",[33,11600,11601,11602],{},"Cost per 1M tokens: $0.24 \u002F (8,018,153 \u002F 1,000,000) = $0.030 per 1M tokens ",[14,11603,11604],{},"with a context length of 22048Tenstorrent Loudbox:",[33,11606,11607],{},"System Tokens\u002Fsec: 1314 tokens\u002Fsec\u002Fcard * 4 cards = 5256 tokens\u002Fsec",[33,11609,11610],{},"Tokens per hour: 5256 tokens\u002Fsec * 3600 sec\u002Fhour = 18,921,600 tokens\u002Fhour",[33,11612,11613],{},"Cost per hour: $12,000 \u002F 26280 hours + 1.9kW * $0.10\u002FkWh = $0.65",[33,11615,11616],{},"Cost per 1M tokens: $0.65 \u002F (18,921,600 \u002F 1,000,000) = $0.034 per 1M tokens",[298,11618,11619,11631],{},[301,11620,11621],{},[304,11622,11623,11627],{},[307,11624,11625],{},[14,11626,11226],{},[307,11628,11629],{},[14,11630,6335],{},[316,11632,11633,11640,11647,11653,11662],{},[304,11634,11635,11637],{},[321,11636,191],{},[321,11638,11639],{},"$0.053",[304,11641,11642,11644],{},[321,11643,6150],{},[321,11645,11646],{},"$0.049",[304,11648,11649,11651],{},[321,11650,6102],{},[321,11652,11639],{},[304,11654,11655,11657],{},[321,11656,11049],{},[321,11658,11659],{},[630,11660,11661],{},"$0.030 (lower context length)",[304,11663,11664,11666],{},[321,11665,11339],{},[321,11667,11668],{},"$0.034",[18,11670,11671],{},[283,11672],{"alt":285,"src":11673},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fcharttenstorrent.webp",[10,11675,11677,11680],{"id":11676},"observationsamd-mi300x-competitive-throughput-with-massive-vram-ideal-for-mid-to-large-batch-sizes-and-larger-models-like-llama-3-70b-and-beyond-comparing-a-model-like-llama3-8b-was-purely-done-due-to-the-other-cards-not-being-able-to-load-larger-models-on-a-single-gpu-the-mi300x-should-not-be-used-for-these-use-cases-unless-gpu-partitioning-is-done",[14,11678,11679],{},"ObservationsAMD MI300X:"," Competitive throughput with massive VRAM. Ideal for mid-to-large batch sizes and larger models like Llama 3-70B and beyond. Comparing a model like Llama3-8B was purely done due to the other cards not being able to load larger models on a single GPU, the MI300X should not be used for these use-cases unless GPU partitioning is done.",[18,11682,11683,11686,11687,11690],{},[14,11684,11685],{},"AMD MI300X + Paiton:"," Including the observations we’ve discussed above, Paiton achieved a ",[14,11688,11689],{},"8.2% lower cost per million tokens"," compared to the standard MI300X setup, demonstrating improved cost-efficiency from software optimizations. The Paiton Framework is continuously improving, and as development progresses, this cost will continue to decrease over time.",[18,11692,11693,11696],{},[14,11694,11695],{},"NVIDIA H200:"," Industry-leading speed, similar to the AMD MI300X, and a mature CUDA software stack. Ease-of-use is definitely the case with Nvidia.",[18,11698,11699,11702],{},[14,11700,11701],{},"RX 7900 XTX:"," A cost-effective choice for individual developers. Not ideal for larger workloads due to VRAM limitations and the annoying context-length limitation, but great for light inferencing and development. Useful for AI\u002FML workloads and any other general GPU related workload.",[18,11704,11705,11708,11709,46],{},[14,11706,11707],{},"Tenstorrent n300s:"," Innovative RISC-V architecture tailored for ML workloads. Emerging support in inference frameworks like vLLM, but the ecosystem is still growing. The cost per 1M tokens is competitive for smaller models, ",[14,11710,11711],{},"if the model is supported",[10,11713,11715],{"id":11714},"final-thoughts",[14,11716,11717],{},"Final Thoughts",[18,11719,11720],{},"Choosing the right server hinges on your use case:",[30,11722,11723,11729],{},[33,11724,11725,11728],{},[14,11726,11727],{},"Startups or researcher:"," RX 7900 XTX workstation or Tenstorrent Loudbox for low-cost experimentation and inference (just make sure the model you want to use is supported by Tenstorrent).",[33,11730,11731,11734],{},[14,11732,11733],{},"Enterprises:"," MI300X and H200 balance performance and efficiency well, however, the MI300X GPU does provide significantly larger VRAM at a lower cost.",[18,11736,11737],{},"Given the surprisingly competitive cost-efficiency of the RX 7900 XTX in this analysis, we’re excited to announce that RDNA support for Paiton is currently in development to further unlock the potential of these GPUs.",[18,11739,11740],{},"In a fast-evolving AI hardware landscape, pairing the right accelerator with optimized inference frameworks like vLLM ensures you’re making the most of your infrastructure.",{"title":285,"searchDepth":673,"depth":673,"links":11742},[11743,11744,11745,11746,11747,11748,11750,11752],{"id":11064,"depth":676,"text":11067},{"id":11185,"depth":676,"text":11188},{"id":11212,"depth":676,"text":11215},{"id":727,"depth":676,"text":11375},{"id":11422,"depth":676,"text":11425},{"id":11508,"depth":676,"text":11749},"Cost per Million TokensNote: For the simplicity of comparison, we are using a 3-year lifespan (26280 hours). Power costs are calculated with a rate of $0.10 per kWh.",{"id":11676,"depth":676,"text":11751},"ObservationsAMD MI300X: Competitive throughput with massive VRAM. Ideal for mid-to-large batch sizes and larger models like Llama 3-70B and beyond. Comparing a model like Llama3-8B was purely done due to the other cards not being able to load larger models on a single GPU, the MI300X should not be used for these use-cases unless GPU partitioning is done.",{"id":11714,"depth":676,"text":11717},[688,689,68,3971,1149,690,694,966,11754,11755],"RX7900XTX","tenstorrent","2025-05-09T14:03:58","As large language models (LLMs) become a foundational part of modern applications, picking the right server for deployment is more important than ever. Whether you’re an enterprise scaling up inference, a startup optimizing for cost, or a researcher pushing throughput boundaries. This blog compares two high-profile server setups and two not so high-profile setups which ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fcomparisontenstor.webp",{},"https:\u002F\u002Feliovp.com\u002Fmi300x-vs-h200-vs-rx-7900-xtx-vs-tenstorrent-n300s-with-vllm\u002F","\u002Fblog\u002Fmi300x-vs-h200-vs-rx-7900-xtx-vs-tenstorrent-n300s-with-vllm",{"title":11029,"description":11757},"mi300x-vs-h200-vs-rx-7900-xtx-vs-tenstorrent-n300s-with-vllm","blog\u002Fmi300x-vs-h200-vs-rx-7900-xtx-vs-tenstorrent-n300s-with-vllm","DO-_rtr_ivtvd3e3NtaKoKU-X1hgEZ4tcV_0yP5NiAk",{"id":11767,"title":11768,"body":11769,"categories":12231,"date":12233,"description":12234,"extension":699,"image":12235,"meta":12236,"navigation":702,"originalUrl":12237,"path":12238,"seo":12239,"slug":12240,"stem":12241,"__hash__":12242},"blog\u002Fblog\u002Fclusterpl-empowering-gpu-cluster-investors-with-real-world-financial-insights.md","ClusterP&L: Empowering GPU Cluster Investors with Real-World Financial Insights",{"type":7,"value":11770,"toc":12221},[11771,11774,11777,11780,11786,11788,11794,11820,11822,11828,11834,11845,11851,11859,11865,11876,11882,11894,11900,11911,11917,11925,11931,11942,11948,11956,11962,11973,11975,11981,11993,11998,12013,12018,12036,12041,12053,12055,12061,12064,12081,12086,12089,12091,12097,12100,12102,12106,12138,12140,12146,12149,12175,12184,12186,12192],[18,11772,11773],{},"At Eliovp BV, we’ve spent years on the cutting edge of GPU cluster deployment and optimization across Europe. Our team supports leading organizations in AI, finance, and research, architecting, building, and scaling high-performance infrastructure. Over time, our customers, both newcomers and seasoned adopters, repeatedly asked the same question:",[18,11775,11776],{},"“Can you help us build a P&L model for our GPU cluster investment?”",[18,11778,11779],{},"Time and again, we found ourselves crafting custom profitability models, refining spreadsheets, and running dynamic “what-if” scenarios for every project. This made it clear that many organizations need a user-friendly, continually updated tool for accurately forecasting the financial returns of their GPU clusters.",[18,11781,11782,11783,46],{},"That’s why we built ",[14,11784,11785],{},"ClusterP&L",[71,11787],{},[10,11789,11791],{"id":11790},"why-clusterpl",[14,11792,11793],{},"Why ClusterP&L?",[114,11795,11796,11802,11808,11814],{},[33,11797,11798,11801],{},[14,11799,11800],{},"Born from Real-World Demand","ClusterP&L was launched in direct response to our clients’ ongoing needs. Rather than creating a new P&L model from scratch for each engagement, we centralized our expertise in a tool that’s available anytime, anywhere.",[33,11803,11804,11807],{},[14,11805,11806],{},"Built by Practitioners, for Practitioners","Our team has hands-on experience with real-world deployments rather than relying solely on theoretical frameworks. Every ClusterP&L feature is rooted in authentic GPU infrastructure project insights.",[33,11809,11810,11813],{},[14,11811,11812],{},"Always Up to Date","In a rapidly evolving GPU market, staying current is critical. ClusterP&L is a living platform, regularly refreshed to reflect new hardware, pricing dynamics, and best practices.",[33,11815,11816,11819],{},[14,11817,11818],{},"Empowering Our Customers","Instead of waiting for custom spreadsheets, users can instantly explore scenarios, fine-tune assumptions, and access on-demand financial projections that keep them in the driver’s seat.",[71,11821],{},[10,11823,11825],{"id":11824},"key-features",[14,11826,11827],{},"Key Features",[1230,11829,11831],{"id":11830},"comprehensive-gpu-system-configuration",[14,11832,11833],{},"Comprehensive GPU System Configuration",[30,11835,11836,11839,11842],{},[33,11837,11838],{},"Model the economics of leading GPU systems, H200, B200, L40, MI325, and more.",[33,11840,11841],{},"Adjust system count, contract length, utilization, power costs, and PUE to reflect real-time conditions.",[33,11843,11844],{},"Leverage transparent pricing for both on-demand and term-based contracts.",[1230,11846,11848],{"id":11847},"preset-configurations-for-real-world-scenarios",[14,11849,11850],{},"Preset Configurations for Real-World Scenarios",[30,11852,11853,11856],{},[33,11854,11855],{},"Ready-to-use templates for AI Training, Inference, HPC Research, and Startup MVP clusters.",[33,11857,11858],{},"Each preset integrates established best practices to mirror typical industry deployments.",[1230,11860,11862],{"id":11861},"advanced-financial-modeling",[14,11863,11864],{},"Advanced Financial Modeling",[30,11866,11867,11870,11873],{},[33,11868,11869],{},"Generate key profitability metrics like NPV, IRR, ROI, and payback period.",[33,11871,11872],{},"Harness Monte Carlo simulations for risk assessment, enabling you to gauge confidence intervals and scenario variability.",[33,11874,11875],{},"Access a robust three-statement financial model (income statement, balance sheet, and cash flow) for comprehensive analysis.",[1230,11877,11879],{"id":11878},"holistic-networking-considerations",[14,11880,11881],{},"Holistic Networking Considerations",[30,11883,11884,11891],{},[33,11885,11886,11887,11890],{},"In addition to GPU hardware, ",[14,11888,11889],{},"we recognize the critical role of high-speed networking",", such as InfiniBand or advanced Ethernet solutions, to minimize bottlenecks and maximize cluster throughput.",[33,11892,11893],{},"Our real-world expertise with complex networking topologies will be integrated into future versions of ClusterP&L, helping you plan and forecast end-to-end infrastructure ROI.",[1230,11895,11897],{"id":11896},"scenario-management",[14,11898,11899],{},"Scenario Management",[30,11901,11902,11905,11908],{},[33,11903,11904],{},"Save, load, and compare multiple scenario configurations.",[33,11906,11907],{},"Explore future GPU price shifts, demand fluctuations, and hardware launches via the Scenario Explorer.",[33,11909,11910],{},"Export your results as PDF or CSV to share with stakeholders and team members.",[1230,11912,11914],{"id":11913},"interactive-visualizations",[14,11915,11916],{},"Interactive Visualizations",[30,11918,11919,11922],{},[33,11920,11921],{},"Dynamic charts break down costs, project profit, analyze sensitivities, and more.",[33,11923,11924],{},"Optimized for both desktop and mobile with seamless dark and light mode compatibility.",[1230,11926,11928],{"id":11927},"responsive-user-friendly-ui",[14,11929,11930],{},"Responsive, User-Friendly UI",[30,11932,11933,11936,11939],{},[33,11934,11935],{},"Leverages a mobile-first design that looks great on any device.",[33,11937,11938],{},"Simplified tabbed navigation for Core Configuration, Advanced Settings, Deployment Model, and Financial Summary.",[33,11940,11941],{},"A floating summary bar provides instant access to top-level metrics like Profit, ROI, and Payback.",[1230,11943,11945],{"id":11944},"custom-workload-templates",[14,11946,11947],{},"Custom Workload Templates",[30,11949,11950,11953],{},[33,11951,11952],{},"Out-of-the-box templates for LLM training, inference, and additional GPU-intensive workloads.",[33,11954,11955],{},"Token-based cost modeling and performance metrics tailored to AI use cases.",[1230,11957,11959],{"id":11958},"export-and-reporting",[14,11960,11961],{},"Export and Reporting",[30,11963,11964,11967],{},[33,11965,11966],{},"Download comprehensive PDF or CSV reports for offline analysis.",[33,11968,11969,11972],{},[630,11970,11971],{},"(Coming Soon)"," Export charts and insights as images for presentations and stakeholder updates.",[71,11974],{},[10,11976,11978],{"id":11977},"how-it-works-a-quick-walkthrough",[14,11979,11980],{},"How It Works: A Quick Walkthrough",[114,11982,11983],{},[33,11984,11985,11988],{},[14,11986,11987],{},"Start on the Home Page",[30,11989,11990],{},[33,11991,11992],{},"A prominent disclaimer highlights that ClusterP&L is a living tool: “The GPU market changes rapidly, and we are continuously updating this resource.”",[18,11994,11995],{},[283,11996],{"alt":285,"src":11997},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fdisclaimer.webp",[114,11999,12000],{"start":673},[33,12001,12002,12005],{},[14,12003,12004],{},"Configure Your Scenario",[30,12006,12007,12010],{},[33,12008,12009],{},"Select a preset (e.g., “AI Training Cluster”) or build your configuration from scratch.",[33,12011,12012],{},"Adjust variables, system count, contract length, utilization, power cost, and more, to see instant ROI and payback data.",[18,12014,12015],{},[283,12016],{"alt":285,"src":12017},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fconfigure.webp",[114,12019,12020],{"start":676},[33,12021,12022,12025],{},[14,12023,12024],{},"Analyze Results",[30,12026,12027,12030,12033],{},[33,12028,12029],{},"Access a detailed financial breakdown of CAPEX, OPEX, revenue forecasts, and cash flow.",[33,12031,12032],{},"Dive into advanced metrics like NPV, IRR, and Monte Carlo simulations to evaluate performance under different market conditions.",[33,12034,12035],{},"Compare multiple scenarios side by side with ease.",[18,12037,12038],{},[283,12039],{"alt":285,"src":12040},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fanalysis.webp",[114,12042,12043],{"start":8590},[33,12044,12045,12048],{},[14,12046,12047],{},"Export and Share",[30,12049,12050],{},[33,12051,12052],{},"Export the final projections as a PDF or CSV for immediate distribution.",[71,12054],{},[10,12056,12058],{"id":12057},"example-ai-training-cluster",[14,12059,12060],{},"Example: AI Training Cluster",[18,12062,12063],{},"Imagine evaluating a realistic scenario using the “AI Training Cluster” preset:",[30,12065,12066,12069,12072,12075,12078],{},[33,12067,12068],{},"128 H200 systems",[33,12070,12071],{},"1-year contract",[33,12073,12074],{},"85% utilization",[33,12076,12077],{},"Power cost: $0.019\u002FkWh",[33,12079,12080],{},"Optimized for large language model training",[18,12082,12083],{},[283,12084],{"alt":285,"src":12085},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fexampleaitraining.webp",[18,12087,12088],{},"In just a few clicks, you’ll see the entire financial projection, including profit, ROI, and payback period. From there, you can adjust parameters, compare scenarios, and export results to share with stakeholders or your broader team.",[71,12090],{},[10,12092,12094],{"id":12093},"disclaimers-transparency",[14,12095,12096],{},"Disclaimers & Transparency",[18,12098,12099],{},"ClusterP&L is, and will always be, a work in progress. The GPU market moves fast, and we regularly refine the tool to capture new pricing, hardware releases, and emerging best practices. All results are estimates and should not be construed as financial advice.",[71,12101],{},[10,12103,12104],{"id":602},[14,12105,605],{},[30,12107,12108,12114,12120,12126,12132],{},[33,12109,12110,12113],{},[14,12111,12112],{},"Market Benchmarking Dashboard",": Assess your configuration relative to market averages and top-tier implementations.",[33,12115,12116,12119],{},[14,12117,12118],{},"More Templates",": Targeted solutions for LLM workloads, advanced parallelism strategies, and beyond.",[33,12121,12122,12125],{},[14,12123,12124],{},"Integrated Networking Models",": Build in financial projections for InfiniBand and Ethernet infrastructure to deliver a holistic ROI view.",[33,12127,12128,12131],{},[14,12129,12130],{},"Image Export",": Convert charts and analytics into ready-made images for fast, professional presentations.",[33,12133,12134,12137],{},[14,12135,12136],{},"Enhanced Collaboration",": Share scenarios and insights with colleagues or external stakeholders more seamlessly.",[71,12139],{},[10,12141,12143],{"id":12142},"about-eliovp-bv",[14,12144,12145],{},"About Eliovp BV",[18,12147,12148],{},"Eliovp BV is an industry leader in GPU cluster deployment and optimization, with a proven track record worldwide. Our team partners with clients in AI, finance, and research to help them maximize infrastructure ROI.",[30,12150,12151,12157,12163,12169],{},[33,12152,12153,12156],{},[14,12154,12155],{},"10+ Large-Scale GPU Deployments",": Spanning multiple industries and use cases",[33,12158,12159,12162],{},[14,12160,12161],{},"Diverse Client Base",": Financial, AI, and research-focused firms",[33,12164,12165,12168],{},[14,12166,12167],{},"On-Prem & Cloud Expertise",": Flexibility to deliver value in any environment",[33,12170,12171,12174],{},[14,12172,12173],{},"Advanced Networking Know-How",": Expert guidance in InfiniBand and Ethernet for high-bandwidth, low-latency cluster designs",[18,12176,12177,12178,12183],{},"Visit ",[163,12179,12182],{"href":12180,"rel":12181},"https:\u002F\u002Feliovp.com",[167],"eliovp.com"," to learn more.",[71,12185],{},[288,12187,12189],{"id":12188},"try-clusterpl-today",[14,12190,12191],{},"Try ClusterP&L Today",[30,12193,12194,12204,12215],{},[33,12195,12196,7794,12199],{},[14,12197,12198],{},"Launch the Calculator",[163,12200,12203],{"href":12201,"rel":12202},"https:\u002F\u002Fcalculator.eliovp.com",[167],"Calculator",[33,12205,12206,12209,12210,46],{},[14,12207,12208],{},"Contact Eliovp BV",": For a custom consultation, head to ",[163,12211,12214],{"href":12212,"rel":12213},"https:\u002F\u002Feliovp.com\u002Fcontact",[167],"Eliovp.com",[33,12216,12217,12220],{},[14,12218,12219],{},"Feedback & Features",": We thrive on user input. Let us know how we can further refine ClusterP&L.",{"title":285,"searchDepth":673,"depth":673,"links":12222},[12223,12224,12225,12226,12227,12228,12229,12230],{"id":11790,"depth":676,"text":11793},{"id":11824,"depth":676,"text":11827},{"id":11977,"depth":676,"text":11980},{"id":12057,"depth":676,"text":12060},{"id":12093,"depth":676,"text":12096},{"id":602,"depth":676,"text":605},{"id":12142,"depth":676,"text":12145},{"id":12188,"depth":673,"text":12191},[688,689,1148,3971,955,693,695,966,12232],"pnl calculator","2025-05-03T10:52:22","At Eliovp BV, we’ve spent years on the cutting edge of GPU cluster deployment and optimization across Europe. Our team supports leading organizations in AI, finance, and research, architecting, building, and scaling high-performance infrastructure. Over time, our customers, both newcomers and seasoned adopters, repeatedly asked the same question: “Can you help us build a P&L ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fcomparisonscenarios.webp",{},"https:\u002F\u002Feliovp.com\u002Fclusterpl-empowering-gpu-cluster-investors-with-real-world-financial-insights\u002F","\u002Fblog\u002Fclusterpl-empowering-gpu-cluster-investors-with-real-world-financial-insights",{"title":11768,"description":12234},"clusterpl-empowering-gpu-cluster-investors-with-real-world-financial-insights","blog\u002Fclusterpl-empowering-gpu-cluster-investors-with-real-world-financial-insights","Ps2-THLAoAIkRMGW7z8QL6vLBlqiBHcR5vh7y2v6oMs",{"id":12244,"title":12245,"body":12246,"categories":14157,"date":14159,"description":14160,"extension":699,"image":14161,"meta":14162,"navigation":702,"originalUrl":14163,"path":14164,"seo":14165,"slug":14166,"stem":14167,"__hash__":14168},"blog\u002Fblog\u002Fcranking-out-faster-tokens-for-fewer-dollars-amd-mi300x-vs-nvidia-h200.md","Cranking Out Faster Tokens for Fewer Dollars: AMD MI300X vs. NVIDIA H200",{"type":7,"value":12247,"toc":14147},[12248,12253,12259,12266,12269,12271,12277,12280,12303,12306,12309,12314,12319,12324,12327,12330,12333,12336,12338,12344,12347,12350,12353,12355,12361,12364,12378,12381,12387,12390,12407,12412,12815,12820,12840,12845,12847,12852,12855,13162,13167,13172,13174,13180,13183,13188,13653,13658,13672,13677,13679,13684,13986,13991,13996,14009,14014,14016,14022,14029,14046,14049,14054,14059,14061,14067,14073,14093,14096,14098,14104,14123,14125,14131,14137,14140],[18,12249,12250],{},[14,12251,12252],{},"Qwen3-32B on Paiton + AMD MI300x vs.NVIDIA H200",[10,12254,12256],{"id":12255},"_1-introduction",[14,12257,12258],{},"1. Introduction",[5350,12260,12261],{},[18,12262,12263],{},[630,12264,12265],{},"“While we’re actively training models for local customers, automating and streamlining critical business processes, we still found time to push our Paiton framework to the limit on Qwen3-32B.”",[18,12267,12268],{},"In the competitive realm of LLMs, next-gen hardware like the NVIDIA H200 often steals the headlines. But at a significantly lower price point, our AMD MI300x solution, optimized with Paiton, is emerging as the best of both worlds: better or on-par performance plus a compelling cost-per-million-tokens.",[71,12270],{},[10,12272,12274],{"id":12273},"_2-what-we-tested",[14,12275,12276],{},"2. What We Tested",[18,12278,12279],{},"We locked and loaded the newly released Qwen3-32B model on:",[30,12281,12282,12288,12293,12298],{},[33,12283,12284,12287],{},[14,12285,12286],{},"AMD MI300x"," using older 6.3.1 drivers (not even the latest 6.4!)",[33,12289,12290,12292],{},[14,12291,6102],{}," with the newest drivers\u002Ftoolchains",[33,12294,12295,12297],{},[14,12296,68],{},": Our concurrency + kernel-fusion framework, integrated with vLLM 0.8.4",[33,12299,12300,12302],{},[14,12301,8484],{}," with python3 benchmark_serving.py (various configurations, both with and without the –sharegpt-output-len=256 argument)",[18,12304,12305],{},"We also tested an unoptimized (stock) MI300x setup for reference, but the true star here is Paiton on the MI300x, our secret sauce for next-level throughput.",[18,12307,12308],{},"Below is a quick recap of our typical commands:",[18,12310,12311],{},[14,12312,12313],{},"Without –sharegpt-output-len",[5350,12315,12316],{},[18,12317,12318],{},"python3 benchmark_serving.py--backend vllm --model Qwen\u002FQwen3-32B --dataset-name sharegpt --dataset-path .\u002FShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 32 --random-range-ratio 1.0 --host 0.0.0.0 --port 8888 --percentile-metrics ttft,tpot,itl,e2e",[18,12320,12321],{},[14,12322,12323],{},"With –sharegpt-output-len=256",[18,12325,12326],{},"python3 benchmark_serving.py --backend vllm  --model Qwen\u002FQwen3-32B --dataset-name sharegpt --dataset-path .\u002FShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 32 --random-range-ratio 1.0 --host 0.0.0.0 --port 8000 --percentile-metrics ttft,tpot,itl,e2el --sharegpt-output-len 256",[18,12328,12329],{},"We tested batch sizes 1, 2, 4, 8, 16, 32, 64, 128 in each scenario and used environment variables:",[18,12331,12332],{},"HIP_VISIBLE_DEVICES=1 vllm serve -tp 1 --swap-space 16 --port 8888 --disable-log-requests Qwen\u002FQwen3-32B --num-scheduler-steps 10",[18,12334,12335],{},"Our Paiton optimized model runs incorporate specialized concurrency\u002Fkernels beyond these flags.",[71,12337],{},[10,12339,12341],{"id":12340},"_3-headline-figures-beating-the-h200-again",[14,12342,12343],{},"3. Headline Figures: Beating the H200 Again",[18,12345,12346],{},"Despite the H200 being the “new hotness,” in multiple batch-size scenarios, Paiton + MI300x matches or exceeds H200 performance, and at a lower total hardware cost:",[18,12348,12349],{},"8x H200 system vs. 8x MI300x system**$40,000** in savings with AMD.",[18,12351,12352],{},"That’s not chump change. When you factor in how many tokens you’ll generate over the system’s lifetime, the cost per million tokens dips even further in favor of the MI300x.",[71,12354],{},[10,12356,12358],{"id":12357},"_4-detailed-performance-tables",[14,12359,12360],{},"4. Detailed Performance Tables",[18,12362,12363],{},"We’ll show two sets of data:",[114,12365,12366,12372],{},[33,12367,12368,12371],{},[14,12369,12370],{},"Without"," the –sharegpt-output-len flag",[33,12373,12374,12377],{},[14,12375,12376],{},"With"," –sharegpt-output-len=256",[18,12379,12380],{},"In each set, we provide Throughput (Requests\u002Fs, Output Tokens\u002Fs, Total Tokens\u002Fs) and Latency (TTFT, TPOT, ITL, E2E).",[1230,12382,12384],{"id":12383},"_41-withoutsharegpt-output-len",[14,12385,12386],{},"4.1. Without–sharegpt-output-len",[18,12388,12389],{},"We compared three configurations:",[30,12391,12392,12397,12402],{},[33,12393,12394,12396],{},[14,12395,693],{}," (latest drivers & Torch stack)",[33,12398,12399,12401],{},[14,12400,8616],{}," (older 6.3.1 drivers)",[33,12403,12404,12406],{},[14,12405,68],{}," (our specialized concurrency + kernel fusion)",[18,12408,12409],{},[14,12410,12411],{},"4.1.A. Throughput (Nosharegpt-output-len)",[298,12413,12414,12442],{},[301,12415,12416],{},[304,12417,12418,12422,12427,12432,12437],{},[307,12419,12420],{},[14,12421,10576],{},[307,12423,12424],{},[14,12425,12426],{},"Config",[307,12428,12429],{},[14,12430,12431],{},"Req\u002Fs",[307,12433,12434],{},[14,12435,12436],{},"Out Tok\u002Fs",[307,12438,12439],{},[14,12440,12441],{},"Total Tok\u002Fs",[316,12443,12444,12459,12474,12491,12506,12521,12538,12552,12567,12584,12599,12613,12630,12645,12659,12675,12690,12704,12721,12736,12751,12768,12783,12798],{},[304,12445,12446,12448,12450,12453,12456],{},[321,12447,6802],{},[321,12449,693],{},[321,12451,12452],{},"0.33",[321,12454,12455],{},"39.48",[321,12457,12458],{},"43.46",[304,12460,12461,12463,12466,12468,12471],{},[321,12462,6802],{},[321,12464,12465],{},"MI300x (Stock)",[321,12467,12452],{},[321,12469,12470],{},"39.26",[321,12472,12473],{},"43.22",[304,12475,12476,12478,12480,12483,12486],{},[321,12477,6802],{},[321,12479,68],{},[321,12481,12482],{},"0.43",[321,12484,12485],{},"51.14",[321,12487,12488],{},[14,12489,12490],{},"56.30",[304,12492,12493,12495,12497,12500,12503],{},[321,12494,6819],{},[321,12496,693],{},[321,12498,12499],{},"0.11",[321,12501,12502],{},"48.24",[321,12504,12505],{},"50.19",[304,12507,12508,12510,12512,12515,12518],{},[321,12509,6819],{},[321,12511,12465],{},[321,12513,12514],{},"0.10",[321,12516,12517],{},"45.89",[321,12519,12520],{},"47.75",[304,12522,12523,12525,12527,12530,12533],{},[321,12524,6819],{},[321,12526,68],{},[321,12528,12529],{},"0.13",[321,12531,12532],{},"59.04",[321,12534,12535],{},[14,12536,12537],{},"61.43",[304,12539,12540,12542,12544,12547,12550],{},[321,12541,6836],{},[321,12543,693],{},[321,12545,12546],{},"0.21",[321,12548,12549],{},"67.46",[321,12551,8884],{},[304,12553,12554,12556,12558,12561,12564],{},[321,12555,6836],{},[321,12557,12465],{},[321,12559,12560],{},"0.20",[321,12562,12563],{},"67.23",[321,12565,12566],{},"70.86",[304,12568,12569,12571,12573,12576,12579],{},[321,12570,6836],{},[321,12572,68],{},[321,12574,12575],{},"0.26",[321,12577,12578],{},"86.65",[321,12580,12581],{},[14,12582,12583],{},"91.34",[304,12585,12586,12588,12590,12593,12596],{},[321,12587,6853],{},[321,12589,693],{},[321,12591,12592],{},"0.40",[321,12594,12595],{},"109.04",[321,12597,12598],{},"166.55",[304,12600,12601,12603,12605,12607,12610],{},[321,12602,6853],{},[321,12604,12465],{},[321,12606,12592],{},[321,12608,12609],{},"108.94",[321,12611,12612],{},"166.39",[304,12614,12615,12617,12619,12622,12625],{},[321,12616,6853],{},[321,12618,68],{},[321,12620,12621],{},"0.52",[321,12623,12624],{},"141.50",[321,12626,12627],{},[14,12628,12629],{},"216.13",[304,12631,12632,12634,12636,12639,12642],{},[321,12633,6870],{},[321,12635,693],{},[321,12637,12638],{},"0.78",[321,12640,12641],{},"178.20",[321,12643,12644],{},"333.99",[304,12646,12647,12649,12651,12653,12656],{},[321,12648,6870],{},[321,12650,12465],{},[321,12652,12638],{},[321,12654,12655],{},"178.48",[321,12657,12658],{},"334.51",[304,12660,12661,12663,12665,12667,12670],{},[321,12662,6870],{},[321,12664,68],{},[321,12666,6110],{},[321,12668,12669],{},"229.26",[321,12671,12672],{},[14,12673,12674],{},"429.68",[304,12676,12677,12679,12681,12684,12687],{},[321,12678,349],{},[321,12680,693],{},[321,12682,12683],{},"1.47",[321,12685,12686],{},"329.48",[321,12688,12689],{},"677.57",[304,12691,12692,12694,12696,12698,12701],{},[321,12693,349],{},[321,12695,12465],{},[321,12697,12683],{},[321,12699,12700],{},"329.93",[321,12702,12703],{},"679.47",[304,12705,12706,12708,12710,12713,12716],{},[321,12707,349],{},[321,12709,68],{},[321,12711,12712],{},"1.87",[321,12714,12715],{},"417.91",[321,12717,12718],{},[14,12719,12720],{},"860.65",[304,12722,12723,12725,12727,12730,12733],{},[321,12724,6903],{},[321,12726,693],{},[321,12728,12729],{},"2.69",[321,12731,12732],{},"564.33",[321,12734,12735],{},"1226.38",[304,12737,12738,12740,12742,12745,12748],{},[321,12739,6903],{},[321,12741,12465],{},[321,12743,12744],{},"2.53",[321,12746,12747],{},"535.74",[321,12749,12750],{},"1157.55",[304,12752,12753,12755,12757,12760,12763],{},[321,12754,6903],{},[321,12756,68],{},[321,12758,12759],{},"3.30",[321,12761,12762],{},"699.03",[321,12764,12765],{},[14,12766,12767],{},"1511.26",[304,12769,12770,12772,12774,12777,12780],{},[321,12771,6920],{},[321,12773,693],{},[321,12775,12776],{},"4.68",[321,12778,12779],{},"1021.46",[321,12781,12782],{},"2107.07",[304,12784,12785,12787,12789,12792,12795],{},[321,12786,6920],{},[321,12788,12465],{},[321,12790,12791],{},"4.57",[321,12793,12794],{},"995.06",[321,12796,12797],{},"2056.82",[304,12799,12800,12802,12804,12807,12810],{},[321,12801,6920],{},[321,12803,68],{},[321,12805,12806],{},"5.33",[321,12808,12809],{},"1163.39",[321,12811,12812],{},[14,12813,12814],{},"2401.25",[18,12816,12817,12819],{},[14,12818,6961],{}," (No sharegpt-output-len):",[30,12821,12822,12832,12837],{},[33,12823,12824,12827,12828,12831],{},[14,12825,12826],{},"Paiton + MI300x"," leads at ",[14,12829,12830],{},"all"," batch sizes in terms of total throughput (Requests\u002Fs, Total Tokens\u002Fs).",[33,12833,12834,12836],{},[14,12835,693],{}," and stock MI300x are neck-and-neck until about batch size 16, where they start to diverge slightly, but then Paiton leaps ahead significantly.",[33,12838,12839],{},"Even at the largest batch size (128), Paiton is 15–20% ahead of H200 in total tokens\u002Fs.",[18,12841,12842],{},[283,12843],{"alt":285,"src":12844},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Frequests_no_cap.jpg",[71,12846],{},[18,12848,12849],{},[14,12850,12851],{},"4.1.B. Latency (Nosharegpt-output-len)",[18,12853,12854],{},"Below, we show the mean values for TTFT (Time-to-First-Token), TPOT (Time per Output Token), ITL (Inter-Token Latency?), and E2E (End-to-End). We’ve omitted median and P99 for brevity.",[298,12856,12857,12886],{},[301,12858,12859],{},[304,12860,12861,12865,12869,12873,12877,12881],{},[307,12862,12863],{},[14,12864,10576],{},[307,12866,12867],{},[14,12868,12426],{},[307,12870,12871],{},[14,12872,414],{},[307,12874,12875],{},[14,12876,441],{},[307,12878,12879],{},[14,12880,469],{},[307,12882,12883],{},[14,12884,12885],{},"Mean E2E (ms)",[316,12887,12888,12904,12922,12951,12969,12987,13017,13035,13052,13081,13099,13117,13147],{},[304,12889,12890,12892,12894,12896,12899,12901],{},[321,12891,6802],{},[321,12893,693],{},[321,12895,8884],{},[321,12897,12898],{},"24.94",[321,12900,12898],{},[321,12902,12903],{},"3013.7",[304,12905,12906,12908,12911,12914,12917,12919],{},[321,12907,6802],{},[321,12909,12910],{},"MI300x Stock",[321,12912,12913],{},"96.59",[321,12915,12916],{},"24.86",[321,12918,12916],{},[321,12920,12921],{},"3030.07",[304,12923,12924,12928,12932,12937,12942,12946],{},[321,12925,12926],{},[14,12927,6802],{},[321,12929,12930],{},[14,12931,68],{},[321,12933,12934],{},[14,12935,12936],{},"61.42",[321,12938,12939],{},[14,12940,12941],{},"19.19",[321,12943,12944],{},[14,12945,12941],{},[321,12947,12948],{},[14,12949,12950],{},"2325.87",[304,12952,12953,12955,12957,12960,12963,12966],{},[321,12954,6819],{},[321,12956,693],{},[321,12958,12959],{},"70.30",[321,12961,12962],{},"24.15",[321,12964,12965],{},"23.95",[321,12967,12968],{},"10690.70",[304,12970,12971,12973,12975,12978,12981,12984],{},[321,12972,6819],{},[321,12974,12910],{},[321,12976,12977],{},"95.40",[321,12979,12980],{},"25.26",[321,12982,12983],{},"25.12",[321,12985,12986],{},"11235.03",[304,12988,12989,12993,12997,13002,13007,13012],{},[321,12990,12991],{},[14,12992,6819],{},[321,12994,12995],{},[14,12996,68],{},[321,12998,12999],{},[14,13000,13001],{},"66.48",[321,13003,13004],{},[14,13005,13006],{},"19.38",[321,13008,13009],{},[14,13010,13011],{},"19.46",[321,13013,13014],{},[14,13015,13016],{},"8697.51",[304,13018,13019,13021,13023,13026,13029,13032],{},[321,13020,6836],{},[321,13022,693],{},[321,13024,13025],{},"71.00",[321,13027,13028],{},"25.65",[321,13030,13031],{},"25.46",[321,13033,13034],{},"8409.96",[304,13036,13037,13039,13041,13044,13046,13049],{},[321,13038,6836],{},[321,13040,12910],{},[321,13042,13043],{},"94.67",[321,13045,13028],{},[321,13047,13048],{},"25.49",[321,13050,13051],{},"8442.12",[304,13053,13054,13058,13062,13066,13071,13076],{},[321,13055,13056],{},[14,13057,6836],{},[321,13059,13060],{},[14,13061,68],{},[321,13063,13064],{},[14,13065,12566],{},[321,13067,13068],{},[14,13069,13070],{},"19.69",[321,13072,13073],{},[14,13074,13075],{},"19.65",[321,13077,13078],{},[14,13079,13080],{},"6506.55",[304,13082,13083,13085,13087,13090,13093,13096],{},[321,13084,6853],{},[321,13086,693],{},[321,13088,13089],{},"190.87",[321,13091,13092],{},"26.25",[321,13094,13095],{},"26.00",[321,13097,13098],{},"7283.43",[304,13100,13101,13103,13105,13108,13111,13114],{},[321,13102,6853],{},[321,13104,12910],{},[321,13106,13107],{},"244.19",[321,13109,13110],{},"26.09",[321,13112,13113],{},"25.93",[321,13115,13116],{},"7317.94",[304,13118,13119,13123,13127,13132,13137,13142],{},[321,13120,13121],{},[14,13122,6853],{},[321,13124,13125],{},[14,13126,68],{},[321,13128,13129],{},[14,13130,13131],{},"205.81",[321,13133,13134],{},[14,13135,13136],{},"19.91",[321,13138,13139],{},[14,13140,13141],{},"19.90",[321,13143,13144],{},[14,13145,13146],{},"5634.23",[304,13148,13149,13152,13154,13156,13158,13160],{},[321,13150,13151],{},"…",[321,13153,13151],{},[321,13155,13151],{},[321,13157,13151],{},[321,13159,13151],{},[321,13161,13151],{},[18,13163,13164],{},[630,13165,13166],{},"(Table shortened for readability, but the trend is consistent: Paiton reduces time-to-first-token across small batch sizes and can shave E2E latency by a meaningful margin at mid-range batch sizes.)",[18,13168,13169],{},[283,13170],{"alt":285,"src":13171},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fe2e_no_cap.jpg",[71,13173],{},[1230,13175,13177],{"id":13176},"_42-withsharegpt-output-len256",[14,13178,13179],{},"4.2. With–sharegpt-output-len=256",[18,13181,13182],{},"Now, let’s look at the scenario where we fix the output length to 256 tokens. This often helps concurrency and scheduling because the model no longer deals with variable or uncertain completion lengths.",[18,13184,13185],{},[14,13186,13187],{},"4.2.A. Throughput (Withsharegpt-output-len=256)",[298,13189,13190,13214],{},[301,13191,13192],{},[304,13193,13194,13198,13202,13206,13210],{},[307,13195,13196],{},[14,13197,10576],{},[307,13199,13200],{},[14,13201,12426],{},[307,13203,13204],{},[14,13205,12431],{},[307,13207,13208],{},[14,13209,12436],{},[307,13211,13212],{},[14,13213,12441],{},[316,13215,13216,13231,13245,13270,13285,13299,13324,13339,13353,13378,13393,13408,13433,13448,13463,13488,13503,13518,13543,13558,13573,13598,13623,13638],{},[304,13217,13218,13220,13222,13225,13228],{},[321,13219,6802],{},[321,13221,693],{},[321,13223,13224],{},"0.15",[321,13226,13227],{},"39.32",[321,13229,13230],{},"41.17",[304,13232,13233,13235,13237,13239,13242],{},[321,13234,6802],{},[321,13236,12910],{},[321,13238,13224],{},[321,13240,13241],{},"39.12",[321,13243,13244],{},"40.95",[304,13246,13247,13251,13255,13260,13265],{},[321,13248,13249],{},[14,13250,6802],{},[321,13252,13253],{},[14,13254,68],{},[321,13256,13257],{},[14,13258,13259],{},"0.19",[321,13261,13262],{},[14,13263,13264],{},"49.75",[321,13266,13267],{},[14,13268,13269],{},"52.09",[304,13271,13272,13274,13276,13279,13282],{},[321,13273,6819],{},[321,13275,693],{},[321,13277,13278],{},"0.30",[321,13280,13281],{},"76.36",[321,13283,13284],{},"81.73",[304,13286,13287,13289,13291,13293,13296],{},[321,13288,6819],{},[321,13290,12910],{},[321,13292,13278],{},[321,13294,13295],{},"76.30",[321,13297,13298],{},"81.66",[304,13300,13301,13305,13309,13314,13319],{},[321,13302,13303],{},[14,13304,6819],{},[321,13306,13307],{},[14,13308,68],{},[321,13310,13311],{},[14,13312,13313],{},"0.39",[321,13315,13316],{},[14,13317,13318],{},"99.84",[321,13320,13321],{},[14,13322,13323],{},"106.86",[304,13325,13326,13328,13330,13333,13336],{},[321,13327,6836],{},[321,13329,693],{},[321,13331,13332],{},"0.59",[321,13334,13335],{},"152.27",[321,13337,13338],{},"162.83",[304,13340,13341,13343,13345,13347,13350],{},[321,13342,6836],{},[321,13344,12910],{},[321,13346,13332],{},[321,13348,13349],{},"151.23",[321,13351,13352],{},"161.72",[304,13354,13355,13359,13363,13368,13373],{},[321,13356,13357],{},[14,13358,6836],{},[321,13360,13361],{},[14,13362,68],{},[321,13364,13365],{},[14,13366,13367],{},"0.76",[321,13369,13370],{},[14,13371,13372],{},"194.67",[321,13374,13375],{},[14,13376,13377],{},"208.17",[304,13379,13380,13382,13384,13387,13390],{},[321,13381,6853],{},[321,13383,693],{},[321,13385,13386],{},"1.14",[321,13388,13389],{},"291.00",[321,13391,13392],{},"455.11",[304,13394,13395,13397,13399,13402,13405],{},[321,13396,6853],{},[321,13398,12910],{},[321,13400,13401],{},"1.13",[321,13403,13404],{},"289.74",[321,13406,13407],{},"453.14",[304,13409,13410,13414,13418,13423,13428],{},[321,13411,13412],{},[14,13413,6853],{},[321,13415,13416],{},[14,13417,68],{},[321,13419,13420],{},[14,13421,13422],{},"1.44",[321,13424,13425],{},[14,13426,13427],{},"369.11",[321,13429,13430],{},[14,13431,13432],{},"577.28",[304,13434,13435,13437,13439,13442,13445],{},[321,13436,6870],{},[321,13438,693],{},[321,13440,13441],{},"2.08",[321,13443,13444],{},"531.34",[321,13446,13447],{},"996.58",[304,13449,13450,13452,13454,13457,13460],{},[321,13451,6870],{},[321,13453,12910],{},[321,13455,13456],{},"2.13",[321,13458,13459],{},"545.21",[321,13461,13462],{},"1022.13",[304,13464,13465,13469,13473,13478,13483],{},[321,13466,13467],{},[14,13468,6870],{},[321,13470,13471],{},[14,13472,68],{},[321,13474,13475],{},[14,13476,13477],{},"2.63",[321,13479,13480],{},[14,13481,13482],{},"673.64",[321,13484,13485],{},[14,13486,13487],{},"1262.91",[304,13489,13490,13492,13494,13497,13500],{},[321,13491,349],{},[321,13493,693],{},[321,13495,13496],{},"3.75",[321,13498,13499],{},"959.10",[321,13501,13502],{},"1836.45",[304,13504,13505,13507,13509,13512,13515],{},[321,13506,349],{},[321,13508,12910],{},[321,13510,13511],{},"3.72",[321,13513,13514],{},"951.17",[321,13516,13517],{},"1820.83",[304,13519,13520,13524,13528,13533,13538],{},[321,13521,13522],{},[14,13523,349],{},[321,13525,13526],{},[14,13527,68],{},[321,13529,13530],{},[14,13531,13532],{},"4.35",[321,13534,13535],{},[14,13536,13537],{},"1112.82",[321,13539,13540],{},[14,13541,13542],{},"2130.28",[304,13544,13545,13547,13549,13552,13555],{},[321,13546,6903],{},[321,13548,693],{},[321,13550,13551],{},"6.35",[321,13553,13554],{},"1614.17",[321,13556,13557],{},"3186.41",[304,13559,13560,13562,13564,13567,13570],{},[321,13561,6903],{},[321,13563,12910],{},[321,13565,13566],{},"5.19",[321,13568,13569],{},"1328.16",[321,13571,13572],{},"2613.51",[304,13574,13575,13579,13583,13588,13593],{},[321,13576,13577],{},[14,13578,6903],{},[321,13580,13581],{},[14,13582,68],{},[321,13584,13585],{},[14,13586,13587],{},"6.49",[321,13589,13590],{},[14,13591,13592],{},"1661.29",[321,13594,13595],{},[14,13596,13597],{},"3269.05",[304,13599,13600,13604,13608,13613,13618],{},[321,13601,13602],{},[14,13603,6920],{},[321,13605,13606],{},[14,13607,693],{},[321,13609,13610],{},[14,13611,13612],{},"9.21",[321,13614,13615],{},[14,13616,13617],{},"2356.92",[321,13619,13620],{},[14,13621,13622],{},"4547.69",[304,13624,13625,13627,13629,13632,13635],{},[321,13626,6920],{},[321,13628,12910],{},[321,13630,13631],{},"8.18",[321,13633,13634],{},"2086.62",[321,13636,13637],{},"4032.20",[304,13639,13640,13642,13644,13647,13650],{},[321,13641,6920],{},[321,13643,68],{},[321,13645,13646],{},"8.94",[321,13648,13649],{},"2278.53",[321,13651,13652],{},"4405.65",[18,13654,13655,13657],{},[14,13656,6961],{}," (With sharegpt-output-len=256):",[30,13659,13660,13663],{},[33,13661,13662],{},"Performance jumps up across the board because the model can more predictably schedule token generation.",[33,13664,13665,13667,13668,13671],{},[14,13666,68],{}," once again extends the MI300x lead at most batch sizes. By batch=64 and 128, H200 and Paiton are very close, but sometimes H200 does a ",[630,13669,13670],{},"slightly"," higher request throughput. Even then, Paiton’s total tokens\u002Fs is in the same ballpark or surpasses it.",[18,13673,13674],{},[283,13675],{"alt":285,"src":13676},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Frequests_256_cap.jpg",[71,13678],{},[18,13680,13681],{},[14,13682,13683],{},"4.2.B. Latency (Withsharegpt-output-len=256)",[298,13685,13686,13718],{},[301,13687,13688],{},[304,13689,13690,13694,13698,13703,13708,13713],{},[307,13691,13692],{},[14,13693,10576],{},[307,13695,13696],{},[14,13697,12426],{},[307,13699,13700],{},[14,13701,13702],{},"TTFT (ms)",[307,13704,13705],{},[14,13706,13707],{},"TPOT (ms)",[307,13709,13710],{},[14,13711,13712],{},"ITL (ms)",[307,13714,13715],{},[14,13716,13717],{},"E2E (ms)",[316,13719,13720,13737,13754,13783,13800,13817,13846,13863,13880,13909,13926,13943,13972],{},[304,13721,13722,13724,13726,13729,13732,13734],{},[321,13723,6802],{},[321,13725,693],{},[321,13727,13728],{},"143.90",[321,13730,13731],{},"24.96",[321,13733,13731],{},[321,13735,13736],{},"6509.40",[304,13738,13739,13741,13743,13746,13749,13751],{},[321,13740,6802],{},[321,13742,12910],{},[321,13744,13745],{},"176.79",[321,13747,13748],{},"24.97",[321,13750,13748],{},[321,13752,13753],{},"6543.64",[304,13755,13756,13760,13764,13769,13774,13778],{},[321,13757,13758],{},[14,13759,6802],{},[321,13761,13762],{},[14,13763,68],{},[321,13765,13766],{},[14,13767,13768],{},"117.14",[321,13770,13771],{},[14,13772,13773],{},"19.71",[321,13775,13776],{},[14,13777,13773],{},[321,13779,13780],{},[14,13781,13782],{},"5144.32",[304,13784,13785,13787,13789,13792,13795,13797],{},[321,13786,6819],{},[321,13788,693],{},[321,13790,13791],{},"145.96",[321,13793,13794],{},"25.72",[321,13796,13794],{},[321,13798,13799],{},"6703.84",[304,13801,13802,13804,13806,13809,13812,13814],{},[321,13803,6819],{},[321,13805,12910],{},[321,13807,13808],{},"171.51",[321,13810,13811],{},"25.64",[321,13813,13811],{},[321,13815,13816],{},"6708.87",[304,13818,13819,13823,13827,13832,13837,13841],{},[321,13820,13821],{},[14,13822,6819],{},[321,13824,13825],{},[14,13826,68],{},[321,13828,13829],{},[14,13830,13831],{},"113.62",[321,13833,13834],{},[14,13835,13836],{},"19.66",[321,13838,13839],{},[14,13840,13836],{},[321,13842,13843],{},[14,13844,13845],{},"5126.54",[304,13847,13848,13850,13852,13855,13858,13860],{},[321,13849,6836],{},[321,13851,693],{},[321,13853,13854],{},"145.09",[321,13856,13857],{},"25.80",[321,13859,13857],{},[321,13861,13862],{},"6723.31",[304,13864,13865,13867,13869,13872,13875,13877],{},[321,13866,6836],{},[321,13868,12910],{},[321,13870,13871],{},"170.48",[321,13873,13874],{},"25.88",[321,13876,13874],{},[321,13878,13879],{},"6768.63",[304,13881,13882,13886,13890,13895,13900,13904],{},[321,13883,13884],{},[14,13885,6836],{},[321,13887,13888],{},[14,13889,68],{},[321,13891,13892],{},[14,13893,13894],{},"117.72",[321,13896,13897],{},[14,13898,13899],{},"20.16",[321,13901,13902],{},[14,13903,13899],{},[321,13905,13906],{},[14,13907,13908],{},"5257.58",[304,13910,13911,13913,13915,13918,13921,13923],{},[321,13912,6853],{},[321,13914,693],{},[321,13916,13917],{},"263.55",[321,13919,13920],{},"26.56",[321,13922,13920],{},[321,13924,13925],{},"7035.76",[304,13927,13928,13930,13932,13935,13938,13940],{},[321,13929,6853],{},[321,13931,12910],{},[321,13933,13934],{},"320.44",[321,13936,13937],{},"26.45",[321,13939,13937],{},[321,13941,13942],{},"7064.58",[304,13944,13945,13949,13953,13958,13963,13967],{},[321,13946,13947],{},[14,13948,6853],{},[321,13950,13951],{},[14,13952,68],{},[321,13954,13955],{},[14,13956,13957],{},"253.04",[321,13959,13960],{},[14,13961,13962],{},"20.75",[321,13964,13965],{},[14,13966,13962],{},[321,13968,13969],{},[14,13970,13971],{},"5544.18",[304,13973,13974,13976,13978,13980,13982,13984],{},[321,13975,13151],{},[321,13977,13151],{},[321,13979,13151],{},[321,13981,13151],{},[321,13983,13151],{},[321,13985,13151],{},[18,13987,13988],{},[630,13989,13990],{},"(Again, showing partial data for brevity.)",[18,13992,13993,526],{},[14,13994,13995],{},"Latency Takeaways",[30,13997,13998,14003,14006],{},[33,13999,14000,14002],{},[14,14001,68],{}," consistently reduces Time-to-First-Token (TTFT) across small batch sizes.",[33,14004,14005],{},"Mean E2E Latency sees a noticeable drop with Paiton vs. stock MI300x or H200, particularly in the 1–16 batch range.",[33,14007,14008],{},"At higher batch sizes, latencies naturally scale up, but Paiton helps keep them in check.",[18,14010,14011],{},[283,14012],{"alt":285,"src":14013},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fe2e_256_cap.jpg",[71,14015],{},[10,14017,14019],{"id":14018},"_5-cost-per-million-tokens-real-roi-for-paiton-mi300x",[14,14020,14021],{},"5. Cost Per Million Tokens: Real ROI for Paiton + MI300x",[18,14023,14024,14025,14028],{},"From a purely corporate perspective, the cost delta between the 8-GPU H200 system and the 8-GPU MI300x system, ",[14,14026,14027],{},"$40,000",", is substantial. When normalized by total tokens processed (e.g., over the system’s multi-year lifecycle), the math is in AMD’s favor:",[30,14030,14031,14036],{},[33,14032,14033,14035],{},[14,14034,693],{}," might produce marginally higher throughput at extremely large batch sizes, but requires a bigger cash outlay.",[33,14037,14038,14041,14042,14045],{},[14,14039,14040],{},"MI300x + Paiton"," meets or beats the H200 at a much lower hardware price. Thus your ",[14,14043,14044],{},"$ \u002F million tokens"," cost can be significantly lower.",[18,14047,14048],{},"In large-scale inference scenarios (think: billions or trillions of tokens served monthly), that price gap pays off quickly.",[18,14050,14051],{},[283,14052],{"alt":285,"src":14053},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fcost_per_million_tokens_simple.jpg",[18,14055,14056],{},[630,14057,14058],{},"“Cost per million tokens is calculated by taking each system’s approximate hardware cost and dividing by its sustained token throughput at a moderate concurrency level. The exact figure may vary in real-world deployments based on your usage patterns, operational overhead, and chosen batch sizes, but these numbers provide a clear illustration of the relative cost efficiency between the H200 and MI300x (Paiton) solutions.”",[71,14060],{},[10,14062,14064],{"id":14063},"_6-paiton-the-game-changer",[14,14065,14066],{},"6. Paiton: The Game-Changer",[18,14068,14069,14070,14072],{},"While raw AMD silicon is impressive, ",[14,14071,68],{}," is our in-house software layer that optimizes concurrency, kernel launches, and memory usage:",[30,14074,14075,14081,14087],{},[33,14076,14077,14080],{},[14,14078,14079],{},"Kernel Fusion",": Minimizes overhead by merging operations.",[33,14082,14083,14086],{},[14,14084,14085],{},"Adaptive Concurrency",": Exploits the GPU’s HBM memory to handle multi-request bursts.",[33,14088,14089,14092],{},[14,14090,14091],{},"Robust Under Older Drivers",": Even with 6.3.1, we’re beating the H200. Expect even bigger leaps when we move to 6.4+.",[18,14094,14095],{},"In nearly every table above, you’ll notice how “MI300x + Paiton” outpaces “H200” and “Stock MI300x.” That’s not purely hardware; it’s synergy between Paiton and AMD’s robust memory architecture.",[71,14097],{},[10,14099,14101],{"id":14100},"_7-bottom-line-takeaways",[14,14102,14103],{},"7. Bottom-Line Takeaways",[114,14105,14106,14111,14117],{},[33,14107,14108,14110],{},[14,14109,12826],{}," outruns (or meets) the NVIDIA H200 on Qwen3-32B in small-to-mid batch sizes and often holds up well at larger sizes too.",[33,14112,14113,14116],{},[14,14114,14115],{},"$40K Cheaper for an 8-GPU System",": That’s a real difference in capital expenditure, culminating in a better cost-per-million-tokens in many real-world scenarios.",[33,14118,14119,14122],{},[14,14120,14121],{},"Even More Performance Gains Ahead",": New AMD drivers and expanded concurrency in Paiton will keep increasing the performance gap.",[71,14124],{},[10,14126,14128],{"id":14127},"_8-looking-forward-magic-on-the-horizon",[14,14129,14130],{},"8. Looking Forward: Magic on the Horizon",[18,14132,14133,14134],{},"*“Keep an eye on us.*",[630,14135,14136],{},"We’ve got some magical stuff brewing for FP8 and then some, stay tuned!”",[18,14138,14139],{},"We’re not just resting on these results. We’ll continue refining Paiton with advanced quantization strategies, more deep optimization techniques, and ongoing work for AMD’s MI300x platform. As more enterprises opt for large-scale in-house LLM deployments, the synergy of AMD MI300x hardware plus Paiton stands ready to slash costs while raising performance.",[18,14141,14142],{},[14,14143,14144,14145],{},"Thanks for reading, and feel free to reach out if you want more data, a private demo, or a deep dive into our concurrency model.– ",[630,14146,11004],{},{"title":285,"searchDepth":673,"depth":673,"links":14148},[14149,14150,14151,14152,14153,14154,14155,14156],{"id":12255,"depth":676,"text":12258},{"id":12273,"depth":676,"text":12276},{"id":12340,"depth":676,"text":12343},{"id":12357,"depth":676,"text":12360},{"id":14018,"depth":676,"text":14021},{"id":14063,"depth":676,"text":14066},{"id":14100,"depth":676,"text":14103},{"id":14127,"depth":676,"text":14130},[688,689,68,9658,690,693,14158,966,68,696],"MI300","2025-05-02T21:10:30","Qwen3-32B on Paiton + AMD MI300x vs.NVIDIA H200 1. Introduction “While we’re actively training models for local customers, automating and streamlining critical business processes, we still found time to push our Paiton framework to the limit on Qwen3-32B.” In the competitive realm of LLMs, next-gen hardware like the NVIDIA H200 often steals the headlines. But ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002F3ac59a73-2466-4422-b7e5-ef2e4a8ca58e.webp",{},"https:\u002F\u002Feliovp.com\u002Fcranking-out-faster-tokens-for-fewer-dollars-amd-mi300x-vs-nvidia-h200\u002F","\u002Fblog\u002Fcranking-out-faster-tokens-for-fewer-dollars-amd-mi300x-vs-nvidia-h200",{"title":12245,"description":14160},"cranking-out-faster-tokens-for-fewer-dollars-amd-mi300x-vs-nvidia-h200","blog\u002Fcranking-out-faster-tokens-for-fewer-dollars-amd-mi300x-vs-nvidia-h200","x10RPNXDyE3YpaB_dhvXVzKKsMn38W8EZNzDgekSgQU",{"id":14170,"title":14171,"body":14172,"categories":14968,"date":14974,"description":14975,"extension":699,"image":14397,"meta":14976,"navigation":702,"originalUrl":14977,"path":14978,"seo":14979,"slug":14980,"stem":14981,"__hash__":14982},"blog\u002Fblog\u002Fpower-meets-precision-high-density-modular-data-center-for-nvidia-nvl-deployments-1-2-mw.md","Power Meets Precision: High-Density Modular Data Center for NVIDIA NVL Deployments (1–2 MW)",{"type":7,"value":14173,"toc":14939},[14174,14180,14183,14186,14189,14191,14197,14200,14206,14245,14251,14282,14284,14290,14293,14390,14393,14398,14400,14406,14409,14415,14421,14424,14430,14433,14439,14442,14448,14451,14457,14460,14465,14476,14478,14484,14487,14493,14499,14502,14508,14511,14517,14607,14613,14616,14622,14625,14665,14671,14696,14699,14701,14707,14710,14716,14719,14725,14728,14734,14737,14739,14745,14770,14772,14778,14781,14784,14786,14792,14795,14801,14869,14875,14889,14892,14903,14906,14911,14913,14919,14922,14924,14930,14933,14936],[288,14175,14177],{"id":14176},"purpose-built-high-density-infrastructure-for-blackwell-class-ai-workloads",[14,14178,14179],{},"Purpose-Built High-Density Infrastructure for Blackwell-Class AI Workloads",[18,14181,14182],{},"At Eliovp, we’re engineering a new class of AI infrastructure. Our advanced modular platform is built to also support NVIDIA’s cutting-edge NVL architecture, from the efficient NVL4 to the ultra-scale NVL72, enabling deployments that range from distributed edge inference to full-stack model training at hyperscale.",[18,14184,14185],{},"Designed to meet the increasing thermal and power demands of next-generation GPU workloads, Eliovp’s modular systems deliver 1–2 MW of compute power per unit in a compact, prefabricated footprint. These solutions are scalable, efficient, and deployment-ready in under 6 months.",[18,14187,14188],{},"Whether you’re deploying 1 or 2 megawatts of GPU compute, our modular architecture delivers the scalability, efficiency, and resilience your AI workloads demand, all within a 12–17m footprint (depending on level of redundancy\u002Fmaintainability).",[71,14190],{},[288,14192,14194],{"id":14193},"built-for-nvidia-nvl-peak-performance-in-a-compact-footprint",[14,14195,14196],{},"Built for NVIDIA NVL: Peak Performance in a Compact Footprint",[18,14198,14199],{},"The NVIDIA NVL family is engineered for next-gen AI: from training massive language models like GPT-4 to powering real-time inference in edge environments. Here’s a breakdown of the NVL configurations our modular centers are optimized for:",[10,14201,14203],{"id":14202},"nvidia-nvl4-coming-soon",[14,14204,14205],{},"NVIDIA NVL4 (Coming Soon)",[30,14207,14208,14214,14220,14226],{},[33,14209,14210,14213],{},[14,14211,14212],{},"Configuration:"," 4× Blackwell GPUs (B200-class), 2× Grace CPUs via NVLink",[33,14215,14216,14219],{},[14,14217,14218],{},"Power Draw:"," ~6.6 kW",[33,14221,14222,14225],{},[14,14223,14224],{},"Use Case:"," Supercomputing, scientific simulations, distributed inference",[33,14227,14228,14231],{},[14,14229,14230],{},"Key Benefits:",[30,14232,14233,14236,14239,14242],{},[33,14234,14235],{},"Edge-ready with a smaller power footprint",[33,14237,14238],{},"Low-latency NVLink GPU-to-GPU",[33,14240,14241],{},"Ideal for micro-modular deployments",[33,14243,14244],{},"NVSwitch + NVLink, BlueField connectivity",[10,14246,14248],{"id":14247},"nvidia-nvl72",[14,14249,14250],{},"NVIDIA NVL72",[30,14252,14253,14258,14263,14268],{},[33,14254,14255,14257],{},[14,14256,14212],{}," 144 Blackwell GPUs, 72 Hopper CPUs",[33,14259,14260,14262],{},[14,14261,14218],{}," ~120 kW per rack",[33,14264,14265,14267],{},[14,14266,14224],{}," Full-scale LLM training (GPT-4, Llama 3\u002F4)",[33,14269,14270,14272],{},[14,14271,14230],{},[30,14273,14274,14277,14280],{},[33,14275,14276],{},"Designed for massive parallel compute",[33,14278,14279],{},"Ultra-fast interconnects for model training at scale",[33,14281,14244],{},[71,14283],{},[288,14285,14287],{"id":14286},"technical-architecture-overview",[14,14288,14289],{},"Technical Architecture Overview",[18,14291,14292],{},"Our modular solution isn’t just a shipping container with racks. It’s a complete data ecosystem, pre-engineered, prefabricated, and globally deployable.",[298,14294,14295,14309],{},[301,14296,14297],{},[304,14298,14299,14304],{},[307,14300,14301],{},[14,14302,14303],{},"Parameter",[307,14305,14306],{},[14,14307,14308],{},"Value \u002F Specification",[316,14310,14311,14319,14327,14335,14343,14351,14359,14367,14375,14382],{},[304,14312,14313,14316],{},[321,14314,14315],{},"IT Capacity",[321,14317,14318],{},"6-10 high-density DLC racks per module (2-4 Additional racks for storage and network)",[304,14320,14321,14324],{},[321,14322,14323],{},"Rack-Level Power Envelope",[321,14325,14326],{},"Nominal: 80 kW, Peak: 150 kW per rack",[304,14328,14329,14332],{},[321,14330,14331],{},"Total Module IT Load",[321,14333,14334],{},"1.0 MW operational average, scalable to 1.1–2.2 MW per unit",[304,14336,14337,14340],{},[321,14338,14339],{},"Cooling Design",[321,14341,14342],{},"RDHx\u002FDLC hybrid, N+1 redundant Drycoolers\u002FChillers (Glycol PG25)",[304,14344,14345,14348],{},[321,14346,14347],{},"Power Topology",[321,14349,14350],{},"N+1 UPS per POD, integrated (expandable to 2N)",[304,14352,14353,14356],{},[321,14354,14355],{},"Redundancy Philosophy",[321,14357,14358],{},"Full component-level redundancy (RDHx, chillers, pumps)",[304,14360,14361,14364],{},[321,14362,14363],{},"Glycol Solution",[321,14365,14366],{},"PG25 (≤40%) – anti-freeze, anti-biofilm, low-maintenance",[304,14368,14369,14372],{},[321,14370,14371],{},"Deployment Time",[321,14373,14374],{},"18–24 weeks (spec to install)",[304,14376,14377,14379],{},[321,14378,4158],{},[321,14380,14381],{},"TIER III-ready",[304,14383,14384,14387],{},[321,14385,14386],{},"Rack Density",[321,14388,14389],{},"Up to 130 kW\u002Frack Nominal load.",[18,14391,14392],{},"Each Eliovp unit utilizes a vendor-agnostic, standards-compliant design paradigm, allowing rapid integration of NVIDIA Blackwell-based NVL systems while retaining flexibility for AMD MI300 or future hybrid AI compute environments.",[18,14394,14395],{},[283,14396],{"alt":285,"src":14397},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Feliovp_critical-1mw-pod_rev-2_transparent.webp",[71,14399],{},[288,14401,14403],{"id":14402},"rack-and-cooling-system-integration",[14,14404,14405],{},"Rack and Cooling System Integration",[18,14407,14408],{},"Eliovp’s modular architecture integrates high-density direct liquid to chip cooling (DLC) loops in combination with Rear Door Heat Exchangers (RDHx), enabling precise thermal regulation, reduced loop complexity, and scalable coolant delivery across high-power compute racks.",[10,14410,14412],{"id":14411},"key-cooling-specifications",[14,14413,14414],{},"Key Cooling Specifications",[1230,14416,14418],{"id":14417},"chiller-configuration",[14,14419,14420],{},"Chiller Configuration",[18,14422,14423],{},"N+1 redundant drycoolers or chiller sets with integrated glycol buffer vessels and Grundfos variable-speed pump sets, ensuring stable thermal delivery and hydraulic balancing under dynamic compute loads.",[1230,14425,14427],{"id":14426},"direct-liquid-to-chip-cooling-dlc",[14,14428,14429],{},"Direct-Liquid-to-Chip Cooling (DLC)",[18,14431,14432],{},"High-efficiency cold plate loops directly interfaced with GPU\u002FCPU die surfaces via the CDU loop. Designed for thermal loads exceeding 100 kW\u002Frack, with low ∆T and minimal thermal resistance.",[1230,14434,14436],{"id":14435},"rear-door-heat-exchanger-rdhx-strategy",[14,14437,14438],{},"Rear Door Heat Exchanger (RDHx) Strategy",[18,14440,14441],{},"Rack-mounted liquid-to-air RDHx units extract residual heat (5% for the NVL4 and around 18% for de NVL 72) post-DLC loop. Each unit is designed with passive and active failover capability to maintain airflow and thermal extraction in the event of a localized failure.",[1230,14443,14445],{"id":14444},"coolant-composition",[14,14446,14447],{},"Coolant Composition",[18,14449,14450],{},"Closed-loop circulation of PG25 glycol solution (up to 40%) provides freeze protection, microbial resistance, and minimal fouling across the thermal exchange surfaces.",[1230,14452,14454],{"id":14453},"optional-adiabatic-dry-coolers",[14,14455,14456],{},"Optional Adiabatic Dry Coolers",[18,14458,14459],{},"For deployments in favorable climates, traditional mechanical chillers can be partially or fully replaced with adiabatic dry cooling arrays, significantly reducing PUE and OPEX during shoulder seasons.",[18,14461,14462],{},[14,14463,14464],{},"Built-in Advantages",[30,14466,14467,14470,14473],{},[33,14468,14469],{},"Liquid cooling + RDHx for optimal thermals",[33,14471,14472],{},"Side-by-side container format for site flexibility",[33,14474,14475],{},"Integrated fire suppression, telemetry, and access control",[71,14477],{},[288,14479,14481],{"id":14480},"scalable-redundant-power-distribution",[14,14482,14483],{},"Scalable, Redundant Power Distribution",[18,14485,14486],{},"Eliovp’s power infrastructure is tailored for high-density GPU deployments. Each 1000 kW POD integrates dual power feeds (A and B), physically housed within a single electrical container featuring an internal partition for fault isolation and maintenance access. This modularized format supports fast deployment, high reliability, and future scalability to 2.2 MW through busbar augmentation and utility-side expansion.",[10,14488,14490],{"id":14489},"integrated-electrical-container-1mw-dual-feed-pod",[14,14491,14492],{},"Integrated Electrical Container (1MW Dual Feed POD)",[1230,14494,14496],{"id":14495},"ups-architecture",[14,14497,14498],{},"UPS Architecture",[18,14500,14501],{},"Redundant 2× modular UPS blocks in decentralized parallel architecture (DPA), mounted within a 40ft container. Both A and B feeds share the enclosure, separated by a central fire-rated wall for physical and electrical segregation.",[1230,14503,14505],{"id":14504},"busbar-distribution",[14,14506,14507],{},"Busbar Distribution",[18,14509,14510],{},"IT load and cooling infrastructure are powered via segmented 1600 A copper busbars, with integrated tap-off units at 250 A and 125 A for sub-distribution.",[1230,14512,14514],{"id":14513},"breaker-configuration-1000-kw",[14,14515,14516],{},"Breaker Configuration (1000 kW)",[298,14518,14519,14532],{},[301,14520,14521],{},[304,14522,14523,14526,14529],{},[307,14524,14525],{},"Function",[307,14527,14528],{},"Rating",[307,14530,14531],{},"Configuration",[316,14533,14534,14545,14555,14566,14576,14586,14596],{},[304,14535,14536,14539,14542],{},[321,14537,14538],{},"Transformer Input",[321,14540,14541],{},"2000 A",[321,14543,14544],{},"Dual feed",[304,14546,14547,14550,14552],{},[321,14548,14549],{},"Generator Breaker",[321,14551,14541],{},[321,14553,14554],{},"A+B redundancy",[304,14556,14557,14560,14563],{},[321,14558,14559],{},"UPS Output Breaker",[321,14561,14562],{},"1600 A",[321,14564,14565],{},"Integrated modular UPS x2",[304,14567,14568,14571,14573],{},[321,14569,14570],{},"Maintenance Bypass",[321,14572,14541],{},[321,14574,14575],{},"Fully rated bypass capability",[304,14577,14578,14581,14583],{},[321,14579,14580],{},"IT Load Distribution",[321,14582,14562],{},[321,14584,14585],{},"+1×250 A, +1×125 A",[304,14587,14588,14591,14594],{},[321,14589,14590],{},"Cooling Distribution",[321,14592,14593],{},"400 A",[321,14595,14585],{},[304,14597,14598,14601,14604],{},[321,14599,14600],{},"Total Switchgear Width",[321,14602,14603],{},"5.4 m",[321,14605,14606],{},"Including full-length LV panels",[1230,14608,14610],{"id":14609},"fire-safety-integration",[14,14611,14612],{},"Fire Safety Integration",[18,14614,14615],{},"Each container includes full VESDA smoke detection, argon gas suppression, and local emergency shutoff interfaces.",[10,14617,14619],{"id":14618},"emergency-power-generation-nsa-containerized-system",[14,14620,14621],{},"Emergency Power Generation (NSA Containerized System)",[18,14623,14624],{},"To ensure grid-agnostic, uninterrupted operation, each 1 MW POD is backed by 2N containerized gensets systems that are prime rated to the maximum load of the IT and cooling equipment with the following specs:",[30,14626,14627,14633,14639,14645],{},[33,14628,14629,14632],{},[14,14630,14631],{},"Prime Power:"," 1224 kVA \u002F 979 kW",[33,14634,14635,14638],{},[14,14636,14637],{},"Standby Power:"," 1320 kVA (continuously running capacity)",[33,14640,14641,14644],{},[14,14642,14643],{},"Genset:"," Baudouin 16M33G1320\u002F5 engine + Xingnuo XN6F alternator",[33,14646,14647,14650,14651],{},[14,14648,14649],{},"Container:"," 40ft high-cube with:\n",[30,14652,14653,14656,14659,14662],{},[33,14654,14655],{},"Integrated sound dampening up to 80 dB(A) @1m",[33,14657,14658],{},"Internal fire detection and access security",[33,14660,14661],{},"1500-liter integrated fuel tank (48 hrs supply)",[33,14663,14664],{},"Dry Weight: 11,183 kg",[10,14666,14668],{"id":14667},"scalability-path",[14,14669,14670],{},"Scalability Path",[30,14672,14673,14679],{},[33,14674,14675,14678],{},[14,14676,14677],{},"Base Configuration:"," 1.0 MW per POD (dual A+B feed)",[33,14680,14681,14684,14685],{},[14,14682,14683],{},"Scalable To:"," 2.2 MW per module via:\n",[30,14686,14687,14690,14693],{},[33,14688,14689],{},"Busbar capacity upgrade",[33,14691,14692],{},"Additional chiller\u002FUPS capacity",[33,14694,14695],{},"Genset pairing with 1640 kW (2×2000 kVA) NSA modules",[18,14697,14698],{},"All power delivery components are fully compliant with IEC 61439, and configured for integration into Tier III\u002FIV-aligned operational topologies (including 2N-ready configurations for mission-critical deployments).",[71,14700],{},[288,14702,14704],{"id":14703},"why-tier-iii-matters-enterprise-grade-uptime",[14,14705,14706],{},"Why TIER III Matters: Enterprise-Grade Uptime",[18,14708,14709],{},"We’ve designed our system to meet Uptime Institute’s TIER III standards, making it ideal for mission-critical workloads where every second counts.",[10,14711,14713],{"id":14712},"concurrent-maintainability",[14,14714,14715],{},"Concurrent Maintainability",[18,14717,14718],{},"Perform maintenance without downtime. Power, cooling, and connectivity systems are all independently serviceable.",[10,14720,14722],{"id":14721},"redundant-paths-n1",[14,14723,14724],{},"Redundant Paths (N+1)",[18,14726,14727],{},"Every critical system has at least one backup, ensuring resilience and availability.",[10,14729,14731],{"id":14730},"uptime-sla-99982",[14,14732,14733],{},"Uptime SLA: 99.982%",[18,14735,14736],{},"That’s less than 1.6 hours of downtime per year. Perfect for AI-powered enterprises where reliability is everything.",[71,14738],{},[288,14740,14742],{"id":14741},"fire-safety-maintenance-and-environmental-controls",[14,14743,14744],{},"Fire Safety, Maintenance, and Environmental Controls",[30,14746,14747,14750,14753,14767],{},[33,14748,14749],{},"Fire Detection\u002FSuppression: Dual-gas Argonite system with continuous VESDA laser detection",[33,14751,14752],{},"Environmental Monitoring: Leak detection, pressure sensors, temperature\u002Fflow telemetry",[33,14754,14755,14756],{},"Maintenance Schedule:\n",[30,14757,14758,14761,14764],{},[33,14759,14760],{},"UPS checks: Annually",[33,14762,14763],{},"Chillers\u002Ffilters: Biannually to quarterly",[33,14765,14766],{},"Transformer\u002FGenerator: 3-year cycle with monthly ops check",[33,14768,14769],{},"Integration Partners: Maintenance contracts available with our partners",[71,14771],{},[288,14773,14775],{"id":14774},"global-ready-ai-optimized",[14,14776,14777],{},"Global-Ready, AI-Optimized",[18,14779,14780],{},"Whether you’re scaling up a cloud platform, building a national AI hub, or deploying mission-critical AI in the field: Eliovp delivers fast, scalable infrastructure wherever you need it.",[18,14782,14783],{},"From regional clusters to hyperscale-ready edge deployments, our modular systems are built for speed, flexibility, and global compliance.",[71,14785],{},[288,14787,14789],{"id":14788},"eliovp-lite-cost-effective-solution-for-non-critical-workloads",[14,14790,14791],{},"Eliovp Lite: Cost-Effective Solution for Non-Critical Workloads",[18,14793,14794],{},"Understanding that not every AI deployment requires enterprise-grade redundancy, we’ve developed A Lite setup, a streamlined solution with a significantly smaller footprint and lower cost structure while maintaining compatibility with NVIDIA NVL architectures.",[10,14796,14798],{"id":14797},"eliovp-lite-specifications",[14,14799,14800],{},"Eliovp Lite Specifications",[298,14802,14803,14815],{},[301,14804,14805],{},[304,14806,14807,14811],{},[307,14808,14809],{},[14,14810,14303],{},[307,14812,14813],{},[14,14814,14308],{},[316,14816,14817,14825,14833,14840,14847,14854,14861],{},[304,14818,14819,14822],{},[321,14820,14821],{},"ITE Capacity",[321,14823,14824],{},"10-14 racks per module",[304,14826,14827,14830],{},[321,14828,14829],{},"Rack-Level Power",[321,14831,14832],{},"Up to 120 kW per rack",[304,14834,14835,14837],{},[321,14836,14331],{},[321,14838,14839],{},"500-1.000 kW per unit",[304,14841,14842,14844],{},[321,14843,14339],{},[321,14845,14846],{},"DLC\u002FRDHx Hybrid",[304,14848,14849,14851],{},[321,14850,14347],{},[321,14852,14853],{},"Single-path power distribution (non-redundant)",[304,14855,14856,14858],{},[321,14857,14371],{},[321,14859,14860],{},"16-18 weeks",[304,14862,14863,14866],{},[321,14864,14865],{},"Footprint",[321,14867,14868],{},"8m × 17m compact configuration",[10,14870,14872],{"id":14871},"ideal-for",[14,14873,14874],{},"Ideal for:",[30,14876,14877,14880,14883,14886],{},[33,14878,14879],{},"Academic and research environments",[33,14881,14882],{},"Development and testing clusters",[33,14884,14885],{},"Small to medium AI startups",[33,14887,14888],{},"Edge computing applications",[18,14890,14891],{},"Eliovp Lite eliminates costly redundancies while maintaining thermal efficiency for NVL hardware. This solution offers a perfect balance for organizations with:",[30,14893,14894,14897,14900],{},[33,14895,14896],{},"Budget constraints",[33,14898,14899],{},"Space limitations",[33,14901,14902],{},"Workloads that can tolerate occasional maintenance windows",[18,14904,14905],{},"By removing dual power paths, redundant cooling systems, and enterprise-grade backup generators, we’re able to reduce initial capital expenditure by up to 40% while still providing a professional-grade environment for AI computation. The streamlined design also reduces ongoing operational costs through simplified maintenance procedures and reduced power overhead.",[18,14907,14908],{},[283,14909],{"alt":285,"src":14910},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Feliovp_non-critical-1mw-pod_rev-2_transparent.webp",[71,14912],{},[288,14914,14916],{"id":14915},"versatile-hardware-support",[14,14917,14918],{},"Versatile Hardware Support",[18,14920,14921],{},"And it’s not just NVIDIA, our modular AI data centers are also designed to support AMD GPUs, making them an excellent choice for inference-focused deployments where cost-efficiency and performance go hand-in-hand.",[71,14923],{},[288,14925,14927],{"id":14926},"conclusion-eliovp-nvl-grade-ai-infrastructure",[14,14928,14929],{},"Conclusion: Eliovp = NVL-Grade AI Infrastructure",[18,14931,14932],{},"Our modular AI architecture is uniquely equipped to deploy the most power- and cooling-intensive NVL configurations, NVL72-scale LLM training clusters or NVL4 edge AI inference units with uncompromising uptime, thermal stability, and operational flexibility.",[18,14934,14935],{},"We engineer infrastructure, not containers.",[18,14937,14938],{},"Ready to get started? Get in touch to explore which design and configuration best fits your needs and secure your place in this transformative AI era.",{"title":285,"searchDepth":673,"depth":673,"links":14940},[14941,14942,14946,14947,14950,14955,14960,14961,14962,14966,14967],{"id":14176,"depth":673,"text":14179},{"id":14193,"depth":673,"text":14196,"children":14943},[14944,14945],{"id":14202,"depth":676,"text":14205},{"id":14247,"depth":676,"text":14250},{"id":14286,"depth":673,"text":14289},{"id":14402,"depth":673,"text":14405,"children":14948},[14949],{"id":14411,"depth":676,"text":14414},{"id":14480,"depth":673,"text":14483,"children":14951},[14952,14953,14954],{"id":14489,"depth":676,"text":14492},{"id":14618,"depth":676,"text":14621},{"id":14667,"depth":676,"text":14670},{"id":14703,"depth":673,"text":14706,"children":14956},[14957,14958,14959],{"id":14712,"depth":676,"text":14715},{"id":14721,"depth":676,"text":14724},{"id":14730,"depth":676,"text":14733},{"id":14741,"depth":673,"text":14744},{"id":14774,"depth":673,"text":14777},{"id":14788,"depth":673,"text":14791,"children":14963},[14964,14965],{"id":14797,"depth":676,"text":14800},{"id":14871,"depth":676,"text":14874},{"id":14915,"depth":673,"text":14918},{"id":14926,"depth":673,"text":14929},[688,1148,14969,4865,5188,14970,5189,5190,14971,14972,5193,14973],"1-2MW Data Center","HPC","NVIDIA Blackwell","NVIDIA NVL","Precision Cooling","2025-05-02T14:09:59","Purpose-Built High-Density Infrastructure for Blackwell-Class AI Workloads At Eliovp, we’re engineering a new class of AI infrastructure. Our advanced modular platform is built to also support NVIDIA’s cutting-edge NVL architecture, from the efficient NVL4 to the ultra-scale NVL72, enabling deployments that range from distributed edge inference to full-stack model training at hyperscale. Designed to meet ...",{},"https:\u002F\u002Feliovp.com\u002Fpower-meets-precision-high-density-modular-data-center-for-nvidia-nvl-deployments-1-2-mw\u002F","\u002Fblog\u002Fpower-meets-precision-high-density-modular-data-center-for-nvidia-nvl-deployments-1-2-mw",{"title":14171,"description":14975},"power-meets-precision-high-density-modular-data-center-for-nvidia-nvl-deployments-1-2-mw","blog\u002Fpower-meets-precision-high-density-modular-data-center-for-nvidia-nvl-deployments-1-2-mw","72u0J6ka3w6cZV8Vz5sSdSQ5hnvkx2UuOpdoCppIWGM",{"id":14984,"title":14985,"body":14986,"categories":15441,"date":15443,"description":15444,"extension":699,"image":15445,"meta":15446,"navigation":702,"originalUrl":15447,"path":15448,"seo":15449,"slug":15450,"stem":15451,"__hash__":15452},"blog\u002Fblog\u002Fexamining-ai-agents-in-the-medical-field-ai-that-speaks-dicom.md","Examining AI agents in the medical field: AI that speaks DICOM",{"type":7,"value":14987,"toc":15434},[14988,14995,15001,15008,15032,15035,15041,15044,15047,15053,15056,15063,15070,15077,15084,15089,15095,15098,15109,15115,15118,15121,15186,15191,15194,15200,15205,15210,15215,15220,15225,15229,15234,15239,15244,15249,15254,15259,15264,15269,15274,15278,15283,15288,15293,15298,15303,15308,15313,15318,15323,15327,15332,15337,15342,15347,15352,15357,15362,15367,15372,15377,15382,15386,15391,15396,15401,15406,15411,15416,15421,15423,15426,15431],[18,14989,14990,14991,14994],{},"At Eliovp, we’re constantly keeping up with the newest AI trends. Consequently, we have been looking into AI agents and have created a medical agent designed to ",[14,14992,14993],{},"seamlessly interact with DICOM servers inside hospitals",". This isn’t just another chatbot or AI tool. This is an intelligent assistant that understands the language of radiology and is built to support the people at the heart of healthcare: radiologists, technicians, and hospital staff.",[10,14996,14998],{"id":14997},"what-can-the-medical-agent-do",[14,14999,15000],{},"What can the medical agent do?",[18,15002,15003,15004,15007],{},"Our agent connects directly to your hospital’s ",[14,15005,15006],{},"DICOM infrastructure",", which means it can:",[30,15009,15010,15016,15022],{},[33,15011,15012,15015],{},[14,15013,15014],{},"Look up patients"," by name, ID, or metadata",[33,15017,15018,15021],{},[14,15019,15020],{},"Retrieve study information",", including modalities, timestamps, and diagnostic notes",[33,15023,15024,15027,15028,15031],{},[14,15025,15026],{},"Analyze medical images"," for abnormalities, pathologies, and injuries. The agent can retrieve the relevant images directly from the DICOM server and give an analysis ",[14,15029,15030],{},"solely based on the images",". No reports needed!",[18,15033,15034],{},"With this agent, radiologists don’t need to spend hours each day looking up patients, searching through all studies and series, and analysing images manually. They can just ask the agent in plain language and it will do the rest.",[10,15036,15038],{"id":15037},"seamless-workflow-no-extra-tools-and-secure-connections",[14,15039,15040],{},"Seamless workflow, no extra tools and secure connections",[18,15042,15043],{},"What makes this agent truly powerful is how naturally it integrates into existing hospital workflows. There’s no need to learn new software or navigate complex menus. Just ask and get the information you need.",[18,15045,15046],{},"Many questions may arise about security concerns. Don’t worry, because the agent will never share sensitive data to the outside world as it can run on your own servers with self-hosted models. Built-in limitations in its tools also form a second protection layer for the hospital’s data. So, data leakage will never occur!",[10,15048,15050],{"id":15049},"see-it-in-action",[14,15051,15052],{},"See it in action",[18,15054,15055],{},"We’ve recorded a few demos to show the agent in action. They range from fetching patient records to detecting pathologies in diagnostic images. These videos demonstrate real-world tasks that hospital staff perform every day, now made faster and easier with agentic AI support.",[18,15057,15058],{},[163,15059,15062],{"href":15060,"rel":15061},"https:\u002F\u002Feliovp.com\u002Fwp-content\u002Fuploads\u002FUntitled1.m4v",[167],"Video: Untitled1.m4v",[18,15064,15065],{},[163,15066,15069],{"href":15067,"rel":15068},"https:\u002F\u002Feliovp.com\u002Fwp-content\u002Fuploads\u002FUntitled.m4v",[167],"Video: Untitled.m4v",[18,15071,15072],{},[163,15073,15076],{"href":15074,"rel":15075},"https:\u002F\u002Feliovp.com\u002Fwp-content\u002Fuploads\u002FUntitled3-1.m4v",[167],"Video: Untitled3-1.m4v",[18,15078,15079,15080,15083],{},"These demos were performed in a test environment and ",[14,15081,15082],{},"the execution speed can still be optimized",". The agent always takes a step-by-step approach in gathering the most relevant information, so multiple tool calls like querying patients, studies and series, are performed sequentially.",[18,15085,15086],{},[14,15087,15088],{},"All patient information has also been anonymized.",[10,15090,15092],{"id":15091},"technicaldetails",[14,15093,15094],{},"Technicaldetails",[18,15096,15097],{},"Here’s a small summary of the technical side of the agent:",[30,15099,15100,15103,15106],{},[33,15101,15102],{},"Agent LLM: Qwen2.5-32B-Instruct served with vLLM on 1 AMD MI300x GPU",[33,15104,15105],{},"DICOM server in the demo: Self-hosted Orthanc",[33,15107,15108],{},"Vision language model: several state-of-the-art VLMs were compared in analyzing the images and forming a diagnosis, and are listed below. All models were served with vLLM on respectively 4, 1, 2 and 1 AMD MI300x GPUs. The model used in the last demo video above is Qwen\u002FQwen2.5-VL-72B-Instruct.",[10,15110,15112],{"id":15111},"vision-language-model-comparison",[14,15113,15114],{},"Vision language model comparison",[18,15116,15117],{},"By examining the performance on an MRI scan of a knee, we highlight the variability in diagnostic conclusions and the clinical relevance of leading vision language models. Below, the outputs from these models are set out and you can compare them yourself with the radiologist’s report. Clearly, Llama-4-Scout-17B-16E-Instruct and Qwen2.5-VL-32B-Instruct are not able to give a correct diagnosis. Qwen2.5-VL-72B-Instruct and gemma-3-27b-it perform better on the other hand. They are able to diagnose the patient correctly to a great degree.",[18,15119,15120],{},"For your convenience we have created a small scoring system to determine whether the models performed well or poorly. From the radiologist’s report we extract four key aspects in the diagnosis: ACL tear (1), limited Segond fracture and elongation of the anterolateral ligament (2), bone edema (3) and presumably no meniscus tear (4). If the VLM mentions one of these key aspects in its diagnosis, it gets one point. If it mentions two aspects it gets two points and so on.",[298,15122,15123,15142],{},[301,15124,15125],{},[304,15126,15127,15132,15137],{},[307,15128,15129],{},[14,15130,15131],{},"Model",[307,15133,15134],{},[14,15135,15136],{},"Score (points)",[307,15138,15139],{},[14,15140,15141],{},"Score (percentage)",[316,15143,15144,15155,15166,15175],{},[304,15145,15146,15149,15152],{},[321,15147,15148],{},"meta-llama\u002FLlama-4-Scout-17B-16E-Instruct",[321,15150,15151],{},"0\u002F4",[321,15153,15154],{},"0%",[304,15156,15157,15160,15163],{},[321,15158,15159],{},"google\u002Fgemma-3-27b-it",[321,15161,15162],{},"3\u002F4",[321,15164,15165],{},"75%",[304,15167,15168,15171,15173],{},[321,15169,15170],{},"Qwen\u002FQwen2.5-VL-72B-Instruct",[321,15172,15162],{},[321,15174,15165],{},[304,15176,15177,15180,15183],{},[321,15178,15179],{},"Qwen\u002FQwen2.5-VL-32B-Instruct",[321,15181,15182],{},"1\u002F4",[321,15184,15185],{},"25%",[18,15187,15188],{},[630,15189,15190],{},"Disclaimer: We are no medical professionals and have based the scoring solely on the text output.",[18,15192,15193],{},"The output from the models and the radiologist’s conclusion can be found below.",[18,15195,15196,15197],{},"Input message: ",[630,15198,15199],{},"Can you analyze the images in series pd_tse_sag from patient ID 0000850361?",[18,15201,15202],{},[14,15203,15204],{},"Radiologist",[18,15206,15207],{},[630,15208,15209],{},"Complete tear of the anterior cruciate ligament; associated limited Segond fracture with some elongation of the anterolateral ligament.",[18,15211,15212],{},[630,15213,15214],{},"Significant bone edema primarily lateral tibial, somewhat less medial tibial, and even less medial femoral, with inset bone fissures primarily lateral tibial.",[18,15216,15217],{},[630,15218,15219],{},"Presumably no tear of the posterior horn of the lateral meniscus see description.",[18,15221,15222],{},[630,15223,15224],{},"No tear of the medial meniscus.",[18,15226,15227],{},[14,15228,15148],{},[18,15230,15231],{},[630,15232,15233],{},"The analysis of the MRI images from the series ‘pd_tse_sag’ has been completed. The conclusion is as follows:",[18,15235,15236],{},[630,15237,15238],{},"Diagnosis:",[18,15240,15241],{},[630,15242,15243],{},"Based on the provided MRI image of the knee, the diagnosis appears to be a Posterior Cruciate Ligament (PCL) injury, specifically a PCL tear.",[18,15245,15246],{},[630,15247,15248],{},"Reasoning Steps:",[18,15250,15251],{},[630,15252,15253],{},"Visible Disruption: The PCL appears disrupted or torn, which is a common injury in sports and trauma.",[18,15255,15256],{},[630,15257,15258],{},"Clinical Context: PCL injuries often result from a direct blow to the front of the knee or from a fall onto a flexed knee. Symptoms may include pain, instability, and swelling.",[18,15260,15261],{},[630,15262,15263],{},"Imaging Features: The MRI shows a discontinuity or abnormal signal in the PCL, indicative of a tear.",[18,15265,15266],{},[630,15267,15268],{},"Conclusion:",[18,15270,15271],{},[630,15272,15273],{},"The diagnosis of a PCL injury is critical for guiding treatment, which may include conservative management or surgical intervention, depending on the severity and the patient’s activity level.",[18,15275,15276],{},[14,15277,15159],{},[18,15279,15280],{},[630,15281,15282],{},"The analysis of the knee MRI series “pd_tse_sag” from patient ID 0000850361 suggests the following:",[18,15284,15285],{},[630,15286,15287],{},"Key Findings:",[18,15289,15290],{},[630,15291,15292],{},"ACL Tear: The anterior cruciate ligament (ACL) appears to be torn as it is frayed and discontinuous.",[18,15294,15295],{},[630,15296,15297],{},"Bone Bruising: There is bone marrow edema in the lateral femoral condyle and the posterior aspect of the lateral tibial plateau, indicating impact forces at the time of injury.",[18,15299,15300],{},[630,15301,15302],{},"Possible Meniscal Tear: The posterior horn of the medial meniscus appears irregular and possibly disrupted.",[18,15304,15305],{},[630,15306,15307],{},"Joint Effusion: There is some fluid within the joint space, which is common after an acute knee injury.",[18,15309,15310],{},[630,15311,15312],{},"Posterior Translation of Tibia: The tibia appears to be slightly translated posteriorly relative to the femur, a sign of ACL insufficiency.",[18,15314,15315],{},[630,15316,15317],{},"Likely Diagnosis: An acute ACL tear with associated bone contusions and a possible medial meniscal tear.",[18,15319,15320,46],{},[630,15321,15322],{},"However, I am an AI and my analysis should not replace professional medical advice. Please consult with a qualified healthcare professional for an accurate diagnosis and treatment plan",[18,15324,15325],{},[14,15326,15170],{},[18,15328,15329],{},[630,15330,15331],{},"The MRI image series ‘pde_tse_sag’ from patient ID 0000850361 shows a sagittal view of the left knee joint. Based on the image analysis, here are some observations and potential diagnoses:",[18,15333,15334],{},[630,15335,15336],{},"ACL (Anterior Cruciate Ligament) Tear: The ACL appears to be disrupted or torn. The normal continuity of the ligament is not visible, which is a strong indicator of an ACL tear.",[18,15338,15339],{},[630,15340,15341],{},"Bone Bruise: There may be evidence of bone bruising (bone marrow edema) in the femur and tibia, which is often associated with ACL injuries.",[18,15343,15344],{},[630,15345,15346],{},"Meniscus Tear: The meniscus appears to be intact based on this image, but a more detailed analysis would be needed to rule out any tears.",[18,15348,15349],{},[630,15350,15351],{},"Joint effusion: There is a significant amount of fluid in the joint space, which is consistent with an acute injury.",[18,15353,15354],{},[630,15355,15356],{},"Other Soft Tissue Injuries: There may be additional soft tissue injuries, such as damage to the collateral ligaments or other structures, but this would require further evaluation.",[18,15358,15359],{},[630,15360,15361],{},"Recommendations:",[18,15363,15364],{},[630,15365,15366],{},"Clinical Correlation: The findings should be correlated with the patient’s clinical history and physical examination.",[18,15368,15369],{},[630,15370,15371],{},"Further Imaging: Additional MRI sequences (e.g., T1-weighted, PD-weighted) may be needed for a more comprehensive evaluation.",[18,15373,15374],{},[630,15375,15376],{},"Referral to Orthopedic Specialist: Given the potential for an ACL tear, referral to an orthopedic surgeon for further management and possible surgical intervention is recommended.",[18,15378,15379],{},[630,15380,15381],{},"Please consult with a medical professional for a definitive diagnosis and treatment plan.",[18,15383,15384],{},[14,15385,15179],{},[18,15387,15388],{},[630,15389,15390],{},"Based on the analysis of MRI images from the series “pd_tse_sag” for the patient with ID 0000850361, here are some potential findings:",[18,15392,15393],{},[630,15394,15395],{},"The joint space appears intact, suggesting no significant degenerative changes.",[18,15397,15398],{},[630,15399,15400],{},"There is a focal area of increased signal intensity in the medial meniscus, which could indicate a tear or degeneration.",[18,15402,15403],{},[630,15404,15405],{},"The cruciate ligaments (ACL and PCL) appear intact, but subtle tears or sprains might not be visible without specific sequences.",[18,15407,15408],{},[630,15409,15410],{},"There is a region of increased signal intensity in the bone marrow of the tibial plateau, which could indicate bone marrow edema (BME). This is often associated with trauma, stress fractures, or inflammatory conditions.",[18,15412,15413],{},[630,15414,15415],{},"The surrounding soft tissues do not show significant abnormalities, but there may be some fluid accumulation around the joint.",[18,15417,15418],{},[630,15419,15420],{},"However, it’s important to note that these findings are purely observational and cannot replace a proper medical diagnosis. Consulting with a healthcare professional who can consider the patient’s symptoms, medical history, and conduct a physical examination would be necessary for a definitive diagnosis and treatment plan.",[18,15422,643],{},[18,15424,15425],{},"Humanity is one step closer to achieving a simplified and automated diagnosis tool. However, it’s important to tread carefully. Oversimplification can lead to overly broad or inaccurate diagnoses, potentially overlooking critical details and increasing the risk of worst-case outcomes. Striking the right balance between usability and clinical precision is key.",[18,15427,15428],{},[283,15429],{"alt":285,"src":15430},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Flh7-rt.googleusercontent.com-3691ed194a23.gif",[18,15432,15433],{},"While general-purpose Vision-Language Models (VLMs) demonstrate impressive capabilities, their outputs can be significantly improved through fine-tuning on domain-specific datasets. In the context of healthcare, this means training the model with high-quality medical images, radiology reports, pathology notes, or other clinical documentation. Fine-tuning allows the model to learn medical terminology, visual patterns specific to diseases, and nuanced diagnostic cues, resulting in more accurate and context-aware outputs. This domain adaptation is crucial when deploying AI tools in sensitive environments like hospitals, where precision and reliability are paramount.",{"title":285,"searchDepth":673,"depth":673,"links":15435},[15436,15437,15438,15439,15440],{"id":14997,"depth":676,"text":15000},{"id":15037,"depth":676,"text":15040},{"id":15049,"depth":676,"text":15052},{"id":15091,"depth":676,"text":15094},{"id":15111,"depth":676,"text":15114},[688,689,3971,1149,9658,690,15442],"Healthcare","2025-04-11T14:45:48","At Eliovp, we’re constantly keeping up with the newest AI trends. Consequently, we have been looking into AI agents and have created a medical agent designed to seamlessly interact with DICOM servers inside hospitals. This isn’t just another chatbot or AI tool. This is an intelligent assistant that understands the language of radiology and is ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fhealthcareblog-1.webp",{},"https:\u002F\u002Feliovp.com\u002Fexamining-ai-agents-in-the-medical-field-ai-that-speaks-dicom\u002F","\u002Fblog\u002Fexamining-ai-agents-in-the-medical-field-ai-that-speaks-dicom",{"title":14985,"description":15444},"examining-ai-agents-in-the-medical-field-ai-that-speaks-dicom","blog\u002Fexamining-ai-agents-in-the-medical-field-ai-that-speaks-dicom","NG5uitE2W-xbOSpYSjzFDc23Plv9Z3jwFwyuqbldM90",{"id":15454,"title":15455,"body":15456,"categories":15697,"date":15702,"description":15703,"extension":699,"image":15704,"meta":15705,"navigation":702,"originalUrl":15706,"path":15707,"seo":15708,"slug":15709,"stem":15710,"__hash__":15711},"blog\u002Fblog\u002Feliovp-bv-your-trusted-partner-for-supply-chain-resilience-amidst-new-u-s-tariffs.md","Eliovp BV: Your Trusted Partner for Supply Chain Resilience Amidst New U.S. Tariffs",{"type":7,"value":15457,"toc":15682},[15458,15461,15467,15479,15488,15494,15501,15507,15516,15522,15525,15528,15534,15537,15543,15552,15558,15567,15573,15576,15582,15585,15599,15602,15608,15611,15618,15621,15623,15625,15631],[18,15459,15460],{},"In today’s rapidly evolving global trade landscape, businesses face unprecedented challenges in maintaining efficient and cost-effective IT infrastructure. The recent U.S. tariff adjustments have created waves of uncertainty across international markets, particularly for companies relying on high-performance computing and AI solutions. At Eliovp BV, we want to assure our valued clients that our comprehensive end-to-end solutions, from modular data centers to cutting-edge AI and HPC server offerings, remain a reliable, resilient, and cost-effective choice despite these new trade policies.",[10,15462,15464],{"id":15463},"understanding-the-2025-us-tariff-landscape",[14,15465,15466],{},"Understanding the 2025 U.S. Tariff Landscape",[18,15468,15469,15470,15474,15475],{},"On April 2, 2025, the U.S. administration announced significant changes to its import tariff structure, implementing what many industry analysts are calling the most substantial trade policy shift in recent years. These changes include a 10% baseline tariff on all imports effective April 5, 2025, with additional country-specific tariffs taking effect on April 9, 2025.",[163,15471,6802],{"href":15472,"rel":15473},"https:\u002F\u002Fwww.tradecomplianceresourcehub.com\u002F2025\u002F04\u002F03\u002Fu-s-imposes-10-baseline-tariffs-higher-reciprocal-tariffs-for-targeted-countries\u002F",[167]," Among these targeted measures is a 32% tariff on Taiwanese goods, which has raised concerns throughout the technology sector.",[163,15476,6819],{"href":15477,"rel":15478},"https:\u002F\u002Fwww.reuters.com\u002Fmarkets\u002Ftaiwan-us-driver-shared-prosperity-us-business-group-says-after-tariffs-2025-04-03\u002F",[167],[18,15480,15481,15482,15487],{},"However, in a critical development for the high-performance computing industry, semiconductors have been explicitly exempted from these tariffs.",[163,15483,15486],{"href":15484,"rel":15485},"https:\u002F\u002Fwww.techpowerup.com\u002F335012\u002Fus-exempts-semiconductors-from-taiwan-tariffs-but-chip-making-equipment-remains-on-the-list",[167],"3"," This exemption represents a strategic recognition of the semiconductor industry’s vital importance to technological advancement and economic growth, particularly as AI infrastructure demands continue to accelerate worldwide.",[10,15489,15491],{"id":15490},"eliovp-bvs-strategic-supply-chain-advantage",[14,15492,15493],{},"Eliovp BV’s Strategic Supply Chain Advantage",[18,15495,15496,15497],{},"As an EU-based company with a global supply network, Eliovp BV has implemented a robust supply chain resilience strategy that ensures minimal impact from these tariffs. Our dual-sourcing approach provides significant advantages in the current trade environment:",[163,15498,6836],{"href":15499,"rel":15500},"https:\u002F\u002Fwww.datacenterfrontier.com\u002Fhyperscale\u002Farticle\u002F55279670\u002Fhow-tariffs-could-impact-data-centers-ai-and-energy-amid-supply-chain-shifts",[167],[10,15502,15504],{"id":15503},"taiwan-shipments-leveraging-the-semiconductor-exemption",[14,15505,15506],{},"Taiwan Shipments: Leveraging the Semiconductor Exemption",[18,15508,15509,15510,15515],{},"The semiconductor exemption is particularly beneficial for our clients requiring advanced AI and HPC servers. Our direct drop shipments from Taiwan are strategically positioned to take advantage of this exemption, ensuring that critical components for your high-performance computing needs are not subject to the 32% tariff.",[163,15511,15514],{"href":15512,"rel":15513},"https:\u002F\u002Fwww.taipeitimes.com\u002FNews\u002Ffront\u002Farchives\u002F2025\u002F04\u002F04\u002F2003834588",[167],"5"," This allows us to continue providing cutting-edge technology without the burden of substantial price increases that would otherwise impact your IT budget.",[10,15517,15519],{"id":15518},"us-shipments-minimizing-tariff-impact",[14,15520,15521],{},"U.S. Shipments: Minimizing Tariff Impact",[18,15523,15524],{},"For products shipped directly from the U.S., only the 10% baseline tariff applies, significantly lower than the country-specific tariffs imposed on many other nations. Our established relationships with U.S. suppliers enable us to maintain competitive pricing while ensuring compliance with all trade regulations. This approach provides our clients with greater flexibility in sourcing decisions based on specific project requirements and timeline considerations.",[18,15526,15527],{},"This dual-source strategy allows us to navigate the new tariff landscape effectively, ensuring that our clients receive top-tier hardware without unexpected cost increases. In an era where supply chain disruptions have become commonplace, our approach provides the resilience and adaptability necessary for successful IT infrastructure deployment.",[10,15529,15531],{"id":15530},"beyond-hardware-comprehensive-solutions-for-the-ai-era",[14,15532,15533],{},"Beyond Hardware: Comprehensive Solutions for the AI Era",[18,15535,15536],{},"While tariff considerations are important, Eliovp BV’s value proposition extends far beyond hardware sourcing. As an end-to-end solution provider, we deliver comprehensive services that address the full spectrum of modern IT infrastructure needs:",[10,15538,15540],{"id":15539},"iso-certified-modular-data-centers",[14,15541,15542],{},"ISO-Certified Modular Data Centers",[18,15544,15545,15546,15551],{},"Our ISO-certified containerized data centers represent the pinnacle of flexibility and scalability in modern IT infrastructure. These modular solutions enable rapid deployment of computing resources while meeting stringent international standards for quality and reliability.",[163,15547,15550],{"href":15548,"rel":15549},"https:\u002F\u002Fwww.whitehouse.gov\u002Ffact-sheets\u002F2025\u002F04\u002Ffact-sheet-president-donald-j-trump-declares-national-emergency-to-increase-our-competitive-edge-protect-our-sovereignty-and-strengthen-our-national-and-economic-security\u002F",[167],"6"," In a market where speed-to-implementation can provide critical competitive advantages, our modular approach delivers exceptional value regardless of tariff considerations.",[10,15553,15555],{"id":15554},"edge-ai-and-quantum-ready-infrastructure",[14,15556,15557],{},"Edge AI and Quantum-Ready Infrastructure",[18,15559,15560,15561,15566],{},"The future of computing increasingly demands distributed intelligence capabilities. Our edge AI solutions bring computational power closer to data sources, reducing latency and enabling real-time decision-making.",[163,15562,15565],{"href":15563,"rel":15564},"https:\u002F\u002Fwww.itprotoday.com\u002Fai-machine-learning\u002Fai-trends-and-predictions-2025-from-industry-insiders",[167],"7"," Additionally, our forward-looking infrastructure designs incorporate quantum-readiness principles, ensuring your investments remain valuable as quantum computing technologies mature in the coming years.",[10,15568,15570],{"id":15569},"ai-enhanced-security-and-compliance",[14,15571,15572],{},"AI-Enhanced Security and Compliance",[18,15574,15575],{},"In today’s threat landscape, security cannot be an afterthought. Our solutions incorporate AI-enhanced security features that provide proactive threat detection and mitigation capabilities. Furthermore, our comprehensive compliance expertise ensures that your infrastructure meets all relevant regulatory requirements, from GDPR to industry-specific standards.",[10,15577,15579],{"id":15578},"commitment-to-client-success-in-uncertain-times",[14,15580,15581],{},"Commitment to Client Success in Uncertain Times",[18,15583,15584],{},"At Eliovp BV, we understand that external trade fluctuations can create uncertainty in your technology planning. That’s why we’re dedicated to providing seamless, cost-effective solutions regardless of the global trade environment. Our proactive measures and adaptable logistics underscore our commitment to delivering exceptional value through:",[30,15586,15587,15590,15593,15596],{},[33,15588,15589],{},"Continuous monitoring of global trade policies to anticipate changes",[33,15591,15592],{},"Transparent communication about potential impacts on project timelines and costs",[33,15594,15595],{},"Flexible financing options that accommodate budget constraints",[33,15597,15598],{},"Ongoing optimization services that maximize the efficiency of your IT investments",[18,15600,15601],{},"Our team of certified Tier 4 specialists, AI infrastructure experts, and performance optimization professionals works tirelessly to ensure that your technology investments deliver maximum value, regardless of external economic factors.",[10,15603,15605],{"id":15604},"navigating-the-future-together",[14,15606,15607],{},"Navigating the Future Together",[18,15609,15610],{},"The intersection of high-performance computing, artificial intelligence, and global trade policy presents both challenges and opportunities. By partnering with Eliovp BV, you gain not just a technology provider, but a strategic ally committed to your long-term success. Our holistic approach to IT infrastructure, encompassing hardware, software, colocation services, and ongoing support, provides the foundation you need to thrive in an increasingly complex technological landscape.",[18,15612,15613,15614],{},"For organizations seeking to maintain competitive advantages while managing costs effectively, our comprehensive solutions offer the perfect balance of performance, reliability, and value. We invite you to explore how our approach can help your organization navigate the current tariff environment while positioning you for future success in the rapidly evolving world of high-performance computing and artificial intelligence.",[163,15615,6853],{"href":15616,"rel":15617},"https:\u002F\u002Fwww.hpcwire.com\u002F2025\u002F01\u002F28\u002Ffive-big-questions-for-hpc-ai-in-2025\u002F",[167],[18,15619,15620],{},"For further details on the recent tariff changes and their implications for your IT infrastructure strategy, please contact our team or refer to the resources below.",[71,15622],{},[288,15624,4559],{"id":4556},[10,15626,15628],{"id":15627},"footnotes",[14,15629,15630],{},"Footnotes",[114,15632,15633,15640,15646,15652,15658,15664,15670,15676],{},[33,15634,15635,15636,15639],{},"Trade Compliance Resource Hub. (April 3, 2025). “U.S. imposes 10% baseline tariffs; higher reciprocal tariffs for targeted countries.” Retrieved from ",[163,15637,15472],{"href":15472,"rel":15638},[167],"↩",[33,15641,15642,15643,15639],{},"Reuters. (April 3, 2025) . “Taiwan says US tariffs unreasonable, partly blames Trump.” Retrieved from ",[163,15644,15477],{"href":15477,"rel":15645},[167],[33,15647,15648,15649,15639],{},"TechPowerUp. (April 3, 2025) . “US Exempts Semiconductors From Taiwan Tariffs, But Chip-Making Equipment Remains on the List.” Retrieved from ",[163,15650,15484],{"href":15484,"rel":15651},[167],[33,15653,15654,15655,15639],{},"Data Center Frontier. (April 3, 2025) . “How Tariffs Could Impact Data Centers, AI, and Energy Amid Supply Chain Shifts.” Retrieved from ",[163,15656,15499],{"href":15499,"rel":15657},[167],[33,15659,15660,15661,15639],{},"Taipei Times. (April 4, 2025) . “Cabinet decries US’ 32 percent tariffs.” Retrieved from ",[163,15662,15512],{"href":15512,"rel":15663},[167],[33,15665,15666,15667,15639],{},"White House. (April 2, 2025) . “Fact Sheet: President Donald J. Trump Declares National Emergency to Increase Our Competitive Edge.” Retrieved from ",[163,15668,15548],{"href":15548,"rel":15669},[167],[33,15671,15672,15673,15639],{},"ITPro Today. “AI Trends and Predictions 2025 From Industry Insiders.” Retrieved from ",[163,15674,15563],{"href":15563,"rel":15675},[167],[33,15677,15678,15679,15639],{},"HPCwire. (January 28, 2025) . “Five Big Questions for HPC-AI in 2025.” Retrieved from ",[163,15680,15616],{"href":15616,"rel":15681},[167],{"title":285,"searchDepth":673,"depth":673,"links":15683},[15684,15685,15686,15687,15688,15689,15690,15691,15692,15693,15694],{"id":15463,"depth":676,"text":15466},{"id":15490,"depth":676,"text":15493},{"id":15503,"depth":676,"text":15506},{"id":15518,"depth":676,"text":15521},{"id":15530,"depth":676,"text":15533},{"id":15539,"depth":676,"text":15542},{"id":15554,"depth":676,"text":15557},{"id":15569,"depth":676,"text":15572},{"id":15578,"depth":676,"text":15581},{"id":15604,"depth":676,"text":15607},{"id":4556,"depth":673,"text":4559,"children":15695},[15696],{"id":15627,"depth":676,"text":15630},[688,2125,9658,690,15698,15699,15700,15701],"import","Taiwan","Tariffs","Trump","2025-04-04T10:01:27","In today’s rapidly evolving global trade landscape, businesses face unprecedented challenges in maintaining efficient and cost-effective IT infrastructure. The recent U.S. tariff adjustments have created waves of uncertainty across international markets, particularly for companies relying on high-performance computing and AI solutions. At Eliovp BV, we want to assure our valued clients that our comprehensive end-to-end ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Ftariffsshipping.webp",{},"https:\u002F\u002Feliovp.com\u002Feliovp-bv-your-trusted-partner-for-supply-chain-resilience-amidst-new-u-s-tariffs\u002F","\u002Fblog\u002Feliovp-bv-your-trusted-partner-for-supply-chain-resilience-amidst-new-u-s-tariffs",{"title":15455,"description":15703},"eliovp-bv-your-trusted-partner-for-supply-chain-resilience-amidst-new-u-s-tariffs","blog\u002Feliovp-bv-your-trusted-partner-for-supply-chain-resilience-amidst-new-u-s-tariffs","4PHFr4ioSdxMNmRmMTRZammGqYdje31ZAnirFCkNul4",{"id":15713,"title":15714,"body":15715,"categories":15930,"date":15933,"description":15934,"extension":699,"image":15935,"meta":15936,"navigation":702,"originalUrl":15937,"path":15938,"seo":15939,"slug":15940,"stem":15941,"__hash__":15942},"blog\u002Fblog\u002Fwhy-ai-agents-are-the-future.md","Why AI Agents Are the Future",{"type":7,"value":15716,"toc":15922},[15717,15722,15725,15730,15733,15738,15741,15743,15749,15752,15772,15774,15780,15783,15788,15791,15796,15799,15804,15807,15809,15815,15818,15823,15826,15831,15834,15839,15842,15844,15850,15870,15872,15878,15881,15892,15894,15900,15903,15908,15911,15913],[18,15718,15719],{},[14,15720,15721],{},"1. Versatile Integration",[18,15723,15724],{},"AI Agents are designed to integrate seamlessly with your existing software stack. This includes ERP, CRM, and marketing automation platforms. Instead of disrupting current systems, they complement and enhance them, all while learning from and adapting to your specific operational needs.",[18,15726,15727],{},[14,15728,15729],{},"2. Intelligent Decision-Making",[18,15731,15732],{},"Conventional automation scripts handle if-then scenarios, but they fall short when variables shift unexpectedly. AI Agents, on the other hand, can interpret data in real time, spot trends, and make informed decisions based on evolving conditions. This dynamic adaptability translates to fewer bottlenecks and more consistent performance across your organization.",[18,15734,15735],{},[14,15736,15737],{},"3. Scalability and Flexibility",[18,15739,15740],{},"Whether you’re a small startup exploring targeted automation or a global enterprise seeking broad-scale transformation, AI Agents can be implemented in a modular fashion. You can start small, say, automating invoice handling in your finance department, then expand into other areas, like customer service or supply chain management, as the benefits become evident.",[71,15742],{},[10,15744,15746],{"id":15745},"cost-savings-and-bottom-line-impact",[14,15747,15748],{},"Cost Savings and Bottom-Line Impact",[18,15750,15751],{},"One of the most compelling aspects of AI Agents is their rapid ROI. By offloading routine tasks from your team to automated systems, you free up valuable talent to focus on strategic initiatives. Meanwhile, your day-to-day operations become faster, leaner, and more accurate.",[30,15753,15754,15760,15766],{},[33,15755,15756,15759],{},[14,15757,15758],{},"Reduced Labor Costs:"," Fewer tedious, repetitive processes lead to lower overhead and decreased need for overtime or extra hires.",[33,15761,15762,15765],{},[14,15763,15764],{},"Error Minimization:"," Automated data entry and anomaly detection result in fewer costly mistakes, which can otherwise ripple through departments and budgets.",[33,15767,15768,15771],{},[14,15769,15770],{},"Resource Optimization:"," AI Agents can quickly identify inefficiencies, such as underused assets or suboptimal workflows, and take corrective measures without draining management bandwidth.",[71,15773],{},[10,15775,15777],{"id":15776},"enterprise-wide-use-cases1-supply-chain-logistics",[14,15778,15779],{},"Enterprise-Wide Use Cases1. Supply Chain & Logistics",[18,15781,15782],{},"AI Agents track and forecast demand with remarkable precision, syncing inventory levels with real-time insights from sales, production, and external market indicators. This predictive capability leads to just-in-time stock management, slashing both storage costs and missed revenue opportunities due to stockouts.",[18,15784,15785],{},[14,15786,15787],{},"2. Customer Service & Support",[18,15789,15790],{},"Empower your support team with an AI-driven chatbot that handles common queries, instantly troubleshoots technical issues, and seamlessly escalates complex concerns to human agents. Not only do you reduce response times, but your team gains more bandwidth to focus on high-value, relationship-building tasks.",[18,15792,15793],{},[14,15794,15795],{},"3. Marketing & Lead Generation",[18,15797,15798],{},"AI Agents excel at sifting through massive amounts of data, making it easier to personalize content and marketing campaigns. From segmenting email lists based on behavior to automating follow-up messages for warmer leads, these Agents ensure that you’re always engaging your audience with the right message at the right time.",[18,15800,15801],{},[14,15802,15803],{},"4. Finance & Accounting",[18,15805,15806],{},"Tasks like invoice processing, expense reconciliation, and fraud detection can now be executed faster and with fewer errors. AI Agents integrate with your financial software, flagging inconsistencies in real time and maintaining a clear, audit-ready paper trail that keeps you in compliance.",[71,15808],{},[10,15810,15812],{"id":15811},"navigating-the-implementation1-discovery-assessment",[14,15813,15814],{},"Navigating the Implementation1. Discovery & Assessment",[18,15816,15817],{},"We begin by examining your current processes and identifying opportunities for high-impact automation. This helps us define clear KPIs, such as reduced error rates or faster cycle times, so you can measure ROI effectively.",[18,15819,15820],{},[14,15821,15822],{},"2. Custom-Built AI Agents",[18,15824,15825],{},"Every business is unique. That’s why our AI Agents are tailored to your workflows, ensuring they genuinely enhance your processes rather than disrupt them. We’ll work collaboratively with your in-house teams to align on goals and integrate seamlessly into your technology stack.",[18,15827,15828],{},[14,15829,15830],{},"3. Pilot & Optimization",[18,15832,15833],{},"Before full deployment, we conduct pilot programs to gather feedback and refine the solution. This agile approach allows for ongoing iteration and improvements, ensuring minimal downtime and optimal user adoption.",[18,15835,15836],{},[14,15837,15838],{},"4. Deployment & Scale",[18,15840,15841],{},"After a successful pilot, we roll out the solution across relevant departments. As your organization evolves, our AI Agents evolve with it, expanding into new areas and continually learning from performance metrics and changing data sets.",[71,15843],{},[10,15845,15847],{"id":15846},"why-partner-with-us-for-ai-agent-solutions",[14,15848,15849],{},"Why Partner with Us for AI Agent Solutions",[30,15851,15852,15858,15864],{},[33,15853,15854,15857],{},[14,15855,15856],{},"Holistic Approach:"," We don’t just focus on a single pain point; we look at your entire operational ecosystem and identify the most strategic ways to leverage AI across your organization.",[33,15859,15860,15863],{},[14,15861,15862],{},"Transparent ROI:"," Our proven framework ensures you can track savings and performance boosts every step of the way.",[33,15865,15866,15869],{},[14,15867,15868],{},"Long-Term Collaboration:"," Beyond implementation, we remain a trusted partner, offering updates, refinements, and support to keep your AI-driven operations at peak efficiency.",[71,15871],{},[10,15873,15875],{"id":15874},"explore-our-slide-deck",[14,15876,15877],{},"Explore Our Slide Deck",[18,15879,15880],{},"Curious about how AI Agents can deliver impactful business transformation at scale? Check out our Slide Deck for a deeper look into our solutions, their real-world applications, and the data-driven results we’ve achieved for clients in multiple industries. You’ll discover compelling use cases, step-by-step deployment strategies, and quantifiable ROI figures that underscore the value of adopting AI Agents.",[18,15882,15883,15889],{},[163,15884,15886],{"href":15885},"\u002Fasset\u002Ffiles\u002Fblog\u002Felevating-erp-with-tailored-ai-agents.pdf",[14,15887,15888],{},"Custom AI Agents for your business",[163,15890,15891],{"href":15885},"Download",[71,15893],{},[288,15895,15897],{"id":15896},"ready-to-level-up-your-operations",[14,15898,15899],{},"Ready to Level Up Your Operations?",[18,15901,15902],{},"From logistics and supply chain automation to finance, customer service, and beyond, AI Agents can modernize every nook and cranny of your enterprise. If you’re prepared to reduce costs, minimize errors, and reimagine how your teams collaborate and innovate, it’s time to consider the game-changing impact of intelligent automation.",[18,15904,15905],{},[14,15906,15907],{},"Get in Touch",[18,15909,15910],{},"We’re here to guide you through every phase of AI adoption, from initial consults to enterprise-wide deployment. Reach out today to learn how AI Agents can radically optimize your workflows, sharpen your competitive edge, and set you up for sustained success in an ever-evolving marketplace.",[71,15912],{},[18,15914,15915],{},[14,15916,15917,15918],{},"Ready to deploy AI Agents that go beyond ERP and touch every corner of your enterprise? Let’s start the conversation, and watch how quickly your business adapts, excels, and grows in this new era of intelligent automation.",[163,15919,15921],{"href":10000,"rel":15920},[167],"Reach out",{"title":285,"searchDepth":673,"depth":673,"links":15923},[15924,15925,15926,15927,15928,15929],{"id":15745,"depth":676,"text":15748},{"id":15776,"depth":676,"text":15779},{"id":15811,"depth":676,"text":15814},{"id":15846,"depth":676,"text":15849},{"id":15874,"depth":676,"text":15877},{"id":15896,"depth":673,"text":15899},[688,689,3971,9658,15931,15932],"AI Agents","ERP","2025-03-23T22:06:59","1. Versatile IntegrationAI Agents are designed to integrate seamlessly with your existing software stack. This includes ERP, CRM, and marketing automation platforms. Instead of disrupting current systems, they complement and enhance them, all while learning from and adapting to your specific operational needs. 2. Intelligent Decision-MakingConventional automation scripts handle if-then scenarios, but they fall short ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Ferp2.jpeg",{},"https:\u002F\u002Feliovp.com\u002Fwhy-ai-agents-are-the-future\u002F","\u002Fblog\u002Fwhy-ai-agents-are-the-future",{"title":15714,"description":15934},"why-ai-agents-are-the-future","blog\u002Fwhy-ai-agents-are-the-future","nXeI2ctzPJBBYc2sqKT1sfwI04kWFNFdmuzmZ8CyC8Y",{"id":15944,"title":15945,"body":15946,"categories":16162,"date":16164,"description":16165,"extension":699,"image":16166,"meta":16167,"navigation":702,"originalUrl":16168,"path":16169,"seo":16170,"slug":16171,"stem":16172,"__hash__":16173},"blog\u002Fblog\u002Fthe-rise-of-open-source-ai-model-optimization.md","The Rise of Open-Source AI Model Optimization",{"type":7,"value":15947,"toc":16152},[15948,15951,15957,15960,15963,15966,15972,15981,15987,15990,15996,15999,16002,16008,16011,16017,16020,16027,16036,16039,16044,16047,16054,16057,16064,16067,16074,16077,16084,16087,16094,16097,16104,16107,16114,16117,16124,16127,16133,16136,16139,16143,16146,16149],[18,15949,15950],{},"In the rapidly evolving landscape of artificial intelligence (AI), open-source solutions are emerging as pivotal drivers of innovation and performance enhancement. These community-driven platforms democratize access to cutting-edge technologies, fostering collaboration and accelerating advancements in AI model optimization.​",[10,15952,15954],{"id":15953},"the-open-source-revolution-in-ai",[14,15955,15956],{},"The Open-Source Revolution in AI",[18,15958,15959],{},"Open-source AI models have transformed the development and deployment of machine learning applications. By providing transparent, modifiable, and freely accessible codebases, these models empower developers and organizations to tailor AI solutions to their specific needs. This flexibility not only reduces dependency on proprietary systems but also encourages a culture of shared knowledge and continuous improvement.​",[18,15961,15962],{},"One notable example is DeepSeek-R1, an open-source model released by the Chinese AI company DeepSeek. Despite being developed with significantly fewer resources than its counterparts, DeepSeek-R1 has demonstrated performance comparable to industry leaders like OpenAI on various benchmarks. This achievement exemplifies the power of “frugal innovation,” where resource constraints drive creative problem-solving and efficient model optimization. ​",[18,15964,15965],{},"More recently, Anthropic’s Claude 3 family and Meta’s Llama 3 models have demonstrated that open and partially open models can achieve performance rivaling or exceeding closed-source solutions, challenging the notion that AI development requires enormous proprietary resources.​",[10,15967,15969],{"id":15968},"paiton-pioneering-ai-optimization-software",[14,15970,15971],{},"Paiton: Pioneering AI Optimization Software",[18,15973,15974,15975,15980],{},"In this dynamic environment, ",[163,15976,15979],{"href":15977,"rel":15978},"https:\u002F\u002Feliovp.com\u002Fa-first-look-at-paiton-in-action-deepseek-r1-distill-llama-3-1-8b\u002F",[167],"our Paiton tool"," stands out as a leading AI model optimization software. Designed to enhance the performance of AI models, Paiton has consistently delivered improvements ranging from 15% to 30% on AMD Instinct Accelerators such as the MI3** series, often surpassing the performance of equivalent Nvidia hardware. By leveraging advanced optimization techniques, Paiton enables organizations to maximize the efficiency and effectiveness of their AI deployments.​",[10,15982,15984],{"id":15983},"groundbreaking-hardware-advancements-from-gtc-2025",[14,15985,15986],{},"Groundbreaking Hardware Advancements from GTC 2025",[18,15988,15989],{},"The recent NVIDIA GTC 2025 conference has unveiled several game-changing developments in AI hardware optimization that are reshaping the industry landscape:​",[10,15991,15993],{"id":15992},"nvidias-revolutionary-rubin-ai-chip",[14,15994,15995],{},"NVIDIA’s Revolutionary Rubin AI Chip",[18,15997,15998],{},"NVIDIA has just unveiled its next-generation AI accelerator, the Rubin chip, at GTC 2025. Building on the success of the Blackwell architecture, Rubin represents a quantum leap in AI processing capabilities. According to NVIDIA CEO Jensen Huang, the Rubin chip delivers up to 3x better performance-per-watt compared to the previous generation B200 Blackwell GPU, with specialized circuitry for accelerating transformer architectures and multi-modal AI workloads. ​",[18,16000,16001],{},"The Rubin platform introduces a new memory architecture that reduces data movement bottlenecks by 60%, addressing one of the most significant efficiency constraints in current AI systems. This advancement is particularly relevant for optimization-focused solutions like Paiton, as it opens new avenues for hardware-aware model tuning that can exploit these architectural innovations.​",[10,16003,16005],{"id":16004},"arm-based-ai-acceleration",[14,16006,16007],{},"ARM-Based AI Acceleration",[18,16009,16010],{},"Another significant announcement from GTC 2025 was NVIDIA’s expanded partnership with ARM to develop specialized AI cores for edge devices. This collaboration aims to bring server-class AI capabilities to power-constrained environments, enabling efficient inference for complex models on smartphones, vehicles, and IoT devices. The new reference designs demonstrate a 4x improvement in energy efficiency compared to previous generations, making advanced AI applications viable in scenarios where power and thermal constraints were previously prohibitive.​",[10,16012,16014],{"id":16013},"emerging-trends-in-ai-model-optimization",[14,16015,16016],{},"Emerging Trends in AI Model Optimization",[18,16018,16019],{},"The field of AI model optimization is witnessing several key trends that are shaping its future trajectory:​",[30,16021,16022],{},[33,16023,16024],{},[14,16025,16026],{},"Tencent’s T1 Reasoning Model",[18,16028,16029,16030,16035],{},"In a major development for open-source AI, Chinese tech giant Tencent has recently released its ",[163,16031,16034],{"href":16032,"rel":16033},"https:\u002F\u002Ftencent.github.io\u002Fllm.hunyuan.T1\u002FREADME_EN.html",[167],"T1 reasoning model",", specifically designed to excel at complex reasoning tasks. According to Tencent’s research team, the T1 model introduces a novel architecture that separates the reasoning process from knowledge representation, allowing for more efficient parameter utilization.​",[18,16037,16038],{},"Initial benchmarks show that T1 outperforms Meta’s Llama models on Chinese language tasks by a significant margin, while requiring 35% less computational resources for inference. The model also demonstrates outstanding performance on cross-lingual reasoning benchmarks, indicating that its architectural innovations translate well across different languages and domains.​",[18,16040,16041],{},[283,16042],{"alt":285,"src":16043},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Ftencent.png",[18,16045,16046],{},"Tencent has opened access to the model weights and optimization techniques through their GitHub repository, providing valuable resources for the open-source community to build upon. This release represents a significant contribution to democratizing advanced AI capabilities and shows how optimization-focused approaches can lead to models that are both more capable and more efficient.​",[30,16048,16049],{},[33,16050,16051],{},[14,16052,16053],{},"Advanced Quantization Techniques",[18,16055,16056],{},"Recent breakthroughs in quantization are revolutionizing model optimization. Google’s Gemma and Mistral AI’s Mistral models have demonstrated that aggressive quantization (to 2, 3, or 4 bits) can maintain impressive performance while dramatically reducing memory requirements. The GPTQ and AWQ quantization techniques have become industry standards, with research from MIT and Stanford showing memory footprint reductions of up to 55% while improving inference speed by 2-4x.​",[30,16058,16059],{},[33,16060,16061],{},[14,16062,16063],{},"Sparse Mixture of Experts (MoE) Architecture",[18,16065,16066],{},"The MoE architecture, exemplified by models like Mixtral 8x7B and Google’s Gemini, has emerged as a powerful approach to scaling model capabilities without proportionally increasing computational demands. By selectively activating only relevant “expert” sub-networks for each input, MoE models can achieve performance comparable to models several times their size. Recent benchmarks show MoE models delivering 30-40% better performance-per-parameter than dense architectures.​",[30,16068,16069],{},[33,16070,16071],{},[14,16072,16073],{},"Agentic AI Systems",[18,16075,16076],{},"Developing AI agents capable of autonomous decision-making and learning is becoming increasingly prevalent. These systems can adapt to dynamic environments and perform complex tasks with minimal human intervention. Microsoft’s AutoGen and the open-source LangChain framework have become standard tools for building these agent systems, with recent enhancements in reasoning capabilities through ReAct and Tree-of-Thought prompting techniques.​",[30,16078,16079],{},[33,16080,16081],{},[14,16082,16083],{},"Custom Silicon and Hardware Optimization",[18,16085,16086],{},"The design of specialized hardware, such as custom silicon chips, is enhancing the performance and efficiency of AI models. This trend includes the development of processors tailored for specific AI workloads, reducing power consumption and improving processing speeds. Beyond Nvidia’s dominance, companies like AMD with their MI300X accelerators and startups like Cerebras and Groq are pushing the boundaries of AI-specific hardware, with the latter demonstrating unprecedented token generation speeds of over 1,000 tokens per second on large language models.",[30,16088,16089],{},[33,16090,16091],{},[14,16092,16093],{},"Multimodal AI Integration",[18,16095,16096],{},"Integrating multiple data modalities—such as text, images, and audio—into a single model allows for more comprehensive and context-aware AI applications. This trend is driving advancements in areas like natural language processing and computer vision. Recent multimodal models like Claude 3 Opus, GPT-4 Omni, and Gemini Ultra have demonstrated remarkable capabilities in understanding and generating content across different modalities, with the latest benchmarks showing near-human performance on complex reasoning tasks combining visual and textual information.",[30,16098,16099],{},[33,16100,16101],{},[14,16102,16103],{},"Retrieval-Augmented Generation (RAG)",[18,16105,16106],{},"RAG has emerged as a crucial technique for enhancing model accuracy and reducing hallucinations by retrieving factual information from external knowledge bases during generation. Open-source frameworks like LlamaIndex and LangChain have made implementing sophisticated RAG pipelines accessible to developers of all skill levels. Recent innovations include hybrid retrieval methods that combine dense and sparse embeddings, reducing retrieval latency by up to 40% while improving relevance.",[30,16108,16109],{},[33,16110,16111],{},[14,16112,16113],{},"Edge AI Optimization",[18,16115,16116],{},"Deploying AI algorithms directly on edge devices, such as smartphones and IoT sensors, enables real-time data processing and decision-making without relying on centralized cloud servers. This approach reduces latency and enhances privacy. TinyML frameworks and models like Phi-3-mini, designed to run efficiently on consumer hardware, are driving this trend, with recent benchmarks showing optimized edge models achieving 85-90% of the performance of their larger counterparts while using a fraction of the computational resources.",[30,16118,16119],{},[33,16120,16121],{},[14,16122,16123],{},"Continuous Learning and Adaptation",[18,16125,16126],{},"The shift from static to continuously updated models is gaining momentum. Systems capable of learning from new data and adapting to changing environments without complete retraining are becoming essential for practical AI deployments. Meta’s recent research on parameter-efficient fine-tuning techniques like LoRA (Low-Rank Adaptation) and QLoRA have significantly improved continuous model adaptation, requiring as little as 5% of the resources needed for full model retraining.",[10,16128,16130],{"id":16129},"the-impact-of-open-source-ai-optimization",[14,16131,16132],{},"The Impact of Open-Source AI Optimization",[18,16134,16135],{},"The proliferation of open-source AI optimization tools and models is democratizing access to advanced AI capabilities. Organizations of all sizes can leverage these resources to develop sophisticated AI applications without substantial investment in proprietary technologies. This accessibility fosters innovation, accelerates development cycles, and promotes a collaborative ecosystem where knowledge and advancements are shared freely.",[18,16137,16138],{},"Recent industry reports from Gartner and Forrester indicate organizations leveraging open-source AI optimization tools experience up to 40% reductions in deployment costs and 50% faster time-to-market for AI applications. The vibrant open-source community surrounding tools like Hugging Face’s Transformers library and OpenAI’s Triton language for GPU programming has accelerated innovation, with model optimization techniques evolving at unprecedented rates.",[10,16140,16141],{"id":640},[14,16142,643],{},[18,16144,16145],{},"The rise of open-source AI model optimization signifies a transformative shift in the AI landscape. By embracing solutions like Paiton, organizations can achieve significant performance gains, reduce costs, and contribute to a vibrant community of innovation. As computational efficiency becomes increasingly critical for deploying advanced AI capabilities at scale, tools that maximize hardware utilization will play a central role in the next wave of AI transformation.",[18,16147,16148],{},"Hardware innovations announced at GTC 2025, such as NVIDIA’s Rubin AI chip, alongside software advancements like Tencent’s T1 reasoning model, demonstrate an industry increasingly focused on efficiency and optimization as key differentiators. This aligns perfectly with Paiton’s mission to maximize AI performance across diverse hardware platforms.",[18,16150,16151],{},"The future of AI belongs not merely to those with the largest models but to those who can run them most efficiently. As open-source optimization techniques continue to evolve, we can expect a democratization of AI capabilities, enabling innovations across industries and applications previously constrained by computational limitations.",{"title":285,"searchDepth":673,"depth":673,"links":16153},[16154,16155,16156,16157,16158,16159,16160,16161],{"id":15953,"depth":676,"text":15956},{"id":15968,"depth":676,"text":15971},{"id":15983,"depth":676,"text":15986},{"id":15992,"depth":676,"text":15995},{"id":16004,"depth":676,"text":16007},{"id":16013,"depth":676,"text":16016},{"id":16129,"depth":676,"text":16132},{"id":640,"depth":676,"text":643},[688,689,2125,16163,690,772,966],"AI news","2025-03-22T20:59:23","In the rapidly evolving landscape of artificial intelligence (AI), open-source solutions are emerging as pivotal drivers of innovation and performance enhancement. These community-driven platforms democratize access to cutting-edge technologies, fostering collaboration and accelerating advancements in AI model optimization.​ The Open-Source Revolution in AI Open-source AI models have transformed the development and deployment of machine learning ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Friseofopensource.jpeg",{},"https:\u002F\u002Feliovp.com\u002Fthe-rise-of-open-source-ai-model-optimization\u002F","\u002Fblog\u002Fthe-rise-of-open-source-ai-model-optimization",{"title":15945,"description":16165},"the-rise-of-open-source-ai-model-optimization","blog\u002Fthe-rise-of-open-source-ai-model-optimization","d3A2w688e7o-dhSOub_3mjqoPAYebunb68Vjh5qQLos",{"id":16175,"title":16176,"body":16177,"categories":16589,"date":16592,"description":16593,"extension":699,"image":16594,"meta":16595,"navigation":702,"originalUrl":16596,"path":16597,"seo":16598,"slug":16599,"stem":16600,"__hash__":16601},"blog\u002Fblog\u002Fintroducing-our-benchmarking-tool-powered-by-dstack.md","Introducing Our Benchmarking Tool: Powered by dstack",{"type":7,"value":16178,"toc":16580},[16179,16183,16186,16189,16192,16198,16205,16211,16231,16237,16243,16269,16275,16295,16301,16307,16310,16315,16321,16324,16329,16335,16338,16344,16347,16353,16356,16362,16365,16370,16373,16378,16384,16390,16393,16396,16402,16405,16408,16411,16417,16420,16423,16429,16436,16441,16537,16543,16548,16553,16558,16563,16568,16574,16577],[10,16180,16181],{"id":12255},[14,16182,12258],{},[18,16184,16185],{},"Benchmarking is an essential part of optimizing AI models and software applications. Whether you’re testing AI model inference speeds, profiling different hardware configurations, or ensuring system performance over time, having a reliable benchmarking tool is crucial. However, many existing tools suffer from issues like inconsistent environments, difficult configuration setups, and lack of automation.",[18,16187,16188],{},"vLLM’s benchmarking capabilities are excellent, but we recognize that many users build custom wrappers to streamline their benchmarking workflows. Instead of just automating existing processes, we set out to create something more comprehensive with additional features to enhance benchmarking efficiency.",[18,16190,16191],{},"To address these challenges, we built a new benchmarking tool that integrates seamlessly with dstack, allowing for automated, reproducible, and scalable benchmarking workflows.",[10,16193,16195],{"id":16194},"_2-what-is-dstack",[14,16196,16197],{},"2. What is dstack?",[18,16199,16200,16204],{},[163,16201,245],{"href":16202,"rel":16203},"https:\u002F\u002Fdstack.ai\u002F",[167]," is an open-source platform designed to manage ML and AI workflows efficiently. It simplifies infrastructure management, automates execution, and ensures reproducibility; three key aspects that make it perfect for benchmarking orchestration.",[1230,16206,16208],{"id":16207},"key-features-of-dstack-that-we-utilize",[14,16209,16210],{},"Key Features of dstack that We Utilize:",[30,16212,16213,16219,16225],{},[33,16214,16215,16218],{},[14,16216,16217],{},"Reproducible Environments",": Containerized execution ensures consistency across different runs.",[33,16220,16221,16224],{},[14,16222,16223],{},"Job Orchestration and Management",": Define benchmarking tasks, automate execution, and manage results seamlessly.",[33,16226,16227,16230],{},[14,16228,16229],{},"Multi-Cloud and Multi-Hardware Support",": Easily benchmark across GPUs, whether locally or in the cloud.",[10,16232,16234],{"id":16233},"_3-why-we-built-this-benchmarking-tool",[14,16235,16236],{},"3. Why We Built This Benchmarking Tool",[1230,16238,16240],{"id":16239},"common-challenges-in-benchmarking",[14,16241,16242],{},"Common Challenges in Benchmarking:",[30,16244,16245,16251,16257,16263],{},[33,16246,16247,16250],{},[14,16248,16249],{},"Difficult Environment Configuration",": Minor system differences can impact results, leading to inconsistent benchmarks.",[33,16252,16253,16256],{},[14,16254,16255],{},"Manual Execution and Lack of Automation",": Re-running benchmarks manually is time-consuming and error-prone.",[33,16258,16259,16262],{},[14,16260,16261],{},"Inability to Track Performance Over Time",": Without versioning, it’s hard to detect regressions or improvements effectively.",[33,16264,16265,16268],{},[14,16266,16267],{},"Conflict Prevention",": When multiple developers use shared GPUs, resource conflicts can arise.",[1230,16270,16272],{"id":16271},"how-our-tool-solves-these-challenges",[14,16273,16274],{},"How Our Tool Solves These Challenges:",[30,16276,16277,16283,16289],{},[33,16278,16279,16282],{},[14,16280,16281],{},"Automated Benchmarking Runs",": No manual setup required; everything is orchestrated with dstack.",[33,16284,16285,16288],{},[14,16286,16287],{},"Reproducible and Isolated Execution",": Each run happens in a controlled environment, ensuring consistency.",[33,16290,16291,16294],{},[14,16292,16293],{},"Version-Controlled Results",": Logs and reports are stored for easy comparison across different runs.",[10,16296,16298],{"id":16297},"_4-key-features-of-our-benchmarking-tool",[14,16299,16300],{},"4. Key Features of Our Benchmarking Tool",[1230,16302,16304],{"id":16303},"_1-simple-web-interface",[14,16305,16306],{},"1. Simple Web Interface",[18,16308,16309],{},"Running benchmarks should be straightforward. Our tool provides a minimal web interface that allows users to start benchmarking with just a few clicks.",[18,16311,16312],{},[283,16313],{"alt":285,"src":16314},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fsimpleclicksbenchmarktool.jpg",[1230,16316,16318],{"id":16317},"_2-reproducible-and-isolated-environments",[14,16319,16320],{},"2. Reproducible and Isolated Environments",[18,16322,16323],{},"Our tool leverages dstack to spin up containerized environments. This eliminates issues caused by dependency mismatches and system variations.",[18,16325,16326],{},[283,16327],{"alt":285,"src":16328},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fexamplerunsdstack.jpg",[1230,16330,16332],{"id":16331},"_3-vllm-based-benchmarking",[14,16333,16334],{},"3. vLLM-Based Benchmarking",[18,16336,16337],{},"Our tool exclusively benchmarks using vLLM (currently), due to its popularity, available benchmark scripts and massive community.",[1230,16339,16341],{"id":16340},"_4-automated-parameter-sweeping",[14,16342,16343],{},"4. Automated Parameter Sweeping",[18,16345,16346],{},"Our tool provides the flexibility to test various batch sizes, learning rates, and GPU configurations effortlessly. Users can reuse completed benchmarks, eliminating the need to repeatedly input the same parameters, streamlining the benchmarking process.",[1230,16348,16350],{"id":16349},"_5-scalable-execution",[14,16351,16352],{},"5. Scalable Execution",[18,16354,16355],{},"Our tool adapts to your hardware (thanks to dstack), whether you’re running benchmarks on a local machine, cloud instance, or multi-GPU cluster.",[1230,16357,16359],{"id":16358},"_6-auto-generated-performance-reports",[14,16360,16361],{},"6. Auto-Generated Performance Reports",[18,16363,16364],{},"After execution, performance data is logged and can be exported:",[18,16366,16367],{},[283,16368],{"alt":285,"src":16369},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fgraphexamplebenchtool.jpg",[18,16371,16372],{},"On top of that, it also auto generates a markdown table:",[18,16374,16375],{},[283,16376],{"alt":285,"src":16377},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fmarkdowntablebenchtool.jpg",[10,16379,16381],{"id":16380},"_5-how-it-works",[14,16382,16383],{},"5. How It Works",[1230,16385,16387],{"id":16386},"step-1-configure-your-benchmark",[14,16388,16389],{},"Step 1: Configure Your Benchmark",[18,16391,16392],{},"By default the tool fills in the fields for a standard benchmark on top of vllm. You can adjust the parameters, models, arguments as you need using the interface.",[18,16394,16395],{},"There are a few environment variables that can be used in the arguments, as well as the option to use .. and | to multi-schedule.",[1230,16397,16399],{"id":16398},"step-2-run-the-benchmark",[14,16400,16401],{},"Step 2: Run the Benchmark",[18,16403,16404],{},"Once the setup is filled in, you schedule the benchmark. The button will show, based on the configuration, how many benchmarks will run.",[18,16406,16407],{},"For example, if you fill in num prompts with: 10|20|30 it will schedule 3 benchmarks.",[18,16409,16410],{},"If on top of that you want to test various –num-scheduler-steps 10|20|30 then it will result in 9 benchmarks, one for each of the possible combinations.",[1230,16412,16414],{"id":16413},"step-3-view-results",[14,16415,16416],{},"Step 3: View Results",[18,16418,16419],{},"When all benchmarks are complete you can click on the row to see graphs regarding gpu usage and energy consumption. Selecting a row will also compare various parameters against other runs.",[18,16421,16422],{},"To export, tick the box before every benchmark and press export, a new window will open with a graph and results table ready to be pasted in a markdown document!",[1230,16424,16426],{"id":16425},"benchmark-tool-walk-through-video-by-rob-h",[14,16427,16428],{},"Benchmark tool walk-through video by Rob H..",[18,16430,16431],{},[163,16432,16435],{"href":16433,"rel":16434},"https:\u002F\u002Fwww.youtube.com\u002Fembed\u002FTenPJk6k7Xw?feature=oembed",[167],"Eliovp Benchmark Tool",[18,16437,16438],{},[14,16439,16440],{},"6. Comparing our benchmark tool",[298,16442,16443,16461],{},[301,16444,16445],{},[304,16446,16447,16451,16456],{},[307,16448,16449],{},[14,16450,6215],{},[307,16452,16453],{},[14,16454,16455],{},"Existing Tools",[307,16457,16458],{},[14,16459,16460],{},"Our Benchmarking Tool",[316,16462,16463,16478,16493,16507,16522],{},[304,16464,16465,16468,16473],{},[321,16466,16467],{},"Automated Execution",[321,16469,16470],{},[223,16471,16472],{},"Manual setup",[321,16474,16475],{},[223,16476,16477],{},"Fully automated",[304,16479,16480,16483,16488],{},[321,16481,16482],{},"Reproducibility",[321,16484,16485],{},[223,16486,16487],{},"Environment-dependent",[321,16489,16490],{},[223,16491,16492],{},"Containerized environments",[304,16494,16495,16498,16502],{},[321,16496,16497],{},"Cloud Support",[321,16499,16500],{},[223,16501,6026],{},[321,16503,16504],{},[223,16505,16506],{},"Works locally & in cloud",[304,16508,16509,16512,16517],{},[321,16510,16511],{},"CI\u002FCD Integration",[321,16513,16514],{},[223,16515,16516],{},"Manual testing",[321,16518,16519],{},[223,16520,16521],{},"Seamless integration",[304,16523,16524,16527,16532],{},[321,16525,16526],{},"vLLM-Based Benchmarking",[321,16528,16529],{},[223,16530,16531],{},"Not always supported",[321,16533,16534],{},[223,16535,16536],{},"Fully integrated with vLLM",[10,16538,16540],{"id":16539},"_7-real-world-use-casesai-model-benchmarking",[14,16541,16542],{},"7. Real-World Use CasesAI Model Benchmarking",[30,16544,16545],{},[33,16546,16547],{},"Compare model inference speeds using vLLM across different hardware configurations.",[18,16549,16550],{},[14,16551,16552],{},"Software Performance Testing",[30,16554,16555],{},[33,16556,16557],{},"Measure system performance across hardware configurations and different optimization settings.",[18,16559,16560],{},[14,16561,16562],{},"Continuous Integration (CI) Benchmarking",[30,16564,16565],{},[33,16566,16567],{},"Integrate into CI\u002FCD pipelines to fail builds if performance regresses.",[10,16569,16571],{"id":16570},"_8-conclusion",[14,16572,16573],{},"8. Conclusion",[18,16575,16576],{},"Benchmarking doesn’t have to be complex. Our new tool, powered by dstack, brings automation, reproducibility, and scalability to the benchmarking process. By leveraging vLLM, we ensure that our benchmarks maintain consistency, reliability, and broad applicability across different hardware configurations.",[18,16578,16579],{},"Yes, this was just a Blog where we boast about our great benchmarking tool. But we’re thinking about open-sourcing it, let us know if you’re interested!",{"title":285,"searchDepth":673,"depth":673,"links":16581},[16582,16583,16584,16585,16586,16587,16588],{"id":12255,"depth":676,"text":12258},{"id":16194,"depth":676,"text":16197},{"id":16233,"depth":676,"text":16236},{"id":16297,"depth":676,"text":16300},{"id":16380,"depth":676,"text":16383},{"id":16539,"depth":676,"text":16542},{"id":16570,"depth":676,"text":16573},[688,689,68,9658,690,16590,16591,694,68],"benchmark","LLM","2025-03-20T14:21:59","1. Introduction Benchmarking is an essential part of optimizing AI models and software applications. Whether you’re testing AI model inference speeds, profiling different hardware configurations, or ensuring system performance over time, having a reliable benchmarking tool is crucial. However, many existing tools suffer from issues like inconsistent environments, difficult configuration setups, and lack of automation. ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fbenchmarktool.jpeg",{},"https:\u002F\u002Feliovp.com\u002Fintroducing-our-benchmarking-tool-powered-by-dstack\u002F","\u002Fblog\u002Fintroducing-our-benchmarking-tool-powered-by-dstack",{"title":16176,"description":16593},"introducing-our-benchmarking-tool-powered-by-dstack","blog\u002Fintroducing-our-benchmarking-tool-powered-by-dstack","7KvYSwLAc-Dsl2-yAMD9M9MiQOaNMSzyCyxr_z0FfJg",{"id":16603,"title":16604,"body":16605,"categories":17543,"date":17544,"description":17545,"extension":699,"image":17546,"meta":17547,"navigation":702,"originalUrl":17548,"path":17549,"seo":17550,"slug":17551,"stem":17552,"__hash__":17553},"blog\u002Fblog\u002Foptimizing-qwq-32b-by-qwen-amd-mi300x-vs-nvidia-h200.md","Optimizing QwQ-32B (by Qwen): AMD MI300X vs. NVIDIA H200",{"type":7,"value":16606,"toc":17531},[16607,16611,16614,16619,16647,16653,16666,16671,16705,16708,16714,16720,16740,16745,16748,16753,16761,16767,16769,16775,16778,16784,16790,16916,16922,17047,17053,17180,17182,17188,17193,17198,17205,17207,17213,17218,17223,17228,17233,17238,17243,17245,17251,17256,17261,17271,17273,17279,17283,17288,17295,17297,17303,17309,17320,17325,17329,17334,17341,17343,17349,17357,17362,17367,17372,17379,17381,17387,17398,17403,17407,17412,17423,17425,17434,17436,17442,17462,17465,17467,17473,17498,17500,17504,17507,17518,17521,17523,17526],[10,16608,16609],{"id":12255},[14,16610,12258],{},[18,16612,16613],{},"In the world of large language models (LLMs), most benchmarks center on Llama or DeepSeek derivatives. We decided to diversify by adding the Qwen2 architecture, using our Paiton framework. This 32-billion-parameter model pushes GPU resources to the limit, perfect for comparing NVIDIA’s new H200 to our AMD MI300X, which leverages Paiton for advanced concurrency and custom kernel compilation.",[18,16615,16616],{},[14,16617,16618],{},"Why QwQ-32B?",[30,16620,16621,16627,16633],{},[33,16622,16623,16626],{},[14,16624,16625],{},"High-Parameter Model",": 32B parameters can handle intricate dialogues, domain-specific knowledge, and advanced reasoning.",[33,16628,16629,16632],{},[14,16630,16631],{},"Less Common",": Goes beyond the typical Llama\u002FDeepSeek solutions, letting us highlight Paiton’s flexibility in optimizing a broader range of LLMs.",[33,16634,16635,16638,16639,16641,16642],{},[14,16636,16637],{},"Incredible results",": QwQ-32B, specifically, is showing great results compared to Deepseek-R1, GPT-4o, Claude 3.7 and other models which are relatively large.",[1671,16640],{},"These results can be found here: ",[163,16643,16646],{"href":16644,"rel":16645},"https:\u002F\u002Fartificialanalysis.ai\u002Fmodels\u002Fqwq-32b",[167],"Artificial Analysis AI",[10,16648,16650],{"id":16649},"_2-paiton-vllm-versionspaiton-on-amd-mi300x",[14,16651,16652],{},"2. Paiton & vLLM VersionsPaiton on AMD MI300X",[30,16654,16655,16660],{},[33,16656,16657,16659],{},[14,16658,696],{},": We used vLLM 0.7.3 for Paiton. Although newer versions (like 0.8.0) exist, our test system was configured with 0.7.3, which still benefits from many concurrency improvements.",[33,16661,16662,16665],{},[14,16663,16664],{},"Flags",": Standard environment, HIP-based concurrency, no hidden toggles besides Paiton’s kernel fusion.",[18,16667,16668],{},[14,16669,16670],{},"Stock Model Runs",[30,16672,16673,16686],{},[33,16674,16675,16677,16678],{},[14,16676,6102],{},": Running vLLM 0.8.0 with the best tested options:\n",[30,16679,16680,16683],{},[33,16681,16682],{},"CUDA Graph capturing enabled (via default or explicit flags)",[33,16684,16685],{},"“V1” concurrency scheduling (default in vLLM 0.8.0, if use-case supported)",[33,16687,16688,16691,16692],{},[14,16689,16690],{},"AMD MI300X (Stock)",": For direct “stock vs. Paiton” comparison, we also tested the same vLLM version as Paiton (vLLM 0.7.3) with “best” possible flags:\n",[30,16693,16694,16699,16702],{},[33,16695,16696],{},[223,16697,16698],{},"--num_sched_steps=10",[33,16700,16701],{},"CUDA Graph capturing (where relevant)",[33,16703,16704],{},"Disabling or enabling certain features for optimal concurrency",[18,16706,16707],{},"We wanted to ensure each GPU had the best possible environment.",[18,16709,16710,16713],{},[14,16711,16712],{},"Note",": On the AMD side (stock runs), we do not apply our custom kernel compilation or concurrency logic from Paiton, so we can see the native performance of the MI300X.",[10,16715,16717],{"id":16716},"_3-test-setup-methodologyhardware",[14,16718,16719],{},"3. Test Setup & MethodologyHardware",[30,16721,16722,16727],{},[33,16723,16724,16726],{},[14,16725,6102],{}," (single GPU)",[33,16728,16729,16731,16732],{},[14,16730,191],{}," (single GPU), tested in two modes:\n",[114,16733,16734,16737],{},[33,16735,16736],{},"Paiton-optimized",[33,16738,16739],{},"Stock vLLM, same version as Paiton (0.7.3), with best concurrency flags.",[18,16741,16742],{},[14,16743,16744],{},"Model & Command",[18,16746,16747],{},"QwQ-32B served with:",[18,16749,16750],{},[223,16751,16752],{},"vllm serve Qwen\u002FQwQ-32B --num-scheduler-steps=10 --swap-space=16 ...",[30,16754,16755,16758],{},[33,16756,16757],{},"For the H200, we used the default concurrency approach (-tp 1 or similar) plus “V1 scheduling.”",[33,16759,16760],{},"Batch sizes tested: from 1 up to 128.",[18,16762,16763,16766],{},[14,16764,16765],{},"Metrics",": Requests\u002Fs, Output Token Throughput, TTFT, TPOT, ITL, and E2E Latency.",[71,16768],{},[10,16770,16772],{"id":16771},"_4-results-stock-model-runs-vs-paiton",[14,16773,16774],{},"4. Results: Stock Model Runs vs. Paiton",[18,16776,16777],{},"We’ve split the performance data into two main tables, one for Throughput metrics and one for Latency metrics. Afterward, you’ll see Graphs integrated into the text, each accompanied by Notes on how they illustrate the comparative performance of AMD MI300X vs. NVIDIA H200.",[10,16779,16781],{"id":16780},"_41-throughput-tables",[14,16782,16783],{},"4.1 Throughput Tables",[1230,16785,16787],{"id":16786},"stock-model-amd-mi300x",[14,16788,16789],{},"Stock Model: AMD MI300X",[298,16791,16792,16812],{},[301,16793,16794],{},[304,16795,16796,16800,16804,16808],{},[307,16797,16798],{},[14,16799,6775],{},[307,16801,16802],{},[14,16803,373],{},[307,16805,16806],{},[14,16807,387],{},[307,16809,16810],{},[14,16811,401],{},[316,16813,16814,16826,16838,16851,16864,16877,16890,16903],{},[304,16815,16816,16818,16820,16823],{},[321,16817,6802],{},[321,16819,12592],{},[321,16821,16822],{},"48.02",[321,16824,16825],{},"52.86",[304,16827,16828,16830,16832,16835],{},[321,16829,6819],{},[321,16831,12499],{},[321,16833,16834],{},"49.51",[321,16836,16837],{},"51.52",[304,16839,16840,16842,16845,16848],{},[321,16841,6836],{},[321,16843,16844],{},"0.22",[321,16846,16847],{},"73.42",[321,16849,16850],{},"77.38",[304,16852,16853,16855,16858,16861],{},[321,16854,6853],{},[321,16856,16857],{},"0.44",[321,16859,16860],{},"120.28",[321,16862,16863],{},"183.72",[304,16865,16866,16868,16871,16874],{},[321,16867,6870],{},[321,16869,16870],{},"0.96",[321,16872,16873],{},"220.17",[321,16875,16876],{},"412.65",[304,16878,16879,16881,16884,16887],{},[321,16880,349],{},[321,16882,16883],{},"1.79",[321,16885,16886],{},"403.23",[321,16888,16889],{},"829.22",[304,16891,16892,16894,16897,16900],{},[321,16893,6903],{},[321,16895,16896],{},"3.21",[321,16898,16899],{},"676.22",[321,16901,16902],{},"1467.65",[304,16904,16905,16907,16910,16913],{},[321,16906,6920],{},[321,16908,16909],{},"4.94",[321,16911,16912],{},"1079.45",[321,16914,16915],{},"2226.07",[1230,16917,16919],{"id":16918},"stock-model-nvidia-h200",[14,16920,16921],{},"Stock Model: NVIDIA H200",[298,16923,16924,16944],{},[301,16925,16926],{},[304,16927,16928,16932,16936,16940],{},[307,16929,16930],{},[14,16931,6775],{},[307,16933,16934],{},[14,16935,373],{},[307,16937,16938],{},[14,16939,387],{},[307,16941,16942],{},[14,16943,401],{},[316,16945,16946,16959,16971,16983,16995,17008,17021,17034],{},[304,16947,16948,16950,16953,16956],{},[321,16949,6802],{},[321,16951,16952],{},"0.42",[321,16954,16955],{},"49.74",[321,16957,16958],{},"54.76",[304,16960,16961,16963,16965,16968],{},[321,16962,6819],{},[321,16964,12529],{},[321,16966,16967],{},"57.84",[321,16969,16970],{},"60.18",[304,16972,16973,16975,16977,16980],{},[321,16974,6836],{},[321,16976,12575],{},[321,16978,16979],{},"85.11",[321,16981,16982],{},"89.71",[304,16984,16985,16987,16989,16992],{},[321,16986,6853],{},[321,16988,12621],{},[321,16990,16991],{},"142.21",[321,16993,16994],{},"217.21",[304,16996,16997,16999,17002,17005],{},[321,16998,6870],{},[321,17000,17001],{},"1.01",[321,17003,17004],{},"231.77",[321,17006,17007],{},"434.38",[304,17009,17010,17012,17015,17018],{},[321,17011,349],{},[321,17013,17014],{},"1.86",[321,17016,17017],{},"418.62",[321,17019,17020],{},"860.89",[304,17022,17023,17025,17028,17031],{},[321,17024,6903],{},[321,17026,17027],{},"3.44",[321,17029,17030],{},"728.66",[321,17032,17033],{},"1574.56",[304,17035,17036,17038,17041,17044],{},[321,17037,6920],{},[321,17039,17040],{},"6.00",[321,17042,17043],{},"1311.51",[321,17045,17046],{},"2704.78",[1230,17048,17050],{"id":17049},"paiton-on-amd-mi300x",[14,17051,17052],{},"Paiton on AMD MI300X",[298,17054,17055,17075],{},[301,17056,17057],{},[304,17058,17059,17063,17067,17071],{},[307,17060,17061],{},[14,17062,6775],{},[307,17064,17065],{},[14,17066,373],{},[307,17068,17069],{},[14,17070,387],{},[307,17072,17073],{},[14,17074,401],{},[316,17076,17077,17089,17102,17115,17128,17141,17154,17167],{},[304,17078,17079,17081,17083,17086],{},[321,17080,6802],{},[321,17082,16857],{},[321,17084,17085],{},"52.03",[321,17087,17088],{},"57.27",[304,17090,17091,17093,17096,17099],{},[321,17092,6819],{},[321,17094,17095],{},"0.14",[321,17097,17098],{},"60.27",[321,17100,17101],{},"62.71",[304,17103,17104,17106,17109,17112],{},[321,17105,6836],{},[321,17107,17108],{},"0.27",[321,17110,17111],{},"87.92",[321,17113,17114],{},"92.67",[304,17116,17117,17119,17122,17125],{},[321,17118,6853],{},[321,17120,17121],{},"0.53",[321,17123,17124],{},"144.27",[321,17126,17127],{},"220.35",[304,17129,17130,17132,17135,17138],{},[321,17131,6870],{},[321,17133,17134],{},"1.02",[321,17136,17137],{},"234.10",[321,17139,17140],{},"438.76",[304,17142,17143,17145,17148,17151],{},[321,17144,349],{},[321,17146,17147],{},"1.90",[321,17149,17150],{},"427.54",[321,17152,17153],{},"879.22",[304,17155,17156,17158,17161,17164],{},[321,17157,6903],{},[321,17159,17160],{},"3.40",[321,17162,17163],{},"715.60",[321,17165,17166],{},"1552.62",[304,17168,17169,17171,17174,17177],{},[321,17170,6920],{},[321,17172,17173],{},"5.56",[321,17175,17176],{},"1211.53",[321,17178,17179],{},"2503.30",[71,17181],{},[1230,17183,17185],{"id":17184},"graph-1-requestss-vs-batch-size",[14,17186,17187],{},"Graph 1: Requests\u002Fs vs. Batch Size",[18,17189,17190],{},[630,17191,17192],{},"(Higher is better.)",[18,17194,17195],{},[283,17196],{"alt":285,"src":17197},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fgraph1_requests_vs_batch.jpg",[18,17199,17200],{},[630,17201,17202,17204],{},[14,17203,16712],{},": Notice how at small-to-mid batch sizes (e.g., 1 to 32), Paiton on MI300X matches or exceeds the H200’s requests-per-second. As batch size grows beyond 64, the H200 still holds a slight lead, but Paiton narrows the gap compared to the “stock” AMD runs. This underscores AMD MI300X’s capability to outperform or closely match NVIDIA H200 under many concurrency conditions when optimizations (Paiton) are applied.",[71,17206],{},[10,17208,17210],{"id":17209},"_42-latency-tables",[14,17211,17212],{},"4.2 Latency Tables",[1230,17214,17216],{"id":17215},"stock-model-amd-mi300x-1",[14,17217,16789],{},[18,17219,17220],{},[283,17221],{"alt":285,"src":17222},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fstockamdmi300xtable.jpg",[1230,17224,17226],{"id":17225},"stock-model-nvidia-h200-1",[14,17227,16921],{},[18,17229,17230],{},[283,17231],{"alt":285,"src":17232},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fh200table.jpg",[1230,17234,17236],{"id":17235},"paiton-on-amd-mi300x-1",[14,17237,17052],{},[18,17239,17240],{},[283,17241],{"alt":285,"src":17242},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fpaitontable.jpg",[71,17244],{},[1230,17246,17248],{"id":17247},"graph-2-time-to-first-token-ttft-vs-batch-size",[14,17249,17250],{},"Graph 2: Time-to-First-Token (TTFT) vs. Batch Size",[18,17252,17253],{},[630,17254,17255],{},"(Lower is better)",[18,17257,17258],{},[283,17259],{"alt":285,"src":17260},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fgraph2_ttft_vs_batch.jpg",[18,17262,17263],{},[630,17264,17265,17267,17268,17270],{},[14,17266,16712],{},": TTFT is the delay before a single token is produced. At low batch sizes, H200 is slightly ahead. However, as soon as we apply ",[14,17269,68],{}," on the MI300X, the difference at small concurrency shrinks. The data also shows that once you move beyond very small batch sizes (e.g., 8 or 16), AMD + Paiton can keep TTFT in a competitive range.",[71,17272],{},[1230,17274,17276],{"id":17275},"graph-3-mean-end-to-end-latency-e2e-vs-batch-size",[14,17277,17278],{},"Graph 3: Mean End-to-End Latency (E2E) vs. Batch Size",[18,17280,17281],{},[630,17282,17255],{},[18,17284,17285],{},[283,17286],{"alt":285,"src":17287},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fgraph3_mean_e2e_vs_batch.jpg",[18,17289,17290],{},[630,17291,17292,17294],{},[14,17293,16712],{},": E2E Latency accounts for the entire request: from the time the prompt is submitted to the time all tokens are generated. With Paiton optimizations, the MI300X consistently lowers E2E latency compared to stock AMD results and can rival the H200 in many batch-size scenarios.",[71,17296],{},[10,17298,17300],{"id":17299},"_5-analysis-observations",[14,17301,17302],{},"5. Analysis & Observations",[1230,17304,17306],{"id":17305},"comparison-at-small-batches",[14,17307,17308],{},"Comparison at Small Batches",[30,17310,17311,17317],{},[33,17312,17313,17316],{},[14,17314,17315],{},"Batch=1 or 2",": Paiton on MI300X and the H200 are extremely close in throughput (~0.4–0.44 req\u002Fs).",[33,17318,17319],{},"TTFT for Paiton on MI300X is somewhat higher (61 ms vs. 44 ms on H200 at batch=1), but the overall E2EL remains in the same range (~2.29–2.48 s).",[18,17321,17322],{},[14,17323,17324],{},"Graph 4: Requests\u002Fs vs. E2E Latency (small batches)",[18,17326,17327],{},[630,17328,17255],{},[18,17330,17331],{},[283,17332],{"alt":285,"src":17333},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fgraph4_small_batches.jpg",[18,17335,17336],{},[630,17337,17338,17340],{},[14,17339,16712],{},": If we specifically plot requests\u002Fs against end-to-end latency for small batch sizes (1–8), the MI300X with Paiton can not only match but, in some runs, exceed the concurrency of the H200. This is a key example of how targeted optimizations can push AMD’s hardware further.",[71,17342],{},[1230,17344,17346],{"id":17345},"mid-batches-1632",[14,17347,17348],{},"Mid Batches (16–32)",[30,17350,17351,17354],{},[33,17352,17353],{},"AMD + Paiton has a slight edge in req\u002Fs (1.9 vs. 1.79 at batch=32, for instance) over AMD Stock and is catching up to H200’s concurrency.",[33,17355,17356],{},"TTFT sees a jump (472–928 ms range), which is typical for big LLMs, but Paiton’s concurrency approach helps keep total tokens\u002Fs high.",[18,17358,17359],{},[14,17360,17361],{},"Graph 5: Total Throughput (tok\u002Fs) vs. Batch Size",[18,17363,17364],{},[630,17365,17366],{},"(Higher**is better)",[18,17368,17369],{},[283,17370],{"alt":285,"src":17371},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fgraph5_total_throughput_vs_batch.jpg",[18,17373,17374],{},[630,17375,17376,17378],{},[14,17377,16712],{},": Total token throughput (input + output tokens) scales significantly with concurrency. Notice how Paiton on MI300X actually outperforms the H200 at certain mid-range batches in terms of tokens processed per second, showcasing that the MI300X architecture + Paiton’s kernel fusion can shine when carefully tuned.",[71,17380],{},[1230,17382,17384],{"id":17383},"high-batches-64128",[14,17385,17386],{},"High Batches (64–128)",[30,17388,17389,17395],{},[33,17390,17391,17394],{},[14,17392,17393],{},"H200 leads"," (for now) at batch=128 with 6.0 req\u002Fs vs. 5.56 on Paiton, but Paiton outperforms the stock AMD MI300X at 4.94 req\u002Fs.",[33,17396,17397],{},"E2EL can still be ~9–12 seconds at these large batches, consistent with the heavy token generation tasks for a 32B model.",[18,17399,17400],{},[14,17401,17402],{},"Graph 6: P99 End-to-End Latency vs. Batch Size",[18,17404,17405],{},[630,17406,17255],{},[18,17408,17409],{},[283,17410],{"alt":285,"src":17411},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fgraph6_p99_e2e_latency_vs_batch-1.jpg",[18,17413,17414],{},[630,17415,17416,17418,17419,17422],{},[14,17417,16712],{},": The P99 E2E Latency at very high concurrency is understandably large. Even though the H200 tops out with a slightly higher request throughput, MI300X with Paiton significantly narrows the performance gap. This points to the strong potential of further concurrency refinements, eventually ",[14,17420,17421],{},"surpassing"," NVIDIA’s performance in certain workloads.",[71,17424],{},[18,17426,17427,17429,17430,17433],{},[14,17428,3515],{},": While H200 might show a slight lead at the highest concurrency, ",[14,17431,17432],{},"Paiton significantly boosts the AMD MI300X performance"," beyond the stock approach, especially at small to mid batch sizes. This suggests real-world AI clusters could see better scaling using Paiton if concurrency or batch scheduling is carefully tuned.",[71,17435],{},[10,17437,17439],{"id":17438},"_6-emphasizing-amds-strength-ongoing-improvements",[14,17440,17441],{},"6. Emphasizing AMD’s Strength & Ongoing Improvements",[30,17443,17444,17450,17456],{},[33,17445,17446,17449],{},[14,17447,17448],{},"HBM Advantage",": The MI300X’s large HBM memory ensures stable performance across a wide token range; even at bigger batch sizes, the system remains stable.",[33,17451,17452,17455],{},[14,17453,17454],{},"Kernel Compilation",": Paiton’s custom approach yields better concurrency, especially for smaller batch operations, bridging typical overhead.",[33,17457,17458,17461],{},[14,17459,17460],{},"Not “There” Yet",": In some highest-batch cases, the H200 retains an edge, but we’re actively refining Paiton’s kernel launching and concurrency strategies to close that gap.",[18,17463,17464],{},"QwQ-32B was less commonly benchmarked, so supporting it within Paiton showcases our library’s model-agnostic design. Expect further gains as we adopt or adapt to vLLM 0.8.x concurrency improvements on AMD.",[71,17466],{},[10,17468,17470],{"id":17469},"_7-next-steps-future-work",[14,17471,17472],{},"7. Next Steps & Future Work",[114,17474,17475,17480,17486,17492],{},[33,17476,17477,17479],{},[14,17478,618],{},": Evaluating multi-GPU scaling for QwQ-32B, especially if AI clusters want sub-2 second latencies.",[33,17481,17482,17485],{},[14,17483,17484],{},"Advanced Quantization",": 8-bit or 4-bit compression to reduce memory footprint and further speed up inference.",[33,17487,17488,17491],{},[14,17489,17490],{},"Further Kernel Optimizations",": Utilize our current and new optimization techniques to push the hardware even further.",[33,17493,17494,17497],{},[14,17495,17496],{},"RAG",": How Paiton improves Retrieval Augmented Generation.",[71,17499],{},[10,17501,17502],{"id":16570},[14,17503,16573],{},[18,17505,17506],{},"QwQ-32B offers a fresh perspective on large-scale inference beyond common Llama-based benchmarks. Our results show:",[30,17508,17509,17512,17515],{},[33,17510,17511],{},"Paiton optimizing the AMD MI300X to significantly outperform “stock” AMD runs.",[33,17513,17514],{},"Competitive or better performance at small to mid batch sizes vs. NVIDIA’s H200.",[33,17516,17517],{},"Room to improve further at the larger batch sizes to beat the H200, which we’ll address with advanced concurrency.",[18,17519,17520],{},"Stay tuned for more updates on further integration and optimizations, cluster concurrency testing, and new optimization breakthroughs for AMD-based solutions. If you have any questions or want to see specific tests, reach out, we’re committed to refining Paiton for the widest range of models and hardware.",[71,17522],{},[18,17524,17525],{},"Here we’ve demonstrated how crucial concurrency and kernel-level optimizations are in closing (and often exceeding) the performance gap with NVIDIA’s latest hardware. With Paiton, the AMD MI3** series is destined to become a top choice for high-throughput, large-model inference.",[18,17527,17528],{},[14,17529,17530],{},"Thank you for reading!– The Paiton Team –",{"title":285,"searchDepth":673,"depth":673,"links":17532},[17533,17534,17535,17536,17537,17538,17539,17540,17541,17542],{"id":12255,"depth":676,"text":12258},{"id":16649,"depth":676,"text":16652},{"id":16716,"depth":676,"text":16719},{"id":16771,"depth":676,"text":16774},{"id":16780,"depth":676,"text":16783},{"id":17209,"depth":676,"text":17212},{"id":17299,"depth":676,"text":17302},{"id":17438,"depth":676,"text":17441},{"id":17469,"depth":676,"text":17472},{"id":16570,"depth":676,"text":16573},[688,689,68],"2025-03-19T21:41:44","1. Introduction In the world of large language models (LLMs), most benchmarks center on Llama or DeepSeek derivatives. We decided to diversify by adding the Qwen2 architecture, using our Paiton framework. This 32-billion-parameter model pushes GPU resources to the limit, perfect for comparing NVIDIA’s new H200 to our AMD MI300X, which leverages Paiton for advanced ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fpaiton4.jpeg",{},"https:\u002F\u002Feliovp.com\u002Foptimizing-qwq-32b-by-qwen-amd-mi300x-vs-nvidia-h200\u002F","\u002Fblog\u002Foptimizing-qwq-32b-by-qwen-amd-mi300x-vs-nvidia-h200",{"title":16604,"description":17545},"optimizing-qwq-32b-by-qwen-amd-mi300x-vs-nvidia-h200","blog\u002Foptimizing-qwq-32b-by-qwen-amd-mi300x-vs-nvidia-h200","jXE9BjCAhzOtHfoRvCJv_evHKwpakueK7K4MdN0m5CA",{"id":17555,"title":17556,"body":17557,"categories":17600,"date":17604,"description":17605,"extension":699,"image":17606,"meta":17607,"navigation":702,"originalUrl":17608,"path":17609,"seo":17610,"slug":17611,"stem":17612,"__hash__":17613},"blog\u002Fblog\u002Feliovp-featured-on-amd-tech-talk-podcast.md","Eliovp Featured on AMD “Tech Talk” Podcast",{"type":7,"value":17558,"toc":17598},[17559,17562,17565,17568,17571,17591],[18,17560,17561],{},"We’re excited to share that Eliovp was recently featured on AMD’s “Tech Talk” podcast!",[18,17563,17564],{},"In this episode, our CEO, Elio Van Puyvelde sits down with Jim greene to talk about the origins of Eliovp, the passion and expertise that brought the company to life, and the innovative full end-to-end solutions we offer today.",[18,17566,17567],{},"From our humble beginnings to our current position at the forefront of cutting-edge technology solutions, we’ve always maintained a commitment to excellence and customer-centric service.",[18,17569,17570],{},"During the conversation, we highlighted how Eliovp has grown and evolved, providing comprehensive hardware and software services that cater to a wide range of AI, GPU, and machine learning needs. We discussed our focus on building scalable solutions, our dedication to continuous innovation, and the importance of working closely with clients to ensure they can fully harness the power of advanced computing.",[18,17572,17573,17574,17579,17580,17585,17586,17590],{},"Tune in to the full interview on ",[163,17575,17578],{"href":17576,"rel":17577},"https:\u002F\u002Fopen.spotify.com\u002Fepisode\u002F4spBelSPIHbXLSlDsXzbEP?si=290cc04a75334129&nd=1&dlsi=f6e00809d14741e3",[167],"Spotify"," or catch a recap on ",[163,17581,17584],{"href":17582,"rel":17583},"https:\u002F\u002Fwww.linkedin.com\u002Fposts\u002Fjames-greene-4000b43_ai-gpus-machinelearning-activity-7307787805802278912-TkMV?utm_source=share&utm_medium=member_desktop&rcm=ACoAADitiKQBlbe96FtSi7Tjp7NR3eLMAFADT9k",[167],"LinkedIn"," to learn more about our journey and where we’re headed next. We’re thrilled to continue expanding our capabilities and partnering with industry leaders like AMD to deliver top-tier solutions for our clients around the world. Keep an eye on ",[163,17587,17589],{"href":12180,"rel":17588},[167],"our website"," for more updates, and thank you for being a part of our story!",[18,17592,17593],{},[163,17594,17597],{"href":17595,"rel":17596},"https:\u002F\u002Fopen.spotify.com\u002Fembed\u002Fepisode\u002F4spBelSPIHbXLSlDsXzbEP?si=dEq0hCF-TS2uVOBoyw0tuw&utm_source=oembed",[167],"Spotify Embed: Data Centers in Shipping Containers: Thinking Outside the Box with ElioVP",{"title":285,"searchDepth":673,"depth":673,"links":17599},[],[688,690,17601,17602,17603],"Jim Greene","Podcast","Tech Talk","2025-03-19T07:53:39","We’re excited to share that Eliovp was recently featured on AMD’s “Tech Talk” podcast! In this episode, our CEO, Elio Van Puyvelde sits down with Jim greene to talk about the origins of Eliovp, the passion and expertise that brought the company to life, and the innovative full end-to-end solutions we offer today. From our ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Ftechtalkjimgreene.jpeg",{},"https:\u002F\u002Feliovp.com\u002Feliovp-featured-on-amd-tech-talk-podcast\u002F","\u002Fblog\u002Feliovp-featured-on-amd-tech-talk-podcast",{"title":17556,"description":17605},"eliovp-featured-on-amd-tech-talk-podcast","blog\u002Feliovp-featured-on-amd-tech-talk-podcast","2StUeZTqLxZeGOWLkcDTWjzyVng_zzHnAXBw3Ts4Nj4",{"id":4,"title":5,"body":17615,"categories":18089,"date":697,"description":698,"extension":699,"image":700,"meta":18090,"navigation":702,"originalUrl":703,"path":704,"seo":18091,"slug":706,"stem":707,"__hash__":708},{"type":7,"value":17616,"toc":18076},[17617,17621,17627,17647,17651,17653,17657,17661,17675,17677,17681,17713,17720,17722,17726,17796,17798,17802,17806,17808,17972,17976,17990,17992,17996,18000,18004,18008,18024,18026,18030,18044,18050,18052,18056,18058,18062,18066,18068,18072],[10,17618,17619],{"id":12},[14,17620,16],{},[18,17622,20,17623,24,17625,28],{},[14,17624,23],{},[14,17626,27],{},[30,17628,17629,17637,17643],{},[33,17630,17631,38,17633,42,17635,46],{},[14,17632,37],{},[14,17634,41],{},[14,17636,45],{},[33,17638,17639,52,17641,56],{},[14,17640,51],{},[14,17642,55],{},[33,17644,17645,62],{},[14,17646,61],{},[18,17648,65,17649,69],{},[14,17650,68],{},[71,17652],{},[10,17654,17655],{"id":75},[14,17656,78],{},[10,17658,17659],{"id":81},[14,17660,84],{},[30,17662,17663,17667,17671],{},[33,17664,17665,92],{},[14,17666,91],{},[33,17668,17669,98],{},[14,17670,97],{},[33,17672,17673,104],{},[14,17674,103],{},[71,17676],{},[10,17678,17679],{"id":109},[14,17680,112],{},[114,17682,17683,17695,17705],{},[33,17684,17685,17687],{},[14,17686,120],{},[30,17688,17689,17693],{},[33,17690,125,17691,129],{},[14,17692,128],{},[33,17694,132],{},[33,17696,17697,17699],{},[14,17698,137],{},[30,17700,17701,17703],{},[33,17702,142],{},[33,17704,145],{},[33,17706,17707,17709],{},[14,17708,150],{},[30,17710,17711],{},[33,17712,155],{},[18,17714,17715,161,17717,169],{},[14,17716,160],{},[163,17718,168],{"href":165,"rel":17719},[167],[71,17721],{},[10,17723,17724],{"id":174},[14,17725,177],{},[114,17727,17728,17742,17780],{},[33,17729,17730,17732],{},[14,17731,184],{},[30,17733,17734,17738],{},[33,17735,17736,192],{},[14,17737,191],{},[33,17739,17740,198],{},[14,17741,197],{},[33,17743,17744,17746],{},[14,17745,203],{},[30,17747,17748,17762,17766],{},[33,17749,17750,211,17752],{},[14,17751,210],{},[30,17753,17754,17756,17758],{},[33,17755,216],{},[33,17757,219],{},[33,17759,17760,226],{},[223,17761,225],{},[33,17763,17764,231],{},[14,17765,68],{},[33,17767,17768,237,17770],{},[14,17769,236],{},[30,17771,17772,17776,17778],{},[33,17773,242,17774,246],{},[14,17775,245],{},[33,17777,249],{},[33,17779,252],{},[33,17781,17782,237,17784],{},[14,17783,257],{},[30,17785,17786,17792],{},[33,17787,262,17788,266,17790,270],{},[14,17789,265],{},[14,17791,269],{},[33,17793,17794,276],{},[14,17795,275],{},[18,17797,279],{},[18,17799,17800],{},[283,17801],{"alt":285,"src":286},[288,17803,17804],{"id":290},[14,17805,293],{},[18,17807,296],{},[298,17809,17810,17822],{},[301,17811,17812],{},[304,17813,17814,17816,17818,17820],{},[307,17815],{},[307,17817],{},[307,17819],{},[307,17821],{},[316,17823,17824,17842,17852,17862,17872,17882,17892,17902,17912,17922,17932,17942,17952,17962],{},[304,17825,17826,17830,17834,17838],{},[321,17827,17828],{},[14,17829,325],{},[321,17831,17832,331],{},[14,17833,330],{},[321,17835,17836,331],{},[14,17837,336],{},[321,17839,17840],{},[14,17841,341],{},[304,17843,17844,17846,17848,17850],{},[321,17845,346],{},[321,17847,349],{},[321,17849,349],{},[321,17851,354],{},[304,17853,17854,17856,17858,17860],{},[321,17855,359],{},[321,17857,362],{},[321,17859,365],{},[321,17861,368],{},[304,17863,17864,17866,17868,17870],{},[321,17865,373],{},[321,17867,376],{},[321,17869,379],{},[321,17871,382],{},[304,17873,17874,17876,17878,17880],{},[321,17875,387],{},[321,17877,390],{},[321,17879,393],{},[321,17881,396],{},[304,17883,17884,17886,17888,17890],{},[321,17885,401],{},[321,17887,404],{},[321,17889,407],{},[321,17891,396],{},[304,17893,17894,17896,17898,17900],{},[321,17895,414],{},[321,17897,417],{},[321,17899,420],{},[321,17901,55],{},[304,17903,17904,17906,17908,17910],{},[321,17905,427],{},[321,17907,430],{},[321,17909,433],{},[321,17911,436],{},[304,17913,17914,17916,17918,17920],{},[321,17915,441],{},[321,17917,444],{},[321,17919,447],{},[321,17921,450],{},[304,17923,17924,17926,17928,17930],{},[321,17925,455],{},[321,17927,458],{},[321,17929,461],{},[321,17931,464],{},[304,17933,17934,17936,17938,17940],{},[321,17935,469],{},[321,17937,472],{},[321,17939,475],{},[321,17941,478],{},[304,17943,17944,17946,17948,17950],{},[321,17945,483],{},[321,17947,486],{},[321,17949,489],{},[321,17951,492],{},[304,17953,17954,17956,17958,17960],{},[321,17955,497],{},[321,17957,500],{},[321,17959,503],{},[321,17961,506],{},[304,17963,17964,17966,17968,17970],{},[321,17965,511],{},[321,17967,514],{},[321,17969,517],{},[321,17971,520],{},[18,17973,17974,526],{},[14,17975,525],{},[30,17977,17978,17982,17986],{},[33,17979,17980,534],{},[14,17981,533],{},[33,17983,17984,540],{},[14,17985,539],{},[33,17987,17988,546],{},[14,17989,545],{},[18,17991,549],{},[10,17993,17994],{"id":552},[14,17995,555],{},[18,17997,558,17998,562],{},[14,17999,561],{},[18,18001,18002],{},[283,18003],{"alt":285,"src":567},[10,18005,18006],{"id":570},[14,18007,573],{},[114,18009,18010,18016,18020],{},[33,18011,18012,581,18014,585],{},[14,18013,580],{},[14,18015,584],{},[33,18017,18018,591],{},[14,18019,590],{},[33,18021,18022,597],{},[14,18023,596],{},[71,18025],{},[10,18027,18028],{"id":602},[14,18029,605],{},[114,18031,18032,18036,18040],{},[33,18033,18034,613],{},[14,18035,612],{},[33,18037,18038,619],{},[14,18039,618],{},[33,18041,18042,625],{},[14,18043,624],{},[18,18045,628,18046,635],{},[630,18047,18048],{},[14,18049,634],{},[71,18051],{},[10,18053,18054],{"id":640},[14,18055,643],{},[18,18057,646],{},[18,18059,18060,652],{},[14,18061,651],{},[18,18063,655,18064,658],{},[14,18065,245],{},[71,18067],{},[18,18069,18070,666],{},[14,18071,665],{},[18,18073,18074],{},[14,18075,671],{},{"title":285,"searchDepth":673,"depth":673,"links":18077},[18078,18079,18080,18081,18082,18083],{"id":12,"depth":676,"text":16},{"id":75,"depth":676,"text":78},{"id":81,"depth":676,"text":84},{"id":109,"depth":676,"text":112},{"id":174,"depth":676,"text":177},{"id":290,"depth":673,"text":293,"children":18084},[18085,18086,18087,18088],{"id":552,"depth":676,"text":555},{"id":570,"depth":676,"text":573},{"id":602,"depth":676,"text":605},{"id":640,"depth":676,"text":643},[688,689,68,690,691,692,693,694,695,68,696],{},{"title":5,"description":698},{"id":18093,"title":18094,"body":18095,"categories":19870,"date":19871,"description":19872,"extension":699,"image":19873,"meta":19874,"navigation":702,"originalUrl":15977,"path":19875,"seo":19876,"slug":19877,"stem":19878,"__hash__":19879},"blog\u002Fblog\u002Fa-first-look-at-paiton-in-action-deepseek-r1-distill-llama-3-1-8b.md","A First Look at Paiton in Action: Deepseek R1 Distill Llama 3.1 8B",{"type":7,"value":18096,"toc":19836},[18097,18101,18105,18114,18117,18129,18132,18156,18169,18175,18179,18194,18198,18209,18219,18225,18290,18293,18299,18303,18306,18338,18341,18830,18834,18897,18902,18906,18914,18919,18924,18952,18958,18961,18981,18985,18991,19315,19319,19327,19331,19350,19354,19382,19387,19392,19414,19420,19424,19427,19431,19438,19441,19444,19458,19461,19466,19472,19476,19479,19515,19522,19526,19529,19540,19545,19549,19569,19573,19576,19596,19603,19609,19613,19635,19639,19667,19670,19695,19699,19713,19717,19725,19727,19733,19736,19756,19758,19789,19795,19797,19801,19823,19832],[288,18098,18100],{"id":18099},"outperforming-stock-models-on-the-amd-mi300x","Outperforming Stock Models on the AMD MI300X",[10,18102,18103],{"id":12255},[14,18104,12258],{},[18,18106,18107,18108,18113],{},"We couldn’t wait to show what Paiton can really do. After detailing our AMD-centric approach and architecture-level optimizations in our ",[163,18109,18112],{"href":18110,"rel":18111},"https:\u002F\u002Feliovp.com\u002Fai-model-optimization-with-paiton\u002F",[167],"previous blog post",", we decided to test-drive Paiton on a hype-worthy model: Deepseek R1 Distill Llama 3.1 8B. By compiling the model into efficient libraries and fusing custom kernels, the core ideas behind Paiton, we eliminate common overheads (e.g., repeated warm-ups, CUDA graph capturing) that often degrade real-world AI performance. Our goal was straightforward: see how a Paiton-optimized version of Deepseek R1 stacks up against a stock version when running on AMD MI300X hardware.",[18,18115,18116],{},"For our latest online serving tests, we turned to vLLM turning off eager mode (cuda graph active), capturing genuine “first-run” performance. Our model of choice is Deepseek R1 Distill Llama 3.1 8B, a specialized Llama variant designed to minimize latency and compute cost. Below, we compare:",[114,18118,18119,18124],{},[33,18120,18121,18123],{},[14,18122,8646],{}," Deepseek R1 – Distill Llama 3.1 8B",[33,18125,18126,18123],{},[14,18127,18128],{},"Paiton-Optimized",[18,18130,18131],{},"We want to emphasize that for the stock model, we enabled all “bells and whistles” that vLLM and our environment could offer, thread tuning, pinned GPU memory, concurrency optimizations, setting all the popular flags to ensure it was as optimized as possible. Despite these efforts, Paiton still comes out on top, particularly for higher batch sizes.",[18,18133,18134,18135,18137,18138,18140,18141,18144,18145,18147,18148,18151,18152,18155],{},"We first examine ",[14,18136,9250],{}," (requests\u002Fs, tokens\u002Fs), then move on to ",[14,18139,9272],{}," (TTFT, TPOT, ITL), culminating in a look at ",[14,18142,18143],{},"E2E latency"," graphs. The short story? ",[14,18146,68],{}," provides consistently better real-world performance overall, and at ",[14,18149,18150],{},"larger batch sizes",", it truly ",[630,18153,18154],{},"eats the stock version alive"," in both throughput and latency.",[18,18157,18158,18160,18161,18164,18165,18168],{},[14,18159,16712],{},": These benchmarks were performed in mid ",[14,18162,18163],{},"January",". While the LLM engine landscape evolves rapidly, ",[14,18166,18167],{},"Paiton’s model-level optimizations"," typically compound any subsequent engine improvements.",[10,18170,18172],{"id":18171},"_2-model-overview",[14,18173,18174],{},"2. Model Overview",[10,18176,18178],{"id":18177},"deepseek-r1-distill-llama-31-8b","Deepseek R1 Distill Llama 3.1 8B",[30,18180,18181,18187],{},[33,18182,18183,18184],{},"A “distilled” Llama 3.1 variant focused on ",[14,18185,18186],{},"lower latency and reduced GPU load.",[33,18188,18189,18190,18193],{},"Ideal for ",[14,18191,18192],{},"online serving"," scenarios demanding high speed and memory efficiency.",[10,18195,18197],{"id":18196},"why-vllm-for-online-serving","Why vLLM for Online Serving?",[30,18199,18200,18203,18206],{},[33,18201,18202],{},"Active development in scheduler and sampler modules.",[33,18204,18205],{},"Benchmarking that aligns with real-time usage patterns.",[33,18207,18208],{},"Adaptability to various concurrency and batch configurations.",[18,18210,18211,18212,18215,18216,18218],{},"(We will continue to explore ",[14,18213,18214],{},"SGlang",", but for now ",[14,18217,696],{}," remains our engine of choice for these online serving tests.)",[10,18220,18222],{"id":18221},"_3-benchmark-setup",[14,18223,18224],{},"3. Benchmark Setup",[30,18226,18227,18236,18246,18252,18257,18263,18269,18274],{},[33,18228,18229,18231,18232,18235],{},[14,18230,962],{},": AMD MI300X server (featuring multiple MI300X GPUs), located in our main data center (",[14,18233,18234],{},"Server A",").",[33,18237,18238,18241,18242,18245],{},[14,18239,18240],{},"Client",": Queries sent from ",[14,18243,18244],{},"Server B"," in a different location, mimicking real-world latency.",[33,18247,18248,18251],{},[14,18249,18250],{},"Engine",": vLLM 0.6.3",[33,18253,18254,18256],{},[14,18255,618],{},": 1",[33,18258,18259,18262],{},[14,18260,18261],{},"Batch Sizes",": Ranging from 1 to 4096.",[33,18264,18265,18268],{},[14,18266,18267],{},"Input\u002FOutput tokens",": default settings",[33,18270,18271,18273],{},[14,18272,11412],{},": We used the ShareGPT dataset to generate realistic user queries.",[33,18275,18276,237,18278],{},[14,18277,16765],{},[30,18279,18280,18285],{},[33,18281,18282,18284],{},[14,18283,533],{},": Requests\u002Fs, Output Tokens\u002Fs, Total Tokens\u002Fs.",[33,18286,18287,18289],{},[14,18288,9525],{},": TTFT (Time to First Token), TPOT (Time Per Output Token), ITL (Inter-Token Latency).",[18,18291,18292],{},"These measurements aim to replicate a production environment, where user queries might arrive sporadically rather than in tidy, pre-scheduled batches.",[10,18294,18296],{"id":18295},"_4-throughput",[14,18297,18298],{},"4. Throughput",[10,18300,18302],{"id":18301},"throughput-comparison-stock-model-vs-paiton","Throughput Comparison: Stock Model vs. Paiton",[18,18304,18305],{},"We collected throughput data across multiple dimensions:",[30,18307,18308,18314,18320,18326,18332],{},[33,18309,18310,18313],{},[14,18311,18312],{},"Succ. Req (Stock\u002FPaiton)",": The actual number of successful requests processed during the benchmarking period.",[33,18315,18316,18319],{},[14,18317,18318],{},"Duration (Stock\u002FPaiton)",": How long each test ran in seconds.",[33,18321,18322,18325],{},[14,18323,18324],{},"Req\u002Fs (Stock\u002FPaiton)",": Requests per second, a direct measure of concurrency handling.",[33,18327,18328,18331],{},[14,18329,18330],{},"Out Tok\u002Fs (Stock\u002FPaiton)",": How many output tokens were generated per second, indicating how quickly the model produces text once it starts.",[33,18333,18334,18337],{},[14,18335,18336],{},"Total Tok\u002Fs (Stock\u002FPaiton)",": The sum of input + output tokens processed per second, a holistic measure of token throughput.",[18,18339,18340],{},"Below is the table of results from batch size 1 to 4096:",[298,18342,18343,18402],{},[301,18344,18345],{},[304,18346,18347,18352,18357,18362,18367,18372,18377,18382,18387,18392,18397],{},[307,18348,18349],{},[14,18350,18351],{},"Batch size",[307,18353,18354],{},[14,18355,18356],{},"Succ. Req (Stock)",[307,18358,18359],{},[14,18360,18361],{},"Succ. Req (Paiton)",[307,18363,18364],{},[14,18365,18366],{},"Duration (s) (Stock)",[307,18368,18369],{},[14,18370,18371],{},"Duration (s) (Paiton)",[307,18373,18374],{},[14,18375,18376],{},"Req\u002Fs (Stock)",[307,18378,18379],{},[14,18380,18381],{},"Req\u002Fs (Paiton)",[307,18383,18384],{},[14,18385,18386],{},"Out Tok\u002Fs (Stock)",[307,18388,18389],{},[14,18390,18391],{},"Out Tok\u002Fs (Paiton)",[307,18393,18394],{},[14,18395,18396],{},"Total Tok\u002Fs (Stock)",[307,18398,18399],{},[14,18400,18401],{},"Total Tok\u002Fs (Paiton)",[316,18403,18404,18435,18465,18495,18526,18559,18592,18625,18658,18691,18726,18760,18795],{},[304,18405,18406,18408,18410,18412,18414,18417,18420,18423,18426,18429,18432],{},[321,18407,6802],{},[321,18409,6802],{},[321,18411,6802],{},[321,18413,17001],{},[321,18415,18416],{},"1.03",[321,18418,18419],{},"0.99",[321,18421,18422],{},"0.97",[321,18424,18425],{},"119.31",[321,18427,18428],{},"116.61",[321,18430,18431],{},"132.24",[321,18433,18434],{},"129.24",[304,18436,18437,18439,18441,18443,18446,18449,18451,18453,18456,18459,18462],{},[321,18438,6819],{},[321,18440,6819],{},[321,18442,6819],{},[321,18444,18445],{},"6.54",[321,18447,18448],{},"6.65",[321,18450,10652],{},[321,18452,13278],{},[321,18454,18455],{},"135.31",[321,18457,18458],{},"132.92",[321,18460,18461],{},"141.12",[321,18463,18464],{},"138.63",[304,18466,18467,18469,18471,18473,18475,18478,18481,18483,18486,18489,18492],{},[321,18468,6836],{},[321,18470,6836],{},[321,18472,6836],{},[321,18474,18445],{},[321,18476,18477],{},"6.69",[321,18479,18480],{},"0.61",[321,18482,10692],{},[321,18484,18485],{},"200.52",[321,18487,18488],{},"196.82",[321,18490,18491],{},"211.98",[321,18493,18494],{},"208.03",[304,18496,18497,18499,18501,18503,18505,18508,18511,18514,18517,18520,18523],{},[321,18498,6853],{},[321,18500,6853],{},[321,18502,6853],{},[321,18504,18448],{},[321,18506,18507],{},"6.82",[321,18509,18510],{},"1.20",[321,18512,18513],{},"1.17",[321,18515,18516],{},"328.76",[321,18518,18519],{},"308.84",[321,18521,18522],{},"503.14",[321,18524,18525],{},"478.87",[304,18527,18528,18530,18533,18535,18538,18541,18544,18547,18550,18553,18556],{},[321,18529,6870],{},[321,18531,18532],{},"15",[321,18534,18532],{},[321,18536,18537],{},"7.12",[321,18539,18540],{},"7.07",[321,18542,18543],{},"2.11",[321,18545,18546],{},"2.12",[321,18548,18549],{},"478.27",[321,18551,18552],{},"481.56",[321,18554,18555],{},"840.52",[321,18557,18558],{},"846.20",[304,18560,18561,18563,18566,18568,18571,18574,18577,18580,18583,18586,18589],{},[321,18562,349],{},[321,18564,18565],{},"31",[321,18567,18565],{},[321,18569,18570],{},"7.69",[321,18572,18573],{},"7.48",[321,18575,18576],{},"4.03",[321,18578,18579],{},"4.14",[321,18581,18582],{},"844.94",[321,18584,18585],{},"867.74",[321,18587,18588],{},"1635.26",[321,18590,18591],{},"1679.88",[304,18593,18594,18596,18599,18601,18604,18607,18610,18613,18616,18619,18622],{},[321,18595,6903],{},[321,18597,18598],{},"62",[321,18600,18598],{},[321,18602,18603],{},"8.23",[321,18605,18606],{},"8.46",[321,18608,18609],{},"7.54",[321,18611,18612],{},"7.33",[321,18614,18615],{},"1540.69",[321,18617,18618],{},"1497.98",[321,18620,18621],{},"3196.12",[321,18623,18624],{},"3108.53",[304,18626,18627,18629,18632,18634,18637,18640,18643,18646,18649,18652,18655],{},[321,18628,6920],{},[321,18630,18631],{},"125",[321,18633,18631],{},[321,18635,18636],{},"9.90",[321,18638,18639],{},"10.84",[321,18641,18642],{},"12.63",[321,18644,18645],{},"11.53",[321,18647,18648],{},"2773.89",[321,18650,18651],{},"2532.12",[321,18653,18654],{},"5505.84",[321,18656,18657],{},"5026.42",[304,18659,18660,18662,18665,18667,18670,18673,18676,18679,18682,18685,18688],{},[321,18661,6937],{},[321,18663,18664],{},"246",[321,18666,18664],{},[321,18668,18669],{},"13.43",[321,18671,18672],{},"15.27",[321,18674,18675],{},"18.32",[321,18677,18678],{},"16.11",[321,18680,18681],{},"4027.02",[321,18683,18684],{},"3539.33",[321,18686,18687],{},"7755.44",[321,18689,18690],{},"6816.71",[304,18692,18693,18696,18699,18702,18705,18708,18711,18714,18717,18720,18723],{},[321,18694,18695],{},"512",[321,18697,18698],{},"487",[321,18700,18701],{},"488",[321,18703,18704],{},"36.43",[321,18706,18707],{},"26.33",[321,18709,18710],{},"13.37",[321,18712,18713],{},"18.54",[321,18715,18716],{},"2981.57",[321,18718,18719],{},"4126.93",[321,18721,18722],{},"5507.35",[321,18724,18725],{},"7639.89",[304,18727,18728,18731,18734,18736,18739,18742,18745,18748,18751,18754,18757],{},[321,18729,18730],{},"1024",[321,18732,18733],{},"974",[321,18735,18733],{},[321,18737,18738],{},"62.91",[321,18740,18741],{},"47.37",[321,18743,18744],{},"15.48",[321,18746,18747],{},"20.56",[321,18749,18750],{},"3153.95",[321,18752,18753],{},"4188.55",[321,18755,18756],{},"6259.23",[321,18758,18759],{},"8316.16",[304,18761,18762,18765,18768,18771,18774,18777,18780,18783,18786,18789,18792],{},[321,18763,18764],{},"2048",[321,18766,18767],{},"1944",[321,18769,18770],{},"1942",[321,18772,18773],{},"151.01",[321,18775,18776],{},"99.34",[321,18778,18779],{},"12.87",[321,18781,18782],{},"19.55",[321,18784,18785],{},"2688.64",[321,18787,18788],{},"4079.61",[321,18790,18791],{},"5387.61",[321,18793,18794],{},"8178.29",[304,18796,18797,18800,18803,18806,18809,18812,18815,18818,18821,18824,18827],{},[321,18798,18799],{},"4096",[321,18801,18802],{},"3897",[321,18804,18805],{},"3889",[321,18807,18808],{},"244.52",[321,18810,18811],{},"190.22",[321,18813,18814],{},"15.94",[321,18816,18817],{},"20.45",[321,18819,18820],{},"3274.32",[321,18822,18823],{},"4197.43",[321,18825,18826],{},"6612.40",[321,18828,18829],{},"8471.24",[10,18831,18833],{"id":18832},"detailed-observations","Detailed Observations",[30,18835,18836,18861,18874,18887],{},[33,18837,18838,18841],{},[14,18839,18840],{},"Req\u002Fs (Requests\u002Fs)",[30,18842,18843,18849,18855],{},[33,18844,18845,18848],{},[14,18846,18847],{},"Low Batches (1–8)",": Paiton is slightly behind by about 1–4% in raw Req\u002Fs. This is because smaller batches currently don’t yet fully exploit Paiton’s fused kernels or concurrency benefits.",[33,18850,18851,18854],{},[14,18852,18853],{},"Mid Batches (16–128)",": Both converge, with Paiton occasionally pulling ahead in Req\u002Fs, indicating better concurrency handling.",[33,18856,18857,18860],{},[14,18858,18859],{},"High Batches (256–4096)",": Paiton scales significantly better, e.g. 512 (Stock ~13.37, Paiton ~18.54) and 4096 (Stock ~15.94, Paiton ~20.45).",[33,18862,18863,18866],{},[14,18864,18865],{},"Out Tok\u002Fs (Output Tokens\u002Fs)",[30,18867,18868,18871],{},[33,18869,18870],{},"Reflects how fast the system can produce response tokens once generation starts.",[33,18872,18873],{},"Notice how at 512, Stock is ~2981.57 vs. Paiton ~4126.93, a 38% improvement in raw text generation. For large text outputs (like summarization or conversation), this difference translates to noticeably faster completions for end users.",[33,18875,18876,18879],{},[14,18877,18878],{},"Total Tok\u002Fs (Input + Output)",[30,18880,18881,18884],{},[33,18882,18883],{},"Holistic measure of tokens processed.",[33,18885,18886],{},"Paiton consistently edges out or greatly surpasses Stock at larger batches. For instance, 4096 sees a jump from ~6612 to ~8471, a ~28% improvement overall.",[33,18888,18889,18892],{},[14,18890,18891],{},"Duration",[30,18893,18894],{},[33,18895,18896],{},"The test run lengths also show that for certain batch sizes, Paiton can complete a similar workload in less time (e.g., 512: Stock took ~36.43s vs. Paiton’s ~26.33s).",[18,18898,18899,18901],{},[14,18900,16712],{},": We also want to emphasize that the total duration length from start to finish (starting to serve the model, in other words: firing up vllm, loading the model and preparing it for serving and initiating a request from a secondary server) is significantly faster.",[10,18903,18905],{"id":18904},"real-world-examples","Real-World Examples",[30,18907,18908,18911],{},[33,18909,18910],{},"Chatbot with bursts of requests: Batches around 64–256 typically form when users spike. Paiton’s concurrency advantage at these sizes ensures higher Req\u002Fs and faster completions.",[33,18912,18913],{},"Bulk inference tasks (batch size 512+): Summarizing large documents, mass processing queries, Paiton handles these heavy loads with minimal slowdown, whereas the stock model’s throughput starts to plateau or drop off.",[18,18915,18916],{},[283,18917],{"alt":285,"src":18918},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Freal-world-examples.png",[18,18920,18921,526],{},[14,18922,18923],{},"Interpretation",[30,18925,18926,18935,18943],{},[33,18927,18928,18931,18932,18934],{},[14,18929,18930],{},"Subplot 1",": Direct concurrency measure (",[223,18933,12431],{},"). Notice how Paiton typically surpasses Stock as batch size grows, reflecting better overall scheduling\u002Ffused kernels.",[33,18936,18937,7794,18940,18942],{},[14,18938,18939],{},"Subplot 2",[14,18941,269],{}," highlights how quickly large volumes of text are generated.",[33,18944,18945,7794,18948,18951],{},[14,18946,18947],{},"Subplot 3",[14,18949,18950],{},"Total Tokens\u002Fs"," merges input and output token processing, useful for analyzing how the pipeline handles end-to-end token throughput.",[10,18953,18955],{"id":18954},"_5-latency",[14,18956,18957],{},"5. Latency",[18,18959,18960],{},"In addition to measuring raw throughput, latency is crucial for any real-time or interactive application. We specifically focused on:",[30,18962,18963,18969,18975],{},[33,18964,18965,18968],{},[14,18966,18967],{},"TTFT (Time-to-First-Token)",": How quickly the very first token is produced after an inference request.",[33,18970,18971,18974],{},[14,18972,18973],{},"TPOT (Time-Per-Output-Token)",": How long it takes to generate each subsequent token.",[33,18976,18977,18980],{},[14,18978,18979],{},"ITL (Inter-Token Latency)",": Another measure of the gap between tokens, highly relevant when you have streaming outputs.",[10,18982,18984],{"id":18983},"stock-model-vs-paiton-ttft-tpot-itl","Stock Model vs. Paiton: TTFT, TPOT, ITL",[18,18986,18987,18988,526],{},"Below is the latency data we collected at batch sizes ranging from ",[14,18989,18990],{},"1 to 4096",[298,18992,18993,19031],{},[301,18994,18995],{},[304,18996,18997,19001,19006,19011,19016,19021,19026],{},[307,18998,18999],{},[14,19000,18351],{},[307,19002,19003],{},[14,19004,19005],{},"TTFT (Stock)",[307,19007,19008],{},[14,19009,19010],{},"TTFT (Paiton)",[307,19012,19013],{},[14,19014,19015],{},"TPOT (Stock)",[307,19017,19018],{},[14,19019,19020],{},"TPOT (Paiton)",[307,19022,19023],{},[14,19024,19025],{},"ITL (Stock)",[307,19027,19028],{},[14,19029,19030],{},"ITL (Paiton)",[316,19032,19033,19052,19074,19096,19117,19139,19161,19183,19205,19227,19249,19271,19293],{},[304,19034,19035,19037,19040,19043,19046,19048,19050],{},[321,19036,6802],{},[321,19038,19039],{},"24.34",[321,19041,19042],{},"20.89",[321,19044,19045],{},"8.24",[321,19047,18606],{},[321,19049,19045],{},[321,19051,18606],{},[304,19053,19054,19056,19059,19062,19065,19068,19071],{},[321,19055,6819],{},[321,19057,19058],{},"147.11",[321,19060,19061],{},"27.90",[321,19063,19064],{},"8.34",[321,19066,19067],{},"8.64",[321,19069,19070],{},"8.31",[321,19072,19073],{},"8.61",[304,19075,19076,19078,19081,19084,19087,19090,19093],{},[321,19077,6836],{},[321,19079,19080],{},"151.61",[321,19082,19083],{},"31.43",[321,19085,19086],{},"8.36",[321,19088,19089],{},"8.72",[321,19091,19092],{},"8.33",[321,19094,19095],{},"8.68",[304,19097,19098,19100,19103,19106,19109,19112,19114],{},[321,19099,6853],{},[321,19101,19102],{},"183.36",[321,19104,19105],{},"80.19",[321,19107,19108],{},"8.59",[321,19110,19111],{},"8.95",[321,19113,18606],{},[321,19115,19116],{},"8.82",[304,19118,19119,19121,19124,19127,19130,19133,19136],{},[321,19120,6870],{},[321,19122,19123],{},"476.79",[321,19125,19126],{},"151.58",[321,19128,19129],{},"9.25",[321,19131,19132],{},"9.68",[321,19134,19135],{},"8.92",[321,19137,19138],{},"9.18",[304,19140,19141,19143,19146,19149,19152,19155,19158],{},[321,19142,349],{},[321,19144,19145],{},"346.01",[321,19147,19148],{},"271.58",[321,19150,19151],{},"13.58",[321,19153,19154],{},"10.29",[321,19156,19157],{},"11.18",[321,19159,19160],{},"9.85",[304,19162,19163,19165,19168,19171,19174,19177,19180],{},[321,19164,6903],{},[321,19166,19167],{},"743.26",[321,19169,19170],{},"568.33",[321,19172,19173],{},"11.59",[321,19175,19176],{},"13.06",[321,19178,19179],{},"10.81",[321,19181,19182],{},"11.51",[304,19184,19185,19187,19190,19193,19196,19199,19202],{},[321,19186,6920],{},[321,19188,19189],{},"951.62",[321,19191,19192],{},"924.81",[321,19194,19195],{},"23.91",[321,19197,19198],{},"27.37",[321,19200,19201],{},"14.35",[321,19203,19204],{},"16.67",[304,19206,19207,19209,19212,19215,19218,19221,19224],{},[321,19208,6937],{},[321,19210,19211],{},"1776.32",[321,19213,19214],{},"1525.37",[321,19216,19217],{},"38.28",[321,19219,19220],{},"51.80",[321,19222,19223],{},"20.53",[321,19225,19226],{},"25.18",[304,19228,19229,19231,19234,19237,19240,19243,19246],{},[321,19230,18695],{},[321,19232,19233],{},"6788.80",[321,19235,19236],{},"4440.15",[321,19238,19239],{},"87.42",[321,19241,19242],{},"57.89",[321,19244,19245],{},"64.75",[321,19247,19248],{},"40.16",[304,19250,19251,19253,19256,19259,19262,19265,19268],{},[321,19252,18730],{},[321,19254,19255],{},"19560.22",[321,19257,19258],{},"14734.59",[321,19260,19261],{},"77.08",[321,19263,19264],{},"57.37",[321,19266,19267],{},"69.45",[321,19269,19270],{},"48.82",[304,19272,19273,19275,19278,19281,19284,19287,19290],{},[321,19274,18764],{},[321,19276,19277],{},"65322.87",[321,19279,19280],{},"37294.28",[321,19282,19283],{},"95.91",[321,19285,19286],{},"58.61",[321,19288,19289],{},"86.34",[321,19291,19292],{},"53.68",[304,19294,19295,19297,19300,19303,19306,19309,19312],{},[321,19296,18799],{},[321,19298,19299],{},"99924.78",[321,19301,19302],{},"84892.12",[321,19304,19305],{},"76.50",[321,19307,19308],{},"59.48",[321,19310,19311],{},"73.74",[321,19313,19314],{},"56.55",[10,19316,19318],{"id":19317},"time-to-first-token-ttft","Time-to-First-Token (TTFT)",[30,19320,19321,19324],{},[33,19322,19323],{},"First Impression Matters: For interactive applications like chatbots, users judge responsiveness the moment they see that first token.",[33,19325,19326],{},"Substantial Gains: Across nearly all batch sizes, Paiton is faster, sometimes by tens of thousands of ms at higher batches.",[10,19328,19330],{"id":19329},"tpot-itl","TPOT & ITL",[30,19332,19333,19339],{},[33,19334,19335,19338],{},[14,19336,19337],{},"Token Generation Speed",": Once the first token is out, we still want the rest of the output quickly. TPOT measures how many ms per token, effectively shaping the overall throughput for a long response.",[33,19340,19341,19344,19345],{},[14,19342,19343],{},"Stability",": For higher batch sizes (256+), you’ll notice Paiton’s TPOT remains relatively consistent, while the stock model can spike more drastically.\n",[30,19346,19347],{},[33,19348,19349],{},"E.g., batch size 2048: TPOT ~95.91 ms (Stock) vs. ~58.61 ms (Paiton). Over 50 tokens, that’s an extra 1.9 seconds just in token generation time.",[10,19351,19353],{"id":19352},"real-world-impact","Real-World Impact",[30,19355,19356,19369],{},[33,19357,19358,237,19361],{},[14,19359,19360],{},"Small Batch Usage",[30,19362,19363,19366],{},[33,19364,19365],{},"The difference in TTFT might only be tens or hundreds of ms, but that still matters in a low-latency setting (like user-type-ahead).",[33,19367,19368],{},"Even a half-second faster first token can “feel” significantly better to end users.",[33,19370,19371,237,19374],{},[14,19372,19373],{},"High Batch Usage",[30,19375,19376,19379],{},[33,19377,19378],{},"If your system queues up many requests or handles surges of traffic, Paiton’s stable TPOT\u002FTTFT ensures you don’t see an exponential rise in overall response time.",[33,19380,19381],{},"This results in lower E2E Latency (Section 6) and higher throughput concurrency.",[18,19383,19384],{},[283,19385],{"alt":285,"src":19386},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Freal-world-impact.jpg",[18,19388,19389,526],{},[14,19390,19391],{},"Interpreting These Plots",[30,19393,19394,19404],{},[33,19395,19396,237,19399],{},[14,19397,19398],{},"TTFT Plot",[30,19400,19401],{},[33,19402,19403],{},"Watch how quickly the stock line grows for higher batch sizes, while Paiton’s remains comparatively lower. This is crucial for first-token responsiveness.",[33,19405,19406,237,19409],{},[14,19407,19408],{},"TPOT Plot",[30,19410,19411],{},[33,19412,19413],{},"Notice that Paiton’s line might be slightly higher at small batch sizes or slightly lower at large batch sizes, it depends on concurrency overhead vs. kernel efficiency. But overall, it trends more stable than the stock model.",[10,19415,19417],{"id":19416},"_6-going-deeper-latency-vs-throughput",[14,19418,19419],{},"6. Going Deeper: Latency vs. Throughput",[10,19421,19423],{"id":19422},"why-latency-matters-alongside-throughput","Why Latency Matters Alongside Throughput",[18,19425,19426],{},"A single “throughput” metric (e.g., requests per second) doesn’t provide the full picture. In many real-world use cases, particularly chatbots, question-answering systems, and interactive applications, user experience is heavily influenced by latency: how quickly the model begins responding (TTFT) and how rapidly subsequent tokens are generated (TPOT or ITL).",[10,19428,19430],{"id":19429},"e2e-latency-a-closer-look","E2E Latency: A Closer Look",[18,19432,19433,19434,19437],{},"We approximate ",[14,19435,19436],{},"End-to-End (E2E) Latency"," with the formula:",[18,19439,19440],{},"E2E Latency ≈ TTFT + (num_tokens×TPOT)",[18,19442,19443],{},"where:",[30,19445,19446,19452],{},[33,19447,19448,19451],{},[14,19449,19450],{},"TTFT"," (Time-to-First-Token): The duration before the very first token is returned.",[33,19453,19454,19457],{},[14,19455,19456],{},"TPOT"," (Time-Per-Output-Token): How long each subsequent token takes after the first one.",[18,19459,19460],{},"This means any improvement in TTFT, TPOT, or both can significantly lower E2E latency.",[18,19462,19463,19464,46],{},"Ideally, you want both high throughput and low latency. In real-time contexts (chatbots, Q&A systems), tokens\u002Fsec\u002Fuser also matters. Below is a simplified chart (conceptual) showing how an optimal approach moves the latency-throughput curve ",[630,19465,561],{},[8451,19467,19470],{"className":19468,"code":19469,"language":8456},[8454],"(Throughput) ↑\n\n             |  ● Paiton model\n\n             |               ● Stock model\n\n             |`\n\n             +--------------------------------→ (E2E Latency)\n",[223,19471,19469],{"__ignoreMap":285},[10,19473,19475],{"id":19474},"tying-the-data-to-real-scenarios","Tying the Data to Real Scenarios",[18,19477,19478],{},"By looking at both Batch Size and TTFT in the tables above:",[30,19480,19481,19499],{},[33,19482,19483,237,19486],{},[14,19484,19485],{},"At Batch Size 512",[30,19487,19488,19496],{},[33,19489,19490,19492,19493,46],{},[14,19491,8646],{}," TTFT is ∼6788 ms, while Paiton TTFT is ∼4440 ms, a ",[14,19494,19495],{},"2.3 second difference",[33,19497,19498],{},"Once you factor in generating multiple tokens (say 50–100 tokens for an average chatbot reply), that difference compounds further in E2E latency.",[33,19500,19501,237,19504],{},[14,19502,19503],{},"At Batch Size 2048",[30,19505,19506],{},[33,19507,19508,19510,19511,19514],{},[14,19509,8646],{}," TTFT grows to ∼65322 ms, whereas Paiton remains at ∼37294 ms. That’s a roughly ",[14,19512,19513],{},"28 second"," reduction in waiting time before the first token. For users requesting large outputs in a high-concurrency environment, this difference can be the deciding factor in whether your application feels “snappy” or “unusable.”",[18,19516,19517,19518,19521],{},"Meanwhile, Throughput at these same batch sizes remains significantly higher with Paiton, indicating you can serve more concurrent queries ",[630,19519,19520],{},"and"," ensure faster time-to-first-token.",[10,19523,19525],{"id":19524},"interpreting-the-curves","Interpreting the Curves",[18,19527,19528],{},"If we plot throughput (Requests\u002Fs) against an approximate E2E latency (TTFT + tokens × TPOT):",[114,19530,19531,19534],{},[33,19532,19533],{},"Paiton’s Curve generally appears above and to the left of the Stock curve, meaning at any given throughput level, its E2E latency is lower, or conversely, at a certain latency requirement, it can handle more requests per second.",[33,19535,19536,19539],{},[14,19537,19538],{},"Scaling Behavior",": As batch size increases (256, 512, 1024, 2048, 4096), the Stock model’s TTFT spikes dramatically, leading to large E2E latencies. Paiton’s TTFT also increases, but at a lower rate, maintaining better responsiveness under load.",[18,19541,19542],{},[283,19543],{"alt":285,"src":19544},"\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fe2e-latency-vs-throughput.png",[10,19546,19548],{"id":19547},"key-takeaways-for-production","Key Takeaways for Production",[30,19550,19551,19557,19563],{},[33,19552,19553,19556],{},[14,19554,19555],{},"Higher concurrency",": The model can handle a large queue of requests without ballooning E2E latency to intolerable levels.",[33,19558,19559,19562],{},[14,19560,19561],{},"Better user experience",": Especially for first-token speed, which psychologically sets the tone for the rest of the session. Even if the final token arrives only a bit sooner or later, that fast initial response keeps end users engaged.",[33,19564,19565,19568],{},[14,19566,19567],{},"Scalability",": If your application occasionally spikes in traffic (hundreds or thousands of simultaneous requests), Paiton’s approach ensures you don’t see an exponential rise in response time.",[10,19570,19572],{"id":19571},"example-if-you-generate-100-tokens-per-request","Example: If You Generate 100 Tokens per Request",[18,19574,19575],{},"Using the formula TTFT + 100 × TPOT:",[30,19577,19578],{},[33,19579,19580,237,19582],{},[14,19581,19485],{},[30,19583,19584,19590],{},[33,19585,19586,19587],{},"Stock E2E ≈ 6788 + 100 × 87.42 = 6788 + 8742 = 15,530 ms ≈ ",[14,19588,19589],{},"15.5 s",[33,19591,19592,19593],{},"Paiton E2E ≈ 4440 + 100 × 57.89 = 4440 + 5789 = 10,229 ms ≈ ",[14,19594,19595],{},"10.2 s",[18,19597,19598,19599,19602],{},"In this scenario, Paiton is delivering ",[14,19600,19601],{},"over 5 seconds"," faster end-to-end latency for the same size response, even ignoring the throughput advantage. Multiply this across thousands of concurrent requests, and the difference in user satisfaction and infrastructure load becomes substantial.",[10,19604,19606],{"id":19605},"_7-key-observations",[14,19607,19608],{},"7. Key Observations",[10,19610,19612],{"id":19611},"overall-superiority","Overall Superiority",[30,19614,19615,19625],{},[33,19616,19617,237,19620],{},[14,19618,19619],{},"Small Batches",[30,19621,19622],{},[33,19623,19624],{},"Throughput is often on par (or slightly lower) compared to Stock, but TTFT (first-token latency) is consistently better. This is important for user-facing scenarios with low concurrency.",[33,19626,19627,237,19630],{},[14,19628,19629],{},"Medium to Large Batches",[30,19631,19632],{},[33,19633,19634],{},"Throughput and latency gains become significant; Paiton excels in harnessing concurrency and fused kernels for AMD GPUs.",[10,19636,19638],{"id":19637},"high-batch-efficacy","High Batch Efficacy",[30,19640,19641,19651,19661],{},[33,19642,19643,237,19646],{},[14,19644,19645],{},"512 Example",[30,19647,19648],{},[33,19649,19650],{},"Stock ≈ 13.37 req\u002Fs vs. Paiton ≈ 18.54, a ∼38% jump in raw throughput.",[33,19652,19653,237,19656],{},[14,19654,19655],{},"4096 Example",[30,19657,19658],{},[33,19659,19660],{},"Stock ≈ 15.94 req\u002Fs vs. Paiton ≈ 20.45, a ∼28% improvement.",[33,19662,19663,19666],{},[14,19664,19665],{},"Latency Gains",": TTFT differences at high batches sometimes range in the thousands of ms, dramatically enhancing user experience for large-scale tasks.",[10,19668,19353],{"id":19669},"real-world-impact-1",[114,19671,19672,19682],{},[33,19673,19674,19677],{},[14,19675,19676],{},"Small Batch Usage (Sporadic Queries)",[30,19678,19679],{},[33,19680,19681],{},"Faster Time-to-First-Token ensures end users see immediate responses, vital for chatbots or real-time prompts.",[33,19683,19684,19687],{},[14,19685,19686],{},"High Batch Usage (Peak Loads, Large Queues)",[30,19688,19689,19692],{},[33,19690,19691],{},"Paiton’s more efficient scaling can handle up to 50% more requests in certain cases.",[33,19693,19694],{},"Ideal for bulk inferencing tasks like summarization or large-scale embeddings.",[10,19696,19698],{"id":19697},"cost-energy-implications","Cost & Energy Implications",[30,19700,19701,19707],{},[33,19702,19703,19706],{},[14,19704,19705],{},"Better Throughput"," often translates to lower cloud costs when renting GPU time, as you can finish workloads faster or serve more users on the same hardware.",[33,19708,19709,19712],{},[14,19710,19711],{},"Lower Latency"," and fewer idle cycles often mean energy savings, especially on large HPC clusters or data centers running AMD MI300X GPUs.",[10,19714,19716],{"id":19715},"multi-gpu-hpc-scaling","Multi-GPU & HPC Scaling",[30,19718,19719,19722],{},[33,19720,19721],{},"While these results focus on a single system, the concurrency and kernel fusion benefits of Paiton tend to compound in multi-GPU or HPC setups.",[33,19723,19724],{},"Future tests will highlight how Paiton’s architecture-centric compilation scales across multiple AMD GPUs with massive VRAM, ensuring consistent performance gains regardless of cluster size.",[71,19726],{},[10,19728,19730],{"id":19729},"_8-conclusion-next-steps",[14,19731,19732],{},"8. Conclusion & Next Steps",[18,19734,19735],{},"Despite giving the Stock Deepseek R1 Distill Llama 3.1 8B model every advantage in vLLM, Paiton maintains a clear lead in both throughput and latency::",[30,19737,19738,19744,19750],{},[33,19739,19740,19743],{},[14,19741,19742],{},"Better TTFT"," for immediate responsiveness.",[33,19745,19746,19749],{},[14,19747,19748],{},"Equal or higher"," throughput at medium batch sizes.",[33,19751,19752,19755],{},[14,19753,19754],{},"Significant"," throughput gains at large batch sizes, demonstrating superior scaling on AMD MI300X hardware.",[10,19757,1119],{"id":1116},[30,19759,19760,19766,19772,19778,19784],{},[33,19761,19762,19765],{},[14,19763,19764],{},"SGlang Evaluations",": We continue to explore how SGlang’s concurrency model pairs with Paiton’s AMD optimizations.",[33,19767,19768,19771],{},[14,19769,19770],{},"Tokens\u002Fsec\u002FUser",": Future posts will highlight multi-user metrics, essential for large-scale chatbot usage.",[33,19773,19774,19777],{},[14,19775,19776],{},"FP8 & Other Quantization",": Balancing accuracy and speed is a priority, and we plan to share how ultra-low-precision inference performs on AMD.",[33,19779,19780,19783],{},[14,19781,19782],{},"Lower batch size optimization:"," As shown in this blog, Paiton clearly excels when it comes to higher batch sizes, ideal for production environments, however, we will continue our work to optimize for smaller batch sizes as well.",[33,19785,19786,19788],{},[14,19787,618],{},": Multi-gpu inference setups, as well as “Paitonizing” :) larger models such as the Deepseek R1 model with 671 billion parameters.",[18,19790,19791,19794],{},[14,19792,19793],{},"Remember",": The LLM space evolves swiftly. Tomorrow’s libraries may surpass today’s, yet Paiton’s architectural approach ensures you’ll always reap compounding speedups from any future engine improvements.",[71,19796],{},[10,19798,19800],{"id":19799},"appendix","Appendix",[30,19802,19803,19813],{},[33,19804,19805,19808,19809,19812],{},[14,19806,19807],{},"Previous Blog",": “",[163,19810,168],{"href":18110,"rel":19811},[167],"” for a deeper dive into fused kernels and AMD-focused compilation.",[33,19814,19815,19818,19819,19822],{},[14,19816,19817],{},"Stay tuned:"," We’ll share more results as we continue optimizing ",[14,19820,19821],{},"every"," aspect of the pipeline, ensuring Paiton delivers even stronger performance.",[18,19824,19825,19826,19831],{},"Thank you for reading! For any questions, collaboration ideas, or specific LLM engine requests, ",[163,19827,19830],{"href":19828,"rel":19829},"https:\u002F\u002Fdocs.google.com\u002Fforms\u002Fd\u002F1TTGg_WpkR8KcNPvVhJZohS60X7LGtIK_q1IUlIuFBM0\u002Fedit",[167],"please contact us",". Stay tuned for further detailed comparisons and environment setups to help you unlock top-tier performance on AMD GPUs.",[18,19833,19834],{},[14,19835,671],{},{"title":285,"searchDepth":673,"depth":673,"links":19837},[19838],{"id":18099,"depth":673,"text":18100,"children":19839},[19840,19841,19842,19843,19844,19845,19846,19847,19848,19849,19850,19851,19852,19853,19854,19855,19856,19857,19858,19859,19860,19861,19862,19863,19864,19865,19866,19867,19868,19869],{"id":12255,"depth":676,"text":12258},{"id":18171,"depth":676,"text":18174},{"id":18177,"depth":676,"text":18178},{"id":18196,"depth":676,"text":18197},{"id":18221,"depth":676,"text":18224},{"id":18295,"depth":676,"text":18298},{"id":18301,"depth":676,"text":18302},{"id":18832,"depth":676,"text":18833},{"id":18904,"depth":676,"text":18905},{"id":18954,"depth":676,"text":18957},{"id":18983,"depth":676,"text":18984},{"id":19317,"depth":676,"text":19318},{"id":19329,"depth":676,"text":19330},{"id":19352,"depth":676,"text":19353},{"id":19416,"depth":676,"text":19419},{"id":19422,"depth":676,"text":19423},{"id":19429,"depth":676,"text":19430},{"id":19474,"depth":676,"text":19475},{"id":19524,"depth":676,"text":19525},{"id":19547,"depth":676,"text":19548},{"id":19571,"depth":676,"text":19572},{"id":19605,"depth":676,"text":19608},{"id":19611,"depth":676,"text":19612},{"id":19637,"depth":676,"text":19638},{"id":19669,"depth":676,"text":19353},{"id":19697,"depth":676,"text":19698},{"id":19715,"depth":676,"text":19716},{"id":19729,"depth":676,"text":19732},{"id":1116,"depth":676,"text":1119},{"id":19799,"depth":676,"text":19800},[688,689,68,690,691,692,693,694,695,68,696],"2025-01-31T09:11:02","Outperforming Stock Models on the AMD MI300X 1. Introduction We couldn’t wait to show what Paiton can really do. After detailing our AMD-centric approach and architecture-level optimizations in our previous blog post, we decided to test-drive Paiton on a hype-worthy model: Deepseek R1 Distill Llama 3.1 8B. By compiling the model into efficient libraries and fusing ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fpaitonpost2.webp",{},"\u002Fblog\u002Fa-first-look-at-paiton-in-action-deepseek-r1-distill-llama-3-1-8b",{"title":18094,"description":19872},"a-first-look-at-paiton-in-action-deepseek-r1-distill-llama-3-1-8b","blog\u002Fa-first-look-at-paiton-in-action-deepseek-r1-distill-llama-3-1-8b","DghiOK0S6GucJOkFSU4kNPuj5LDgGJYjf-GlVfp91Lk",{"id":19881,"title":168,"body":19882,"categories":19992,"date":19993,"description":19994,"extension":699,"image":19995,"meta":19996,"navigation":702,"originalUrl":18110,"path":19997,"seo":19998,"slug":19999,"stem":20000,"__hash__":20001},"blog\u002Fblog\u002Fai-model-optimization-with-paiton.md",{"type":7,"value":19883,"toc":19982},[19884,19887,19893,19896,19902,19905,19911,19914,19920,19923,19929,19932,19938,19941,19952,19958,19961,19967,19975,19978],[18,19885,19886],{},"In the fast-paced world of artificial intelligence, model efficiency and performance are paramount. At ElioVP, we’re redefining what’s possible by delivering unparalleled optimization solutions for AI models with Paiton. By compiling the model’s architecture and leveraging our custom-written kernels, Paiton enables faster inference and reduced resource consumption on AMD GPUs.",[10,19888,19890],{"id":19889},"why-model-optimization-is-more-important-than-ever",[14,19891,19892],{},"Why Model Optimization is More Important Than Ever",[18,19894,19895],{},"As AI models become more sophisticated, their computational demands grow exponentially. Many organizations are hitting performance bottlenecks due to large model sizes and limited hardware efficiency. Conventional frameworks often leave untapped potential on the table, especially when dealing with high-end GPUs equipped with vast amounts of VRAM. That’s where Paiton steps in, bridging the gap between theoretical GPU power and real-world AI performance.",[10,19897,19899],{"id":19898},"our-solution-the-paiton-framework",[14,19900,19901],{},"Our Solution: The Paiton Framework",[18,19903,19904],{},"Paiton’s unique approach addresses these challenges head-on. Here’s how:",[10,19906,19908],{"id":19907},"model-compilation",[14,19909,19910],{},"Model Compilation",[18,19912,19913],{},"We compile the AI model’s architecture into highly optimized .so libraries. This process eliminates inefficiencies inherent in general-purpose frameworks, tailoring the execution pipeline to specific hardware.",[10,19915,19917],{"id":19916},"custom-kernels",[14,19918,19919],{},"Custom Kernels",[18,19921,19922],{},"Our team has developed specialized kernels that go beyond standard libraries by tailoring them to specific AMD GPUs (Yes, we even optimize for the MI200 series, not just the MI300). These kernels are designed to maximize throughput while minimizing latency, ensuring every operation, from matrix multiplications to attention mechanisms, performs at peak efficiency.",[10,19924,19926],{"id":19925},"fused-kernels",[14,19927,19928],{},"Fused Kernels",[18,19930,19931],{},"By combining multiple operations into a single, highly efficient kernel, we reduce memory overhead and improve execution speed. These fused kernels are particularly effective for complex AI operations like multi-head attention and tensor reshaping, ensuring seamless and fast performance on AMD GPUs.",[10,19933,19935],{"id":19934},"real-world-impact-faster-inference-smarter-ai",[14,19936,19937],{},"Real-World Impact: Faster Inference, Smarter AI",[18,19939,19940],{},"By integrating Paiton into your workflow, you can achieve:",[30,19942,19943,19946,19949],{},[33,19944,19945],{},"Up to 80% Faster Online Inference Speeds: Experience a significant reduction in latency and increased throughput.",[33,19947,19948],{},"Reduced Energy Costs: Optimized models consume less power, making them more eco-friendly and cost-effective.",[33,19950,19951],{},"Improved Model Deployment: Simplify the process of deploying AI solutions on diverse platforms, from cloud environments to edge devices.",[10,19953,19955],{"id":19954},"why-choose-paiton",[14,19956,19957],{},"Why Choose Paiton?",[18,19959,19960],{},"Paiton is more than just a tool; it’s a paradigm shift in how AI models are optimized and deployed. Our expertise in compiling architectures and crafting custom kernels for AMD GPUs sets us apart, delivering results that generic frameworks simply can’t match. With Paiton, you’re embracing the future of AI model optimization.",[10,19962,19964],{"id":19963},"get-started-today",[14,19965,19966],{},"Get Started Today",[18,19968,19969,19970,19974],{},"Ready to take your AI models to the next level? ",[163,19971,19973],{"href":10000,"rel":19972},[167],"Contact us to learn more"," about Paiton and how we can help you unlock the full potential of your AI applications.",[18,19976,19977],{},"Let’s build smarter, faster, and more efficient AI together.",[18,19979,19980],{},[14,19981,671],{},{"title":285,"searchDepth":673,"depth":673,"links":19983},[19984,19985,19986,19987,19988,19989,19990,19991],{"id":19889,"depth":676,"text":19892},{"id":19898,"depth":676,"text":19901},{"id":19907,"depth":676,"text":19910},{"id":19916,"depth":676,"text":19919},{"id":19925,"depth":676,"text":19928},{"id":19934,"depth":676,"text":19937},{"id":19954,"depth":676,"text":19957},{"id":19963,"depth":676,"text":19966},[688,689,68,690,692,693,694,695,68,696],"2025-01-30T19:53:25","In the fast-paced world of artificial intelligence, model efficiency and performance are paramount. At ElioVP, we’re redefining what’s possible by delivering unparalleled optimization solutions for AI models with Paiton. By compiling the model’s architecture and leveraging our custom-written kernels, Paiton enables faster inference and reduced resource consumption on AMD GPUs. Why Model Optimization is More ...","\u002Fasset\u002Fimages\u002Fblog\u002Fimported\u002Fpaitonpost1.webp",{},"\u002Fblog\u002Fai-model-optimization-with-paiton",{"title":168,"description":19994},"ai-model-optimization-with-paiton","blog\u002Fai-model-optimization-with-paiton","uKD8O9io_xl6qHMZDK8C_VTPusNR1MjABjo7ZXPSOgs",1783692227812]