# Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA AI Engineer · 2026-04-10 NVIDIA’s Mozhgan Kabiri Chimeh demonstrates that running LLMs locally on the DGX Spark workstation, powered by the GB10 Grace Blackwell superchip and 128GB unified memory, achieves practical performance for models up to 14B parameters. Using a reproducible vLLM benchmarking methodology, she shows that the 14B NVFP4 quantized model delivers 20.19 tokens per second and a time-to-first-token 3.4× faster than the unoptimized 14B base model. The DGX Spark supports the same NVIDIA AI software stack as production environments, enabling local development, fine-tuning, and privacy-sensitive workloads before scaling to the cloud. NVFP4 quantization is highlighted as critical for balancing intelligence and throughput on single-system setups. ## Questions this episode answers ### How many tokens per second can I get running a 14B model on an NVIDIA DGX Spark with NVFP4 quantization? Mozhgan Kabiri Chimeh reports that the DGX Spark achieves 20.19 completion tokens per second with a 14B model using NVFP4 4-bit floating-point quantization. This is nearly 10 times larger than a 1.5B model, yet still delivers throughput faster than average human reading speed, making it practical for local development. The 14B base model without quantization drops to just 8.4 tokens per second, highlighting NVFP4's critical role. [6:17](https://aie.addtry.com/f0b31b29-4002-4db2-9b95-b3e439bcdea4?t=377000) ### How much faster is time-to-first-token on DGX Spark when using NVFP4 for a 14B model compared to the unoptimized version? Mozhgan shows that the 14B NVFP4 model is 3.4 times faster to first token than the unoptimized 14B base model on the DGX Spark. While larger models inherently have higher time-to-first-token due to increased computation, NVFP4 quantization significantly reduces this latency, making the 14B model feel as responsive as a much smaller one. This improvement is crucial for real-time interactive applications. [8:23](https://aie.addtry.com/f0b31b29-4002-4db2-9b95-b3e439bcdea4?t=503000) ### When is it better to run LLMs locally on DGX Spark instead of in the cloud? Mozhgan identifies three use cases: steady-state workloads where cost predictability matters, privacy-sensitive data requiring local processing, and rapid prototyping where iteration speed is critical. The DGX Spark runs the same NVIDIA AI software stack as DGX Cloud, enabling a 'run locally, iterate quickly, and when ready, scale to data center or cloud' workflow. This avoids shared infrastructure delays and data residency concerns. [9:24](https://aie.addtry.com/f0b31b29-4002-4db2-9b95-b3e439bcdea4?t=564000) ## Key moments - **[0:00] Intro & Motivation** - [0:30] Developer systems face two main challenges: running out of memory or lacking the right software stack for AI workloads. - **[1:24] DGX Spark** - [1:24] DGX Spark with 128GB unified memory and NFP4 support enables running models up to 200 billion parameters locally. - **[2:41] Reproducible Benchmarking** - **[4:14] Latency Measurement** - **[5:20] Throughput Results** - [5:43] 1.5B instruct model delivers 61.73 tokens/s while 14B NVFP4 achieves 20.19 tokens/s, faster than human reading speed. - [6:50] "This proves that on Blackwell hardware, the choice of quantization format is just as important as the hardware itself." - **[7:12] First Token Latency** - [8:23] 14B NVFP4 model is 3.4 times faster to first token than unoptimized 14B, significantly improving user-perceived responsiveness. - **[9:04] Key Takeaways** - [9:04] NVFP4 quantization effectively increases intelligence per byte, allowing a 14B model to feel as responsive as a much smaller one. - [9:24] DGX Spark is ideal for steady state workloads, privacy-sensitive data, and rapid prototyping with the same software stack as DGX Cloud. ## Speakers - **Mozhgan Kabiri Chimeh** (guest) ## Topics Inference Optimization, Quantization ## Mentioned NVIDIA (company), DGX Cloud (product), DGX Spark (product), Docker (product), GB10 (product), NVFP4 (product), Quen (product), VLLM (product) ## Transcript ### Intro & Motivation **Mozhgan Kabiri Chimeh** [0:02] Hello everyone. I'm Mozhgan Kabiri chimeh, Developer Relations Manager at NVIDIA, where I work closely with developers building and deploying AI systems. Today we're looking at running LLMs locally, practical LLM performance, and DGX Spark. This isn't a theoretical talk; it's a data-backed journey through the trade-offs of modern AI infrastructure. Findings are based on hands-on experiments, with the goal of understanding what's actually practical on a single system. The evolution in AI puts greater demand on developer systems, creating two main challenges: you either run out of memory or you do not have access to theright software stack. Then you end up pushing everything to the cloud or data center. As models move from experiments to production, concerns like cost predictability, data residency, and deterministic latency take center stage. And iteration speed often depends on access to shared infrastructure, and since your work will be scheduled against other competing workloads, it causes delays in development work as well. ### DGX Spark **Mozhgan Kabiri Chimeh** [1:24] So the question becomes: can we bring some of that workflow closer to where development actually happens? To maximize developer productivity, local solutions to these challenges are required. DGX Spark is designed from the ground up to build and run AI, and can be used as a standalone system. It's powered by the GB10 Grace Blackwell superchip, combining CPU and GPU with a unified memory architecture. With 128 GB of unified memory and NFP4 support, it enables developers to work with models of up to around 200 billion parameters locally, on a system that fits under a desk, or on top of your desk. It runs the same NVIDIA AI software stack used in production environments, meaning workflows can move from desktop to data center or cloud with minimal changes. The key idea here is not replacing the cloud, but bringing powerful AI development closer to the developer. This is my setup: everything runs locally and is reproducible. To serve the models, I used vLLM with a set of quen models across different sizes and precision formats. I ran this inside an NVIDIA-optimized container. This would ensure that our environment is identical to what you would deploy in a data center. ### Reproducible Benchmarking **Mozhgan Kabiri Chimeh** [3:05] Instead of just showing raw results, I want to show you the how. I built an automated benchmarking harness: every model run from 1.5B to 14B follows the same strict protocol. Environment isolation via Docker, three mandatory warmup runs, and background GPU metrics logging at 1 second interval. On the left, you're looking at the orchestrator script. For every execution, the script automatically generates a unique directory using a precise timestamp and a sanitized model ID. For every model run, it captures the full model's endpoint response and necessary metrics. On theright, you see the result: a clean versioned artifact of the run. It contains everything to verify the findings, the metadata, and the text results from the benchmarks. On the lowerright, you can see an example command for getting things started. ### Latency Measurement **Mozhgan Kabiri Chimeh** [4:14] Now, let's look at the actual measurement logic. In an AI application, end-to-end latency is important, but time to first token is the metric that defines the user's perceived performance. If the first token arrives instantly, the application feels responsive. And the script here shows how we timestamp the very first chunk of a streaming response. In this script, we're not just calling an API and waiting for a result; we're explicitly handling the streaming response from the VLLM server. If you look at the highlighted block in the stream_once function, you'll see the timestamping logic. To explore this in practice, I ran a series of experiments using VLLM on DGX Spark. I tested different models, from smaller instruction models to larger optimized variants, all under the same setup. Everything was served locally. The goal here wasn't to push theoretical limits, but to understand realistic behavior in a developer workflow. Let's dive into the raw performance data I captured. This bar chart represents the completion token per second across the test suite. ### Throughput Results **Mozhgan Kabiri Chimeh** [5:43] At the far left, you see the 1.5B instruct model delivering a massive 61.73 tokens per second. But the most interesting data point for us is the 14B NVFP4 model. Despite being nearly 10 times larger than the 1.5B model, it still achieves 20.19 tokens per second. I would say this is a critical engineering sweet spot. By leveraging NVIDIA's NVFP4 4-bit floating-point quantization, we're able to maintain a sophisticated, high-intelligent model at a throughput that is still faster than the average human reading speed. What becomes very clear here is how aggressively throughput drops as models scale, and how much quantization helps. Have a look at the 14B base model. It drops to just 8.4 tokens per second. This proves that on Blackwell hardware, the choice of quantization format is just as important as the hardware itself. It's what allows the DGX Spark to bridge the gap between research flow and production prototyping engine. The Spark allowed me to experiment with different precision formats locally and understand the trade-offs in the real time. ### First Token Latency **Mozhgan Kabiri Chimeh** [7:32] While throughput is a measure of raw power, time to first token is the metric that defines user experience. It determines whether an application feels instant or broken. In other words, it reflects how quickly the model starts responding. As we see in the results, the DGX Spark delivers exceptional responsiveness. The increase for larger models is expected: more parameters mean more computation before the first token is generated. Out of all from this chart, the interesting one is the comparison between the 14B parameter models, the base model, and the NVFP4 model. As we can see, the 14B NVFP4 is 3.4 times faster to first token than the unoptimized 14B model. A key takeaway here from this data is that memory capacity is not the same as the memory bandwidth. While the DGX Spark's 128 GB of unified memory allows us to fit massive models up to 200 billion parameters, our throughput is still governed by how efficiently we can move data. ### Key Takeaways **Mozhgan Kabiri Chimeh** [9:04] This is why NVFP4 is the hero here. It effectively increases our intelligence per byte, allowing a 14B model to feel as responsive as much a smaller one. After running all of these, three things stood out to me: I'm when to use the DGX Spark for steady state workloads, privacy-sensitive data, and rapid prototyping. It allows you to build and fine-tune locally with the exact same software stack used in DGX Cloud. Visit build.nvidia.com/spark to access the playbooks and software stack I used for these benchmarks. And in one line: run locally, iterate quickly, and when ready, scale to data center or cloud. That's the workflow DGX Spark enables. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.