NVIDIA demos TensorRT Edge-LLM on DGX Spark

Image: NVIDIA

On October 30, 2026, at 5:33 pm Brasília time, NVIDIA published a guide to run AI models on DGX Spark with TensorRT Edge-LLM. The example uses Nemotron 3.5 Lightning and shows how to serve a local API.

The material is relevant to anyone prototyping embedded AI applications. It describes the path from downloading a model, creating an optimized inference engine, to sending prompts via terminal or via an OpenAI compatible API.

It is not a new model, an availability announcement for DGX Spark, or a hosted API. It is a technical tutorial from NVIDIA, accompanied by performance measurements made by the company itself in a defined scenario.

In this article you will see:

• The role of TensorRT Edge-LLM and DGX Spark in the flow
• The versions and dependencies cited by NVIDIA
• What the example does with Nemotron 3.5 Lightning
• How to interpret the speed measurement without generalizing too much

What NVIDIA published about TensorRT Edge-LLM

The publication describes TensorRT Edge-LLM as a C++ inference runtime aimed at embedded workloads. Runtime is the layer that loads the model and executes its operations during use, in this case with optimizations from the NVIDIA platform.

The tutorial uses the command line interface trtedge. It downloads the indicated model and prepares a TensorRT engine optimized for inference on DGX Spark. This way, developers can try a model and test an application before moving the workflow to a compatible edge device.

The example is text based. NVIDIA chose the Nemotron 3.5 Lightning NVFP4 checkpoint, available on Hugging Face, and demonstrates a simple conversation in the terminal. The full name of the artifact is NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4.

This approach is different from simply calling a remote model via an API. The first test happens on the equipment itself, which allows studying the behavior of local inference and the integration of the service with the software under development.

The approach also aligns with interest in models that can be run outside a closed API. On the blog, we explain how the Mistral opened the weights of Large 4 for coding tasks, another example of models and tools aimed at own workflows.

How DGX Spark connects prototype and edge device

The DGX Spark mentioned in the guide has a 64 GB configuration and is presented by NVIDIA as a compact workstation for prototyping, tuning and testing models and agents. The tutorial uses this system as a development environment, not as a universal replacement for any server or embedded equipment.

TensorRT Edge-LLM is also described as a foundation for developing on DGX Spark and deploying on compatible platforms, such as Jetson Thor, DRIVE AGX Thor and IGX Thor. This can reduce the distance between prototype and target device, but it does not eliminate the need to validate each hardware, software and model combination.

The official image from the NVIDIA DGX Spark product page shows compact computers from the family in different finishes. It helps visualize the class of equipment used in the example, but it does not represent a performance measurement nor guarantee that all configurations have the same result.

The partnership between Mistral and NVIDIA on AI engineering tools shows another application of the NVIDIA stack.

Guide requirements and what the trtedge CLI does

The environment described requires CUDA 13, TensorRT 10.16.1.11 or a later version in the 10.x series, and Python between versions 3.10 and 3.12. NVIDIA states that the example was tested with TensorRT Edge-LLM 0.11.1 and TensorRT 10.16.1.11.

On first run, downloading the model takes about 21 GB, plus space for the inference engine. Files are cached and compatible engines can be reused in later runs.

How to test today on DGX Spark

The documented path requires a DGX Spark with the supported software stack. The NVIDIA page includes package installation, a dependency check, starting a chat session, and the command to serve the model locally. Check the official TensorRT Edge-LLM documentation before reproducing the steps, because compatible versions and models can change.

In the example, the server listens on 127.0.0.1 on port 8000. This keeps the demo on the device itself. The publication does not describe authentication, remote access, or a production deployment, so the local example should not be treated as an API ready for public exposure.

What the Nemotron measurements show and what they do not prove

NVIDIA compares normal generation with DSpark, a form of speculative decoding. In this method, a draft model proposes tokens and the main model verifies several proposals in one pass. The company tested three speculative tokens.

On HumanEval prompts, the reported average speed went from 73.4 to 126.9 tokens per second with DSpark enabled. Median time to first token went from 98.1 to 98.7 milliseconds. The generation metric excludes time to first token, so the two measures describe different stages of the response.

The table below reproduces the values from NVIDIA's publication. The results are from the manufacturer, not from an independent evaluation, and should not be read as a promise for other models, prompts, or devices.

The measurement was done on a 64 GB DGX Spark with DGX OS 7.6.0, batch size one, one simultaneous streaming request, output limited to 128 tokens and reasoning enabled. Total system memory reached 27,9 GB with DSpark, including the operating system.