What is GGUF? Meaning and examples

What is GGUF?

GGUF: File format for storing AI models optimized for local inference, popularized by llama.cpp. Supports quantization (4, 5, 8 bit) and metadata, allowing large models to run on modest CPU or GPU.

Updated on 11/10/2026 · By Marcelo Santiago

How it works

GGUF (GPT-Generated Unified Format) is a binary format for storing AI model weights, created for the llama.cpp project. It replaces the previous format (GGML) and adds support for metadata, multiple quantizations and better organization.

A GGUF file contains: the model weights (quantized or not), metadata (name, architecture, tokenization parameters) and quantization information. All in a single file, easy to distribute and load.

The format is optimized for inference on CPU and GPU, with support for mmap (memory mapping), which allows the model to be loaded without copying to RAM. This reduces startup time and memory usage.

Practical example

On Hugging Face, you can find models like "Llama-3.1-8B-Instruct-GGUF" with several variants: Q4_K_M (4 bits, medium), Q5_K_M (5 bits, medium), Q8_0 (8 bits). You download the corresponding .gguf file and run it with llama.cpp, Ollama or LM Studio.

With a Q4_K_M file of 4.9 GB, you can run an 8 billion parameter model on a laptop with 8 GB of RAM, without a GPU. The quality is slightly lower than the original 16 bit model, but the speed and cost make up for it.

What it's for

It is used to distribute and run AI models locally, on modest hardware. It is the standard format for local inference and for experimenting with open models.

Difference between GGUF and other formats

GGUF is optimized for inference (llama.cpp, Ollama). Formats like Safetensors or PyTorch (.bin/.pt) are optimized for training and research. GGUF supports native quantization; Safetensors does not. To run locally, GGUF is the standard; to train or fine tune, use Safetensors.

VS Code Extension

Enjoyed learning about GGUF? Put it into practice with AI

Orchestrate squads of agents (Claude, OpenCode, Codex, Antigravity and more) in VS Code with complete control, isolated git worktrees, and your own APIs or subscriptions with zero markup.

Parallel multi-agentsAPIs or subscriptionsAgents for multitasking

Explore bridgecode

Frequently asked questions about GGUF

Is GGUF the same as GGML?

No. GGUF replaced GGML in 2023. GGUF adds metadata, better organization and support for more advanced quantizations. GGML is considered legacy.

Can I convert any model to GGUF?

Yes, using tools like llama.cpp (convert.py). Models in Hugging Face format (Safetensors) can be converted to GGUF with different quantization levels.

Does GGUF run on GPU?

Yes. llama.cpp supports layer offloading to GPU (CUDA, Metal, Vulkan). You can run part of the model on GPU and part on CPU, depending on available VRAM.