In partnership with

What makes a great ad in 2026?

If you want to know the core principles of high-performing advertising in 2026, join our educational webinar with award-winning creative strategist Babak Behrad and Neurons CEO & Founder Thomas Z. Ramsøy.

They’ll show you how standout campaigns capture attention, build memory, and anchor brands. You’ll walk away with clear, practical rules to apply to your next campaign.

You’ll learn how to:

  • Apply neuroscientific principles to every campaign

  • Build powerful branding moments into your ads

  • Make your ads feel relevant to your audience

Master the art of high-impact campaigns in an era of AI-generated noise and declining attention spans

Save your seat

Understanding llama.cpp

Technical Explainer

Understanding llama.cpp

How local AI inference works under the hood
85K
Stars
1,200
Contributors
4,000
Releases

If you have used Ollama, LM Studio, or Jan to run AI models on your computer, you have used llama.cpp. This C++ library handles the actual inference computation for most local AI applications.

This issue covers what llama.cpp does, how it works, and when to use it.

How Local AI Tools Connect

Ollama
LM Studio
Jan
GPT4All
▼ ▼ ▼ ▼
llama.cpp
Inference Engine
▼ ▼ ▼
CPU
GPU
Apple Silicon
Apps you interact with    Shared engine    Hardware

What llama.cpp Does

llama.cpp is an inference engine written in C/C++ with no external dependencies. It loads model files and generates text on your local hardware.

Three properties make it useful: runs on CPUs without requiring GPUs, reduces memory usage through quantization, and stores models in single portable files.

How Quantization Works

Quantization: Compressing Model Weights

FP16 (original)
16 bits
↓ Quantization ↓
Q8_0
8 bits
Q4_K_M
~4 bits
Q2_K
~2 bits
Each weight uses fewer bits = smaller file = less RAM needed

Memory Required: 7B Parameter Model

FP16
14 GB
Q8_0
7.2 GB
Q5_K_M
4.8 GB
Q4_K_M
4.1 GB
Q2_K
2.7 GB
Q4_K_M reduces memory by 70% with minimal quality loss

Performance by Hardware

Tokens Per Second (7B Model, Q4)

M2 Ultra
108 t/s
RTX 4090
95 t/s
M1 Max
55 t/s
Desktop CPU
15 t/s
15 t/s = conversational speed

The GGUF File Format

One File Contains Everything

HEADER Magic number, version
METADATA Architecture, context length
TOKENIZER Vocabulary, merge rules
TENSOR INFO Names, shapes, quant types
WEIGHTS Quantized model parameters
No separate tokenizer files. No config files. Just one .gguf file.

Supported Hardware

Apple Silicon: M1, M2, M3, M4 via Metal
NVIDIA: CUDA with FlashAttention
AMD: ROCm on Linux
CPUs: AVX2, AVX-512, ARM NEON
Cross-platform: Vulkan, OpenCL

When to Use Local Inference

Data privacy: Sensitive information stays on device
Offline access: Works without internet
Cost control: No per-request charges
Experimentation: Test models without rate limits

Getting Started

Use Ollama for the simplest setup, or build llama.cpp directly:

# Build
git clone github.com/ggml-org/llama.cpp
cd llama.cpp && make

# Run
./llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

Links

Source: github.com/ggml-org/llama.cpp
Models: huggingface.co (search GGUF)

Questions? Reply to this email.

ResearchAudio.io

Keep Reading