llama.cpp Tutorial: Run Local LLMs with OpenAI API

Introduction

llama.cpp is a high-performance large language model inference framework developed by Georgi Gerganov. Written purely in C/C++, it has zero external dependencies and supports quantization ranging from 1.5-bit to 8-bit precision. It enables running 7B-scale models on consumer CPUs, and supports offloading computation to GPUs via CUDA, Metal, Vulkan and SYCL acceleration. As of August 2026, the latest release is build b10369 with more than 123,000 GitHub stars.

The core executable llama-server delivers a ready-to-use REST API fully compatible with OpenAI specifications. Developers can run open-source models locally on macOS, Linux and Windows. This guide covers cross-platform installation workflows, GPU acceleration compilation, GGUF model selection, production configuration of llama-server, and practical tuning strategies for low-memory hardware. We will also clarify the differences between llama.cpp and Ollama to help engineers select appropriate tooling for local LLM deployments.

1. llama.cpp Overview & Comparison with Ollama

llama.cpp operates as a low-level inference engine. It directly loads GGUF-formatted model weights, provides command-line tools and REST endpoints, and grants full control over inference hyperparameters, KV cache settings and prompt formatting. It is ideal for developers who need customised pipeline integration and fine-grained performance tuning.

Ollama serves as a higher-level wrapper built on top of llama.cpp (macOS builds migrated to Apple MLX starting in March 2026). It simplifies model management via intuitive commands like ollama pull and ollama run, making it suitable for users who prioritise rapid experimentation without deep configuration.

Relationship summary:

• Learn llama.cpp to understand underlying mechanisms: quantization, layer offloading and KV cache design.
• Use Ollama as a high-level black-box tool.
• Both frameworks rely on GGUF as the standard model storage format.

2. Cross-Platform Installation Workflows

2.1 macOS (Recommended: Homebrew)

brew install llama.cpp

The Homebrew distribution tracks official releases automatically and includes native Metal GPU acceleration, enabled by default on Apple Silicon hardware.

Verify installation:

llama-cli --version
llama-server --version

2.2 Linux (Conda or Homebrew)

# Conda-Forge distribution (includes CUDA / Vulkan builds)
conda install -c conda-forge llama.cpp

# Alternatively, Homebrew for Linux
brew install llama.cpp

2.3 Windows (Three available options)

Option 1: Winget (Simplest)

winget install llama.cpp

Option 2: GitHub Release Precompiled Binaries (Recommended)

Download prebuilt archives matching your graphics hardware from the official repository releases:

After extraction, execute llama-server.exe directly within the extracted directory. No further installation steps are required.

Option 3: Source Compilation (For custom GPU acceleration builds)

Package manager binaries already include GPU support. Manual compilation is required only for customised builds.

2.4 Source Compilation for GPU Acceleration

NVIDIA CUDA (Linux / Windows)

Prerequisite: CUDA Toolkit 12.x

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8

Apple Silicon Metal (macOS)

Metal acceleration is enabled by default, no extra flags required.

cmake -B build
cmake --build build --config Release -j 8

AMD GPU (Vulkan, cross-platform)

cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j 8

Compiled binaries will be located inside build/bin/.

3. Model Acquisition: GGUF Format and Quantization Selection

llama.cpp exclusively supports GGUF model files. You can pull models directly from Hugging Face via the built-in CLI:

# Directly download and run GGUF models from Hugging Face
llama-cli -hf ggml-org/Qwen3.5-0.8B-GGUF

For manual model downloads:

pip install huggingface-hub
huggingface-cli download \
Qwen/Qwen3.5-0.8B-GGUF \
--local-dir ./models

Quantization Selection Guide

Quantization labels encode precision and compression tradeoffs. Taking a 7B parameter model as an example:

Naming convention explanation:

• Q = Quantization; numeric value = bit width
• K = improved quantization algorithm
• M = medium, S = small, L = large quality preset

Approximate memory requirement formula: Model size (GB) + Context window overhead + KV cache allocation

4. Basic Inference with llama-cli

Basic interactive chat command template:

llama-cli \
-m ./models/qwen3-8b-q4_k_m.gguf \
-c 8192 \
-i -ins

Key common parameters:

• -m: Path to GGUF model file
• -ngl N: Offload N transformer layers to GPU (higher values accelerate inference)
• -c: Maximum context window token count
• -n: Maximum tokens generated per completion
• -i -ins: Enable interactive instruction mode

5. llama-server: Launch OpenAI-Compatible API Endpoint

llama-server is the most critical component for production integration. It spins up a local HTTP server implementing the OpenAI REST schema. Existing OpenAI client code requires minimal modification; simply redirect the base URL to the local instance.

Minimal Startup Command

llama-server \
-m ./models/qwen3-8b-q4_k_m.gguf \
--host 0.0.0.0 \
--port 8080 \
--ngl 999

After startup, available endpoints:

• Chat completions: http://localhost:8080/v1/chat/completions
• Model listing: http://localhost:8080/v1/models

Production Optimized Configuration (Multi-request + Flash Attention)

llama-server \
-m ./models/qwen3-8b-q4_k_m.gguf \
--host 0.0.0.0 \
--port 8080 \
-c 8192 \
--parallel 4 \
--flash-attn \
--api-key "your-local-key"

Parameter breakdown:

• --parallel: Concurrent request slots
• --flash-attn: Flash Attention implementation, reduces KV cache memory consumption by roughly 30–50%
• --api-key: Enforce authentication for public network deployments

Client Example (Python OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1/",
    api_key="your-local-key"
)

completion = client.chat.completions.create(
    model="qwen3-8b-q4_k_m",
    messages=[{"role": "user", "content": "Hello"}]
)
print(completion.choices[0].message.content)

Node.js and other OpenAI client libraries work identically by adjusting the base URL value.

6. Memory Limitation Mitigation & Tuning Strategies

Many consumer GPUs lack sufficient VRAM to fully load larger models. The -ngl flag enables hybrid CPU/GPU execution: layers assigned to GPU run faster, remaining layers execute on system RAM.

Sample hybrid offloading commands:

# Offload 32 layers to GPU, remaining layers on CPU
llama-cli -m model.gguf -ngl 32

# Offload 20 layers for GPUs with limited VRAM
llama-cli -m model.gguf -ngl 20

Additional memory-saving techniques:

1. Choose higher compression quantization variants (Q4_K_M > Q5_K_M)
2. Enable --flash-attn to cut KV cache overhead
3. Reduce context window -c to the minimum required by your workload
4. Lower --parallel concurrency limit to reduce simultaneous KV cache allocations

7. Multi-GPU Configuration

llama.cpp automatically distributes workloads across all visible GPUs by default. You can restrict visible devices using environment variables:

# Only use GPU index 0 and 1
CUDA_VISIBLE_DEVICES=0,1 llama-server -m model.gguf

Advanced multi-GPU splitting logic is documented in the official docs/multi-gpu.md.

8. Common Troubleshooting

1. Slow inference speed: Verify -ngl layer offloading value; confirm GPU acceleration compilation flags are enabled.
2. Cannot access API remotely: Ensure --host 0.0.0.0 instead of 127.0.0.1; check firewall port rules.
3. Malformed chat outputs: Confirm the model’s native chat template and use -ins instruction mode flag.
4. Out-of-memory crashes: Reduce context window size, lower parallel concurrency, or switch to a more aggressive quantization variant.

9. Deployment Architecture Considerations

For individual developers, standalone llama-server instances are sufficient for local prototyping. When scaling multiple local LLM backends alongside external model services, teams can streamline routing and access control. 4sapi functions as an API gateway to unify authentication, load balancing and traffic management across heterogeneous LLM endpoints.

Organizations operating persistent local inference fleets often combine llama-server instances with monitoring stacks to track token throughput, memory usage and request latency. Self-hosted llama.cpp deployments avoid third-party API data leakage and deliver predictable long-term cost profiles compared to cloud model services.

Conclusion

llama.cpp provides a lightweight, highly configurable foundation for running open-source LLMs locally across macOS, Linux and Windows. The fastest installation path uses official package managers: Homebrew for macOS, Conda for Linux, and Winget or precompiled archives for Windows.

For most end users, Q4_K_M quantization strikes the optimal balance between model size, memory footprint and output quality. The built-in llama-server delivers drop-in OpenAI-compatible APIs, enabling seamless migration from proprietary cloud LLMs to self-hosted local alternatives. When hardware resources are constrained, hybrid CPU/GPU layer offloading and Flash Attention substantially reduce memory pressure.

Always reference the official GitHub repository for the latest parameter definitions and feature updates, as active development continues to expand hardware support and inference optimizations.