Guide

What is a local-inference PC for AI agents?

A local-inference PC runs large language models and AI agents on your own hardware instead of a cloud API. Your prompts and data never leave the machine. Here is what that means, why it matters now, and how the hardware actually works.

The plain definition

A local-inference PC is a computer built to serve LLM inference on-premises. Instead of sending each request to a cloud endpoint and paying per token, you point your tools at a box on your desk. It loads the model into memory once and answers locally, at a fixed cost, fully private.

Why it matters now

Two things changed. Open models in the 27B class (Qwen, GLM, DeepSeek, Llama) now match what you needed a frontier API for a year ago. And the software to run them fast on local hardware, custom GPU kernels and speculative decoding, finally exists. Together they make a desk-sized box a real alternative to a recurring cloud bill.

How the hardware works

Running a large model quickly comes down to two resources working together:

• Fast VRAM for the hot weights. The Radeon AI PRO R9700 provides 32 GB of GDDR6, 64 compute units, up to 191 TFLOPS at FP16, and 640 GB/s of bandwidth.
• Large unified memory for long context and bigger models. Strix Halo provides 128 GB of LPDDR5X, 40 graphics cores, up to 60 TFLOPS, and 256 GB/s of bandwidth.

The trick is pairing the two and then tuning the inference engine to the exact silicon. Most machines run a general-purpose runtime at stock and leave four to six times the throughput on the table. A tuned stack does not.

What to look for

• Throughput on a real model. Ask for tokens per second on a named 27B model, not a synthetic score.
• Memory pairing. A GPU alone is not enough; you want VRAM plus unified memory.
• A tuned engine, not a stock runtime.
• Tool compatibility. It should speak the OpenAI or Anthropic API so your existing agents just work.
• Privacy and support. Fully local, with a warranty.

A worked example

In our published CUDA reference work, a tuned lucebox-hub stack with custom kernels and DFlash speculative decoding runs Qwen3.5-27B Q4_K_M at up to 207 tok/s. Long-context prefill, normally the slow part, drops from minutes to seconds with speculative prefill. Production Radeon AI PRO R9700 measurements will be published after validation.