A local LLM is a large language model that runs on your own computer instead of a company's cloud servers. Nothing you type gets sent anywhere, it works without an internet connection once it's downloaded, and there's no per-message bill. Running one well just takes the right hardware.

Here's what that hardware looks like, what software makes it possible, and why people bother when ChatGPT and Claude are sitting right there.

The quick answer

  • What it is: an open-weight AI model, like Llama or Mistral, downloaded and run on your own machine instead of through a cloud service like ChatGPT or Claude.
  • Minimum hardware: 8GB of VRAM (or unified memory on a Mac) for a usable 7B model, 16GB of system RAM, and 20 to 50GB of free storage per model.
  • Serious hardware: 24GB or more of VRAM, or 64GB or more of unified memory, to run 70B-class models that get closer to what cloud models can do.
  • The tools: Ollama for the simplest setup, LM Studio for a visual interface, and llama.cpp powering both underneath.
  • The real models: open-weight families like Meta's Llama, Mistral, Alibaba's Qwen, Google's Gemma, and DeepSeek, all free to download.
  • The reason: privacy, predictable cost, offline access, and control over exactly which model you're running, not raw intelligence. Cloud models are still smarter on average.

What makes an LLM "local"

Any large language model can run locally if it's been released as open weights, meaning the company that trained it published the actual model file instead of keeping it behind an API. Meta's Llama, Mistral, Alibaba's Qwen, Google's Gemma, and DeepSeek all do this. ChatGPT and Claude don't: their weights stay on the vendor's servers, so running either one locally isn't an option.

Running a model locally means downloading one of these open-weight files and using software on your own machine to load it into memory and generate responses, with no server in between.

What kind of computer you need

The number that matters most is VRAM, the memory on your graphics card, or unified memory if you're on a Mac. Model size sets the floor: at full precision, a model needs roughly 2GB of memory per 1 billion parameters. Quantization changes that math by compressing a model's weights to a lower precision; a 4-bit quantized version needs closer to 0.5 to 0.75GB per 1 billion parameters, with a small, often hard-to-notice hit to output quality.

That puts a 7B model, a small Llama or Mistral variant, comfortably in reach of a GPU with 8GB of VRAM. A 13B to 14B model wants closer to 12GB. Getting into 70B territory, the range most flagship open models live in, needs 24GB of VRAM at minimum and closer to 40GB to run comfortably. That usually means a high-end GPU like an RTX 4090, two GPUs splitting the load, or a Mac.

Macs handle this differently, and for this specific job, often better than a PC. Apple's M-series chips use unified memory, so the GPU and CPU draw from the same pool of RAM instead of a GPU being capped by its own dedicated VRAM. An M4 Max or M3 Ultra with 64GB or more of unified memory can run 70B models, and a 128GB configuration gets into 100B-plus territory, without a discrete graphics card at all.

Outside of VRAM, the rest of the build is less demanding: 16GB of system RAM as a floor, 32GB more comfortable, any modern multi-core CPU, and an NVMe SSD with room to spare, since model files alone run from a few gigabytes to well over 100GB each.

The software that runs the model

Three tools cover most of what people use to run models locally. Ollama wraps the whole process into a few terminal commands: install it, pull a model by name, then run it to start chatting, and it handles quantization and GPU setup on its own. LM Studio gives the same idea a visual interface, with a model browser and chat window instead of a terminal, built for people who'd rather click than type commands. Both run on llama.cpp underneath, the open-source engine that popularized GGUF, the file format most quantized models ship in today.

The real models people download

None of this works without models companies choose to release publicly. Meta's Llama family is the most widely used open-weight model in general use. Mistral, out of France, focuses on multilingual and coding work. Alibaba's Qwen line covers a wide size range, from small models that fit on a phone to large ones built to compete with closed models. Google's Gemma is built specifically for on-device and edge use. DeepSeek's models are known for reasoning performance at a lower compute cost. All of them publish specific parameter sizes, commonly 7B, 13B, and 70B, and all of them are free to download and run without an account or API key.

Why run a local model instead of just using ChatGPT or Claude

This is the real question, since cloud models are still smarter on average than anything that fits on a home GPU. A few reasons keep people building local setups anyway.

Privacy is the biggest one. Data sent to a cloud model leaves your machine, even when the company promises not to train on it. Healthcare records, legal documents, and a company's own source code often can't leave an organization's systems under regulations like HIPAA, so a local model is the option that keeps everything on hardware the organization controls.

Cost is the second reason. Cloud APIs charge per token. A local model costs nothing per message once the hardware is paid for, which matters most for high-volume use: batch-processing documents, running automated agents around the clock, or building a product that would otherwise carry an unpredictable bill.

Offline access matters for anyone working somewhere without a reliable connection, or inside a secure facility that doesn't allow outbound internet traffic at all.

The last reason is control. A cloud model can change overnight when a vendor pushes an update or retires an older version. A local model stays exactly as downloaded until you decide to change it, and you can fine-tune it, restrict what it will and won't respond to, or keep running an older version indefinitely, on your own schedule instead of someone else's.

Is it worth setting up

For most everyday chat use, a cloud model like Claude is still the better choice on quality and convenience. Local setups earn their keep for specific jobs: handling sensitive data, running at high volume, working offline, or wanting a model that stops changing without notice.

If your organization is weighing that decision, or trying to figure out how to introduce AI tools to a team that's never used them before rolling any of this out, that's worth a conversation.

Sources:
ollama.com · github.com/ggml-org/llama.cpp · plugable.com: GPU VRAM requirements for local LLMs · llm-stats.com: hardware requirements for running LLMs locally