Download and run LLMs on your Mac.

1Where to find models

Hugging Face hosts most open-weight models. The same model usually comes in several builds, and the format decides which app can run it:

Not sure which runtime to use? MLX vs Ollama vs llama.cpp compares them. New to all of this? Start with running an LLM on a Mac.

2Pick a size that fits your memory

A model's weights have to fit in memory with room to spare. MLX stores a quantized weight as its bits plus a shared scale and bias for every group of 64 weights, about half a bit more per weight (MLX). So 4-bit costs about 0.56 GB per billion parameters and 8-bit about 1.06 GB.

macOS lets the GPU use only part of unified memory by default; Compare assumes about three quarters, and mlx-lm's README shows how to raise the limit on macOS 15 or later with sudo sysctl iogpu.wired_limit_mb=N (mlx-lm). For mixture-of-experts models, total parameters decide memory and active parameters decide speed. Compare does this arithmetic for every popular model and every Apple chip, and the best Mac for local LLMs explains why bandwidth sets the speed.

3Quantization: 4-bit, 8-bit and the rest

Quantization stores each weight in fewer bits, so the model takes less memory and runs faster, at some cost in quality. 4-bit is the usual starting point on a Mac. 8-bit is closer to the original and twice the size. Below 4 bits, files get small but answers get worse sooner, so try them only when a bigger model wouldn't fit otherwise.

• MLX builds say it in the name: -4bit, -8bit, and sometimes mixed-precision builds that keep sensitive layers at more bits.
• GGUF files use llama.cpp's names: Q4_K_M, Q5_K_M, Q8_0 and so on. When you run a Hugging Face GGUF repository in Ollama without naming one, it picks Q4_K_M if it's there (Hugging Face).
• Make your own MLX build from original weights: mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 -q writes a 4-bit copy (mlx-lm).

4Disk space and where models are kept

A download is about as big as the weights in the table above, so a 4-bit 8B model is a few gigabytes and a 70B model tens of gigabytes. Trying several sizes or builds of the same model adds up quickly.

mlx-lm and other Hugging Face tools keep downloads in ~/.cache/huggingface/hub; set HF_HOME or HF_HUB_CACHE to move it. hf cache ls shows what's there and hf cache rm deletes a model (Hugging Face). Ollama and LM Studio keep models in their own folders and manage them themselves.

5Gated models and Hugging Face tokens

Some models, Llama among them, are gated: you need to accept the maker's terms before you can download them (Hugging Face).

1. Sign in to a free Hugging Face account and open the model's page.
2. Accept the terms with the button on the page (by default Agree and send request to access repo). Some makers approve requests by hand, which takes longer.
3. Create an access token in your Hugging Face settings.
4. Give the token to your tools: run hf auth login in Terminal for mlx-lm and other Hugging Face tools (Hugging Face), or paste it into the app's Hugging Face token field.

6Run it: four ways

mlx-lm: Apple's MLX in Terminal

Install it with pip, then name any MLX model on Hugging Face; the first run downloads it (mlx-lm).

pip install mlx-lm
mlx_lm.generate --model mlx-community/Llama-3.2-3B-Instruct-4bit --prompt "How tall is Mt Everest?"
mlx_lm.chat --model mlx-community/Llama-3.2-3B-Instruct-4bit

mlx_lm.server serves a model on an OpenAI-style API at localhost:8080 for other apps (mlx-lm):

mlx_lm.server --model mlx-community/Llama-3.2-3B-Instruct-4bit
curl localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Say this is a test!"}]}'

LM Studio: an app with a model browser

LM Studio searches and downloads models from Hugging Face inside the app and runs MLX models on Apple Silicon as well as GGUF through llama.cpp. It needs Apple Silicon and macOS 14 or later, and recommends 16 GB of memory or more (LM Studio).

Ollama: one command

Ollama runs models from its own library, and any GGUF repository on Hugging Face directly, optionally with a quantization tag (Hugging Face):

ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:IQ3_M

MLX Harness: our free app

MLX Harness is our free Mac app, built on vMLX: it searches Hugging Face for MLX models, chats, runs tools and image models, and serves everything on a local API. Its gateway also uses port 8080, so don't run it alongside mlx_lm.server on the same port. When you're not using the Mac, MLX Node can lend it to MLX Network instead.

7Questions

Where do I download LLMs for a Mac?

From Hugging Face. Look for MLX builds, many published by mlx-community, or GGUF files. LM Studio and Ollama can search and download for you.

Which model size should I download?

One whose weights fit comfortably in your Mac's memory: at 4-bit about 0.56 GB per billion parameters, so about 4.5 GB for an 8B model and 39 GB for 70B, plus room for context. Compare on mlx.app checks it for your Mac.

MLX or GGUF on a Mac?

Both run well on Apple Silicon. MLX builds are made for Apple's MLX and run in mlx-lm, LM Studio and MLX Harness; GGUF files run in llama.cpp, Ollama and LM Studio.

Do I need a Hugging Face account?

Not for public models. Gated models need a free account, an accepted licence and an access token.

Can I run downloaded models offline?

Yes. Once the files are on your Mac, the model runs without an internet connection and your prompts stay on the Mac.