How to run AI locally: Ollama and LM Studio step by step

How do you run AI locally? The easiest way is Ollama or LM Studio: you install the program, download an open-weight model and chat with it with no internet connection and no subscription. Ollama is one command in the terminal; LM Studio is a desktop app with windows and a model browser. To start, all you need is a computer with about 8 GB of free graphics card memory, or a Mac with Apple Silicon.

Why run AI locally at all?

New open-weight models come out almost every week. Some are giants like Reflection Beam (MoE 501B), which you won’t run on home hardware, but there are also smaller versions and “Flash”-class models that fit on an ordinary laptop more and more often.

What you get:

1. Privacy. Your prompts and files never leave your computer.
2. No limits and no bills. You only pay for electricity (and wear on your hardware).
3. Works offline. Once the model is downloaded, you don’t need the internet.
4. A local API. You can connect your own scripts and tools to the model on localhost, just like a cloud API.

There’s one problem: a local model is only as good as your hardware allows. The most powerful cloud models are still out of reach for a home GPU.

What hardware do you need?

Memory matters most: your graphics card’s VRAM, or unified memory on a Mac.

• Ollama, in its official quickstart, recommends 8 GB of free VRAM (or unified memory on a Mac) for the example model, which is about a 7.2 GB download. With less VRAM, Ollama falls back to regular RAM, but responses will be slower. A larger context window also eats more memory.
• LM Studio, according to its system requirements, needs an Apple Silicon Mac (M1–M4) with macOS 14 or newer, with 16 GB of RAM recommended (it will run on 8 GB too, but only with smaller models). On Windows it recommends 16 GB of RAM and at least 4 GB of VRAM, and an x64 processor must support AVX2. On Linux it requires Ubuntu 20.04 or newer.

The golden rule: the model file has to fit in memory with room to spare for the context. The model library lists the sizes, e.g. Gemma 4 in Ollama comes in variants from about 4.6 GB (e2b) to about 20 GB (31b).

How to run AI locally with Ollama

Ollama is a free, open-source program (MIT licence, GitHub repository). It runs on macOS, Windows and Linux, and inside it is a server that handles the models in the background.

1. Install. On macOS, download the app from the Ollama website. On Linux, one command is enough:

curl -fsSL https://ollama.com/install.sh | sh

On Windows, in PowerShell:

irm https://ollama.com/install.ps1 | iex

2. Your first model. In the terminal, run:

ollama run gemma4:e2b

Ollama downloads the model itself and opens a chat. End the conversation with /bye. On Linux, if the server isn’t running, start it first with ollama serve.

3. Managing models. A few commands from the CLI documentation that come in handy from day one:

ollama pull gemma4   # download a model without running it
ollama ls            # list downloaded models
ollama ps            # models running right now
ollama stop gemma4   # stop a model
ollama rm gemma4     # delete a model from disk

Interestingly, Ollama can also run coding tools on a local model. The command ollama launch claude starts Claude Code, and ollama launch claude --model qwen3.5 does the same with a model of your choice. On a Mac, the Ollama app also has an Apps tab that connects models to the Claude app or ChatGPT Desktop (in the latter case, only in Codex mode).

How to run AI locally in LM Studio

LM Studio is a desktop app for people who’d rather click than type in a terminal. It has a chat, a model browser for Hugging Face, MCP support and a local API server. Under its licence terms it’s available for personal use and internal use at work, although some features may be paid.

1. Install. Download the installer from the LM Studio website and launch the app.
2. Download a model. Open the Discover tab (⌘ + 2 on a Mac, Ctrl + 2 on Windows and Linux) and type a name, e.g. “gemma” or “qwen”. You can also paste a Hugging Face link.
3. Pick a variant. Names like Q4_K_M or Q8 refer to quantization: the model gets smaller at the cost of some quality. The documentation suggests choosing a 4-bit variant or higher if your hardware allows it.
4. Chat. Load the model and type as you would in any AI chat.

LM Studio also has its own CLI, lms, installed together with the app (you just need to launch the app once). According to the CLI documentation, you can do the same from the terminal:

lms get llama-3.1-8b          # download a model
lms ls                        # list models on disk
lms chat my-model             # chat in the terminal
lms server start              # start the local API server

Ollama or LM Studio?

At heart, both programs do the same thing; they differ in how you work with them.

	Ollama	LM Studio
Interface	terminal (plus a simple app)	windowed app (plus the lms CLI)
Licence	open source (MIT)	proprietary, for personal and internal business use
Finding models	library on ollama.com	Hugging Face browser in the app
Local API	http://localhost:11434/v1/	http://localhost:1234/v1

If you like the terminal or want to hook a model up to scripts and agents, Ollama is the simpler place to start. If you’d rather browse and compare models first, LM Studio is more comfortable for that.

How to connect a local model to your own code

Both programs expose an OpenAI-compatible API, so all you have to do is change the address in the regular openai library. An example from the Ollama documentation:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",  # required by the library, ignored by Ollama
)

reply = client.chat.completions.create(
    model="gpt-oss:20b",
    messages=[{"role": "user", "content": "Give me a one-line fun fact."}],
)
print(reply.choices[0].message.content)

In LM Studio, you turn the server on with the Start server toggle in the Developer tab or with lms server start. In the documentation examples, the server runs on port 1234 (http://localhost:1234/v1) and supports, among others, /v1/chat/completions, /v1/responses and /v1/embeddings.

That opens the door to more interesting things than chat: local agents, searching your own notes, or testing new open-weight models on launch day, such as MiniMax M3.1 Flash. And if you want to see how models like these are served in production, there’s more in the post about DeepSeek DeepInfer.

Frequently asked questions

Is local AI free?

Ollama is free and open source. LM Studio is available for personal use and internal use at work, although under its terms some features may be paid. The open-weight models themselves are free to download, but each has its own licence that’s worth checking.

Can you run AI locally without a graphics card?

Yes, but more slowly. Without VRAM, Ollama uses regular RAM. In that case it’s best to pick a small model, e.g. gemma4:e2b.

How much memory does a local model need?

Roughly as much as the model file weighs, plus room for the context. For the example model in the Ollama quickstart (about 7.2 GB), 8 GB of free VRAM is recommended.

Does a local model work without the internet?

Yes. You only need the internet to download the program and the model.