Using artificial intelligence doesn't always have to happen via the cloud. More and more users are choosing to run a Large Language Model (LLM) locally on their own computer. This might sound complex, but with modern tools, it is now as easy as installing a web browser.

Why run an LLM locally?

The main reason is privacy. When you run a model locally, not a single prompt, document, or line of code leaves your computer. Your data is not used to train models for big tech companies. Additionally, you are completely independent of an internet connection, won't suffer from server outages, and don't pay monthly subscription fees or API rates.

General hardware requirements

An LLM requires memory (RAM/VRAM) to function. Although the requirements depend on the size of the model, you can use the following general guidelines:

Memory (RAM/Unified Memory): Minimum of 8GB for small models, 16GB or more is highly recommended for a smooth experience.
Processor (CPU/GPU): A modern multi-core CPU is sufficient for basic use, but a powerful graphics card (NVIDIA GPU with enough VRAM) or an Apple Silicon Mac (M1/M2/M3) makes text generation much faster.

The best tools right now

You don't need any programming knowledge to get started. Two tools currently stand out from the rest:

LM Studio: The perfect choice for beginners. It offers a clean graphical interface that closely resembles well-known cloud chatbots. You can search for new models directly from the app and download them with a single click.
Ollama: A lightweight, lightning-fast tool that works primarily via the command line (terminal). It is a favorite among developers because it integrates seamlessly with countless other applications and web interfaces.

Step-by-step guide: Your first local LLM (via LM Studio)

Step 1: Download and install

Go to the official LM Studio website and download the version suitable for your operating system (Windows, macOS, or Linux). Install the application just like you would install any other program.

Step 2: Search for a model

Open the application. Use the built-in search bar and type in a popular, efficient model, such as Llama 3 or Mistral. LM Studio will immediately show you whether a model is suitable for the amount of memory in your computer.

Step 3: Download

Click the download button next to the model. Make sure to choose a so-called 'Quantized' version (often indicated by Q4 or Q5); these are compressed and run much more smoothly on consumer hardware without much loss in quality.

Step 4: Start chatting!

Navigate to the chat icon in the left menu, select the model you just downloaded from the dropdown menu at the top, and start typing. You now have your own, completely private AI assistant!

Want to learn more about the technical inner workings of language models, or learn how to integrate them into your own workflows? Take a look at our educational hub for in-depth articles and tutorials.