The moment you successfully run an LLM locally for the first time, a new world opens up. You are no longer dependent on cloud services, your data remains private, and you have complete control over the output. But this freedom comes with an important responsibility: managing the models themselves.
Large Language Models (LLMs) are massive files. A modest model with 8 billion parameters easily takes up 5 gigabytes of disk space. If you start experimenting with multiple models, your hard drive will fill up before you know it. Furthermore, on platforms like Hugging Face, it's easy to lose track: there are countless file formats and variants of the same model. In this guide, we explain how to find reliable models, which formats you need, and how to keep your storage organized.
Reliable Sources: Where Do You Get Your Models From?
The first step in good management is knowing where you can safely download. Unlike regular software, you often download language models as raw data files. It is crucial to obtain these from reputable sources.
Hugging Face
Hugging Face is by far the largest platform for open-weights models. It is often called the "GitHub of AI". Almost every new model is published here first by companies like Meta (Llama), Mistral, and Google (Gemma). However, anyone can upload files to Hugging Face. Always pay attention to the following when downloading:
- The author: Preferably download from the official account of the creator, or from well-known independent quantizers (people who shrink large models for home use). Big names in the community currently include Bartowski and mradermacher. Historically, TheBloke was the standard, but their repository has not been actively updated since early 2024.
- Number of downloads and likes: A model with tens of thousands of downloads and many likes (the heart icon) is generally safe and functional.
- The Community tab: Always check the discussions for a model. If there are issues with the file, you will read about them here first.
Ollama Library
If you use Ollama, you are in luck. The Ollama Library acts as a highly curated catalog. If a model is listed here, you can be sure it works out-of-the-box with the software and that the necessary settings (such as the prompt template) are already correctly configured. You simply download it via your command line or terminal.
Understanding Formats: GGUF vs. Safetensors
Once you know which local model to choose and you visit the page on Hugging Face, you will often see different tabs or filenames. It is important to understand the differences between the formats to prevent error messages.
GGUF (For CPU and Mac/Metal)
The .gguf format has become the undisputed standard for home users. It was developed by the team behind llama.cpp. The brilliant thing about GGUF is that the model, architecture, and all necessary metadata are combined into a single file. Furthermore, GGUF is optimized to efficiently distribute system memory between your graphics card (GPU) and your standard system memory (CPU/RAM). If you use software like LM Studio, Ollama, or GPT4All, you will almost always need a GGUF file.
Safetensors (For Pure GPU Power)
You encounter the .safetensors format when looking at the original, uncompressed model. This format was created to replace the old, insecure PyTorch Pickletensors (which could hide malicious code). Safetensors contain purely the weights of the model and are extremely secure. You primarily use these files when training models or when running very heavy inference servers via systems like vLLM, where the entire model must fit into the VRAM of your graphics card.
Rule of thumb: As a home user, always download the GGUF variant, unless you know exactly why you need the raw Safetensors.
Quantization: The Right Balance Between Speed and Smartness
A GGUF file never comes alone; there are often dozens of versions of the same model. This is due to quantization: the process of reducing the precision of the numbers in the model to save space and increase speed. (Also read our detailed article on what quantization actually is).
You will see terms like Q2_K, Q4_K_M, or Q8_0. The 'Q' stands for quantization, and the number after it represents the number of bits per parameter (the higher, the smarter, but also the heavier).
| Quantization Level | Characteristics | When to Choose? |
|---|---|---|
| Q2 / Q3 | Heavy compression, noticeable loss of logic and language skills. | Only if your hardware is severely underpowered. |
| Q4_K_M | The "Golden Standard". Excellent balance between small size and preservation of intelligence. | Recommended for 90% of users. |
| Q5_K_M | Slightly smarter than Q4, marginally larger. Especially useful for math or programming. | If you have just a bit of extra VRAM to spare. |
| Q6 / Q8 | Large files, the output is virtually identical to the original (FP16). | If you have plenty of hardware capacity and do not want to make compromises. |
To check which variant fits your computer, we recommend consulting our guide on hardware for local LLMs. Always ensure that the GGUF file fits comfortably within the total capacity of your RAM (and ideally your VRAM), with at least 2 to 3 GB of breathing room for the operating system and the context window.
A Practical Step-by-Step Plan for Model Management
Downloading models is simple, but getting them onto your system in an organized way requires some planning. How you do this depends heavily on the software you use.
Option A: Management via the Terminal (Ollama)
Ollama makes model management incredibly easy via the command line interface (CLI). You don't have to search for files yourself.
- Downloading: Use the command
ollama pull modelname(e.g.,ollama pull llama3:8b). - Viewing: To see which models you have locally and how much space they take up, type
ollama list. - Deleting: Free up space using
ollama rm modelname.
The downside of Ollama is that the actual files are hidden deep within your system (on Windows often in C:\Users\Name\.ollama\models, on macOS in ~/.ollama/models). These are stored as blobs, meaning you cannot simply drag individual .gguf files to another folder.
Option B: Management via Graphical Interfaces (LM Studio / GPT4All)
Interfaces like LM Studio offer a search function inside the program that is directly connected to Hugging Face. Once you download a model through the interface, it is neatly organized into folders based on the publisher and the model name.
By default, these are stored in a cache folder on your C drive or primary partition (e.g., ~/.cache/lm-studio/models). Within the application settings, you can easily change this download location. This is crucial for the next step.
Tip: Move Your Models to a Secondary Drive
A common problem is that the C drive fills up unnoticed, causing your operating system to crash. Do you have a D drive, a fast external SSD, or even a NAS with a fast network connection? Create a central folder there, for example D:\AI-Models. You can tell your software (such as LM Studio) to always look in this folder. For Ollama, you can adjust the environment variable OLLAMA_MODELS to point to this new drive. Note: a slow HDD (hard drive with spinning disks) will significantly slow down the loading of the model.
Keeping Storage Clean: Cleaning, Updating, and Caching
AI models do not receive automatic updates. If Meta releases "Llama-3.1" today, your old "Llama-3.0" will simply remain on your drive until you take action yourself. Here are the best practices for clean storage:
1. Avoid the 'Download Addiction'
It is tempting to download every new model that appears on the leaderboards. Before you know it, you have 300 GB of data, 90% of which you will never touch again. Be ruthless: test a new model for a few days. If it is not noticeably better than your current favorite for specific tasks (such as programming, or generating text for an advanced system, see for example our guide on RAG for beginners), delete it immediately.
2. Cleaning Up the Hugging Face Cache
If you manually download models via the terminal or Python scripts using the transformers or huggingface_hub libraries, massive files are silently stored in the hidden folder ~/.cache/huggingface/hub. Even if you delete a project, these cache files often remain.
You can easily inspect and clean up this cache via the terminal. Open your command line and type:
huggingface-cli delete-cacheThis brings up an interactive menu where you can see exactly how much storage is being used per folder, including a clear list of old models, after which you can use the arrow keys to select files for permanent deletion. This is an indispensable tool for preventing disk space issues.
3. A Single 'Source of Truth'
Try to prevent different applications from downloading their own models. It is a waste of disk space if LM Studio, Ollama, and a local Python application all download their own copy of Llama-3 (about 5 GB). Centralize your storage. As you become more advanced, you can create symlinks (symbolic links) from one folder to another via your terminal, or serve a model locally with a single instance (such as LM Studio's built-in local server) that all your other applications communicate with via an API.
Conclusion
Downloading and managing local LLM models requires some initial attention to storage structure and file formats. By default, choosing GGUF files in the Q4_K_M quantization, keeping an eye on reputable uploaders on Hugging Face, and rigorously deleting unused models will keep your local AI setup lightning-fast and your disk storage healthy.