Connecting a Local LLM to VS Code: The Complete Guide for Developers

Artificial intelligence has permanently changed the way we develop software. Tools like GitHub Copilot and ChatGPT have become indispensable for many programmers. But what if you work with business-critical code, have signed strict NDAs (non-disclosure agreements), or simply refuse to pay monthly subscription fees for a cloud-based AI? The solution is now within reach: you can run a Large Language Model (LLM) locally on your own machine and integrate it directly into Visual Studio Code (VS Code).

In this comprehensive guide, we will discuss exactly how to connect a local LLM to VS Code. We will cover the necessary software, the configuration of an OpenAI-compatible server, the best extensions, and we will dive deep into the hardware requirements, specifically for developers working on a Mac with Apple Silicon.

Why Use a Local LLM as a Code Assistant?

Before we dive into the technical configuration, it is important to weigh the pros and cons of a local setup versus a cloud solution.

Absolute privacy and security: This is the main reason for many developers and companies. When you use a cloud-based assistant, your source code (or at least parts of it) is sent over the internet to third-party servers. With a local model, not a single line of code leaves your physical machine. You are operating in a completely privacy-friendly way and eliminating the risk of data leaks via external APIs.
No subscription fees: After the one-time investment in suitable hardware, the operational costs (aside from power consumption) are zero. You are not dependent on price increases or API limits.
Always available: No internet connection? No problem. Your code assistant works offline on the train, on a plane, or during a network outage.

However, there are trade-offs. Depending on your hardware, local models are often slower at generating output than enterprise cloud servers. In addition, they require significant computing power (GPU or powerful CPU with sufficient memory), and you sometimes miss out on the massive context windows offered by models like Claude 3.5 Sonnet or GPT-4o.

The Foundation: Setting Up a Local OpenAI-Compatible Endpoint

To allow VS Code to communicate with a local model, you need a 'bridge'. Most AI extensions for VS Code were originally built to talk to the OpenAI API (via HTTP POST requests to /v1/chat/completions). Fortunately, the developers of local AI tools have recognized this.

To run a local model that speaks this language, you typically use a tool like Ollama or LM Studio. These tools load the model into system memory and start a local web server in the background that uses the exact same API format as OpenAI, but on localhost.

Using Ollama

Ollama is currently the industry standard for developers. It is a lightweight command-line interface that allows you to download and run models with a single command. Read our specific guide on installing Ollama on macOS here.

Once Ollama is installed, start the server (and the model) via your terminal:

ollama run deepseek-coder-v2:lite

When this command runs, Ollama listens on port 11434 by default. Your "OpenAI" API endpoint is now available at: http://localhost:11434/v1.

Which VS Code Extensions Support Local Models?

Not every AI extension offers the ability to customize the endpoint. Here are the three best options for local integration today:

Continue.dev (Highly recommended): This is currently the most powerful, open-source AI assistant for VS Code. Continue stands out with a beautiful interface, excellent support for local context (RAG), and the ability to use both local and cloud models interchangeably.
CodeGPT: A very popular extension that was initially built for OpenAI, but now offers extensive support for local providers like Ollama and LM Studio via the settings menu.
Twinny: An emerging extension that specifically focuses on running AI assistants locally without bloatware. Ideal for older or less powerful systems.

Step-by-Step Configuration of the 'Continue' Extension

For this guide, we focus on Continue.dev, as this extension offers by far the best support for code autocomplete as well as a chat interface, similar to GitHub Copilot.

Step 1: Installation

Open VS Code, go to the Extensions marketplace (Ctrl+Shift+X or Cmd+Shift+X), search for Continue, and click install. After installation, the Continue logo will appear in your left sidebar.

Step 2: Modifying config.json

Continue works with a central configuration file. Click the gear icon in the Continue sidebar to open config.json. To add your locally running Ollama model, locate the models array and add the following configuration:

{
  "models": [
    {
      "title": "Local DeepSeek Coder",
      "provider": "ollama",
      "model": "deepseek-coder-v2:lite",
      "apiBase": "http://localhost:11434"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Starcoder 2 3B",
    "provider": "ollama",
    "model": "starcoder2:3b",
    "apiBase": "http://localhost:11434"
  }
}

Note that in this example, we configure two models: a capable model (like DeepSeek or Llama 3) for the chat interface, and an extremely small, fast model (like Starcoder2 3B or Qwen2.5-Coder 1.5B) specifically for tabAutocompleteModel. Autocomplete needs to respond within milliseconds, whereas you can easily wait a few seconds for a chat response.

Models and Hardware: What Works Well on a Mac with Apple Silicon?

If you have a Mac with an M-series chip (M1, M2, M3, or M4), you are likely in possession of the best consumer hardware for local AI. This is due to the 'Unified Memory Architecture' (UMA). Unlike traditional PCs where the processor (CPU) and the graphics card (GPU) have their own separate memory, an Apple Silicon chip shares one large pool of memory.

This means that a MacBook Pro with 64GB of RAM effectively functions as a graphics card with (almost) 64GB of VRAM. This allows you to run models that would require a graphics card costing thousands of euros on a Windows PC. But which model fits your hardware? Also read our general article on choosing a local model.

Mac with 8GB or 16GB RAM: Limit yourself to models of 7B to 8B parameters. Examples include llama3:8b, qwen2.5-coder:7b, or mistral:7b. These models occupy about 4 to 6 GB of memory (depending on the quantization), leaving enough room for VS Code, your browser, and your operating system.
Mac with 32GB RAM: You can comfortably work with models around 32B parameters. Think of qwen2.5-coder:32b or command-r. These models offer logic and code generation comparable to GPT-3.5 and early versions of GPT-4.
Mac Studio / Max with 64GB+ RAM: The domain of the heavyweights. Models like llama3.1:70b or wizardlm2:8x22b (quantized) can run on this. The response speed will be lower, but the code quality is impressive. You can find more about this in our guide on hardware for local LLMs.

Context Windows and Project Files (RAG in VS Code)

One of the biggest challenges with local AI is the context window. A model needs to understand your existing codebase to make useful additions. Cloud models nowadays have context windows of 200,000 tokens, allowing you to send an entire repository at once. Locally, due to hardware limitations, you are often limited to 8,000 or at most 32,000 tokens.

So you cannot simply send your entire project folder to your local model. This is solved by applying RAG (Retrieval-Augmented Generation) locally. For more theory on this, you can look at RAG for beginners on our learning platform.

The Continue extension solves this brilliantly with @-mentions and local indexing. You can include specific files, folders, or even entire documentation sets in the chat:

Type @file to search directly for a file and add its content as context.
Type @codebase to run a search across your entire workspace. Continue uses locally computed embeddings (with a small embedding model like nomic-embed-text) or a BM25 search algorithm to find the most relevant code snippets from your project and send only those snippets to the LLM.
Type @folder to include the files in a specific subdirectory.

This ensures that you use your tokens efficiently, preventing the local model from becoming overloaded (or crashing with an "Out of Memory" error) and keeping response times acceptable.

Common Pitfalls and Solutions

During the setup of this configuration, many developers run into the same obstacles. Here is a quick troubleshooter:

1. "Connection Refused" Error Messages

If VS Code indicates that it cannot connect to the API, first check if your local AI server is actually running (e.g., is the LM Studio server on, or is ollama serve running in the background?). Then check the apiBase in your config.json. Sometimes http://127.0.0.1:11434 works better than http://localhost:11434 depending on your network settings and DNS resolution.

2. Extremely Slow Response Times or Lagging Text

This is almost always a hardware bottleneck. If the model does not fit entirely within the video memory (VRAM) or Unified Memory, your system will start "swapping" (using virtual memory on your SSD). This slows down the inference speed from, for example, 30 tokens per second to less than 1 token per second. The solution: choose a smaller model or use stronger quantization (e.g., a 4-bit model instead of an 8-bit model).

3. The Model Hallucinates APIs or Functions

If you ask for a specific script and the AI invents libraries that do not exist, this could be due to a Temperature setting that is too high, or simply using the wrong model type. Make sure you use so-called "coder" or "instruct" models (such as DeepSeek-Coder or Qwen-Coder). General chat models (like a base Llama model) are sometimes too creative. In your VS Code extension settings, adjust the temperature to 0.1 or 0.2 for more deterministic and predictable code.

            Pro Tip: Do not underestimate the importance of the system prompt. With extensions like Continue, you can customize the default system prompt. Add rules like: "You are an expert developer. Write only working code. Do not provide extensive explanations unless asked. Format the code correctly." This saves a huge amount of tokens locally that would otherwise be spent on useless introductions or disclaimers.
        

With the right hardware, a tool like Ollama, a powerful extension like Continue, and a well-chosen model, you have an impressive AI assistant directly in VS Code. One that understands your code, but keeps your trade secrets within the walls of your own study.