Installing Tabby as a local code assistant with Docker

Within the route of self-hosting language models, this guide sits in the phase apply and integrate. Once the underlying server and runtime are set up, we now focus specifically on automating software development. Anyone already familiar with the basics of containerization can first consult the guide on running a local language model in Docker for general background on how container isolation is fundamentally built up.

Tabby is an open-source, self-hostable AI coding assistant that acts as a direct alternative to proprietary cloud services such as GitHub Copilot. Unlike generic chat interfaces, Tabby revolves around two specific tasks: real-time inline code completion (Fill-in-the-Middle) and context-aware interaction through a chat panel in your editor. By deploying Tabby you retain full control over your intellectual property and codebase, without compromising on latency while typing.

Hardware requirements and VRAM calculation for code generation

Code generation places fundamentally different demands on hardware than an average chat LLM. For inline autocomplete, latency is the single most important factor: a suggestion has to appear within 100 to 250 milliseconds so as not to interrupt a developer's typing flow. Anyone who wants to understand which graphics card or memory configuration fits can review the system requirements in the guide on hardware for local LLM setups to make a well-founded choice between dedicated VRAM and system bandwidth.

With Tabby we ideally split the workload across two separate models: a compact FIM model (Fill-in-the-Middle) such as StarCoder2-3B or DeepSeek-Coder-1.3B for real-time completions, and a larger chat model such as Qwen2.5-Coder-7B for complex refactoring or explanation. To determine whether both models fit in video memory simultaneously, the table below offers an indicative overview of the estimated VRAM footprint at various precision levels.

For a detailed explanation of how compression techniques affect the balance between precision and model size, the documentation points to the article in which quantization and bit reduction are set out clearly. In practice the guideline is that a dedicated graphics card with at least 8 GB to 12 GB of VRAM is advisable to serve inline autocomplete smoothly.

Tabby architecture: FIM versus Chat

Tabby's architecture rests on a modular design optimized specifically for programming languages. Where regular language models predict only what follows after a given prompt, software development requires context on both sides of the cursor. This mechanism is known as Fill-in-the-Middle (FIM).

The FIM mechanism splits the open source file into a prefix (everything before the cursor) and a suffix (everything after the cursor). The model learns to fill in the missing code in between. Tabby manages these context windows itself and combines this with Git repository indexing. That lets the system pass relevant definitions and functions from other files in your project to the prompt as extra context, which raises the accuracy of the proposed code considerably.

Alongside the generation engine, Tabby contains a built-in search index based on Tree-sitter. This parser converts source files directly into an abstract syntax tree (AST). When the developer calls a function defined elsewhere in the project, the indexer recognizes the relation and automatically injects the function signature into the context window of the FIM model. As a result, generated parameters line up seamlessly with existing interfaces within the repository.

Docker Compose configuration with GPU passthrough

To roll out Tabby reproducibly and in isolation, we use Docker Compose in combination with the NVIDIA Container Toolkit. Make sure the official NVIDIA drivers and the container toolkit are installed and configured on the host system beforehand.

Below is a complete docker-compose.yml in which Tabby is configured with CUDA acceleration and persistent storage for downloaded models and repository indices:

services:
  tabby:
    image: registry.tabbyml.com/tabbyml/tabby:0.26.0
    container_name: tabby-server
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - ./data:/data
    environment:
      - TABBY_LOG_LEVEL=info
    command:
      - serve
      - --model
      - StarCoder2-3B
      - --chat-model
      - Qwen2.5-Coder-7B
      - --device
      - cuda
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Then bring the stack up from the terminal:

docker compose up -d
docker compose logs -f tabby

During the first start the container automatically downloads the requested weights from the model registry to the mounted volume ./data. Once initialization is complete, the web interface listens on port 8080 of the host system.

Initialization and setting up the administrator account

After starting the container, navigate in your browser to http://localhost:8080 (or the IP address of your local server). The first step is creating the primary administrator account.

In the admin panel you configure the following essential parameters:

• Model management: Under the tab Models check that the FIM and chat models have loaded correctly and that the CUDA backend is active.
• Access management: Generate a personal API key (User Token). This key is required to authenticate your development environment securely against the server.
• Context indexing: Optionally connect a Git repository over HTTPS or SSH so that Tabby can build local syntax trees for better suggestions.

Connecting development environments: VS Code and JetBrains

Once the server is running and you have a token, you can set up the client extensions. For Microsoft Visual Studio Code this goes through the official Tabby extension from the marketplace. Anyone who would rather explore alternative extensions can also work through the guide on connecting a local LLM to VS Code for broader integration options.

In VS Code, open the settings (settings.json) and add the server configuration:

{
  "tabby.server.endpoint": "http://127.0.0.1:8080",
  "tabby.server.token": "jouw_gegenereerde_auth_token_hier",
  "tabby.inlineCompletion.triggerMode": "automatic",
  "tabby.inlineCompletion.anonymousUsageTracking": false
}

Within JetBrains IDEs (such as IntelliJ IDEA, PyCharm, or WebStorm) you install the plugin Tabby through the Plugins menu. Then go to Settings > Tools > Tabby, enter the server endpoint, and paste the authentication token. After saving, the status bar in the bottom right immediately shows a status icon once the connection is active.

Comparison: Tabby versus Continue.dev

Developers who want to enrich their workflow with local AI regularly hesitate between Tabby and Continue.dev. Although both solutions run locally, they follow fundamentally different design choices. Anyone who wants to study how that other popular tool works will find all the details in the step-by-step guide on setting up Continue.dev with local models in VS Code.

Measurement method: validating latency and generation speed

The effectiveness of an inline code assistant depends almost entirely on response time. To determine whether your setup meets the practical norms, we measure three crucial parameters: Time To First Token (TTFT), tokens per second during generation, and the acceptance ratio of suggestions.

A reliable measurement method is run directly against Tabby's HTTP endpoints using curl and a timing measurement. This eliminates any network overhead from the IDE extension:

curl -w "\nTijd tot reactie: %{time_starttransfer}s\nTotale tijd: %{time_total}s\n" \
  -X POST http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer jouw_gegenereerde_auth_token_hier" \
  -d '{
    "language": "python",
    "segments": {
      "prefix": "def bereken_gemiddelde(waarden: list[float]) -> float:\n    \"\"\"Bereken het gemiddelde van een lijst getallen.\"\"\"\n    ",
      "suffix": "\n    return resultaat"
    }
  }'

Interpreting the measured values follows strict guideline figures:

• Under 150 ms (guideline: excellent): The code appears smoothly while typing; the programmer experiences no disruptive stutter.
• 150 to 300 ms (guideline: acceptable): Noticeable when typing very fast, but functionally workable for routine syntax and boilerplate.
• Above 400 ms (guideline: slowing): The suggestion arrives only after the developer has already typed on manually, which leads to visual noise and distraction.

Security, isolation, and GDPR aspects

The main argument for a local code assistant is data sovereignty. With cloud-based alternatives, program code, internal function names, API structures, and sometimes even hardcoded configurations are sent to external processors. By running Tabby inside a local network, all source code stays within your own infrastructure.

In the context of data minimization and compliance, this approach aligns neatly with the guidelines for privacy-friendly AI use in the workplace. To strengthen the application sandbox further, applying network isolation is advisable. The article on sandboxing LLM tools through Docker isolation explains how containers can be functionally shielded from the public internet by means of strict Docker network rules.

To prevent the Docker container from initiating outbound traffic after the models have been downloaded, you can attach the container to an internal bridge network without a default gateway. That makes it technically impossible for source code fragments to leak outward over a network connection.

Measuring performance and power consumption

A continuously running model server consumes power permanently, even when idle. A modern mid-range graphics card draws an estimated 10 to 20 watts at idle. As soon as a developer is actively typing code and Tabby handles dozens of FIM requests per minute, power briefly peaks at 150 to 250 watts.

To gain insight into the operational costs of such a machine, you can work through the calculation using the guide on power consumption of local AI systems, in which the difference between peak load and continuous consumption is quantified. The quality of the syntax produced also calls for systematic checking; consult the reference framework for evaluating generated code to spot regressions in model quality in good time.

Practical examples and language-specific scenarios

To clarify how the FIM model behaves in practice, we look at how Tabby handles context-sensitive completions in different programming languages.

TypeScript: interface matching and object completion

Suppose that in a file types.ts the following data structure is defined:

export interface GebruikerProfiel {
  id: string;
  volledigeNaam: string;
  email: string;
  actief: boolean;
  rollen: string[];
}

When the developer, in another file mapper.ts begins writing a transformation function, the editor places the cursor directly after the assignment:

export function formatteerGebruiker(data: GebruikerProfiel) {
  return {
    label: /* CURSOR HIER */
  };
}

Thanks to the repository index, Tabby reads in the definition of GebruikerProfiel . The FIM model completes the missing expression straight away to:

    label: `${data.volledigeNaam} (${data.email})`,
    isAdmin: data.rollen.includes("admin")

Python: error handling and type hints

When writing asynchronous API calls in Python, Tabby completes missing exceptions based on common library conventions:

async def fetch_json(url: str) -> dict:
    async with httpx.AsyncClient() as client:
        try:
            response = await client.get(url, timeout=5.0)
            response.raise_for_status()
            return response.json()
        except httpx.HTTPStatusError as exc:
            # Tabby FIM vult aan:
            logger.error(f"Fout bij ophalen {url}: {exc.response.status_code}")
            raise

Limitations and practical pitfalls

Although Tabby offers an effective solution for local autonomy, the platform has a few clear limitations:

• Limited context compared with frontier models: A local 3B or 7B model can oversee fewer complex abstractions across tens of thousands of lines of code at once than very large commercial cloud models.
• Cold starts and VRAM management: If the chat model and the FIM model do not fit in memory simultaneously, switching between models leads to waiting time.
• Support for niche languages: FIM performance is optimal in TypeScript, Python, Rust, Go, and C++, but drops off with less common programming languages or dated frameworks.
• CPU inference is unsuitable for FIM: Although Tabby can run on a pure CPU through OpenVINO or llama.cpp, latency per FIM suggestion rises considerably. In practice that makes the inline autocomplete experience less fluid.

Maintenance, updates, and backup of repository indices

Because Tabby functions as a permanent background service, the database of repository indices needs periodic maintenance. All configurations, tokens, and generated syntax trees are stored in the directory mounted to the container (in the Compose example ./data).

When updating the model or the container image, the following standard procedure suffices:

# 1. Stop de actieve server
docker compose down

# 2. Haal de nieuwste image op
docker compose pull

# 3. Start de container opnieuw
docker compose up -d

When large changes are made to the source files on the host, the Tree-sitter index can become outdated. Through the web interface at http://localhost:8080/settings/repositories a manual re-indexing can be started at any moment to restore consistency between project files and FIM suggestions.

Conclusion

By installing Tabby through Docker you create a self-contained, privacy-friendly, and fast coding assistant on your own hardware. By tuning FIM models specifically for low latency and deploying chat models for extensive tasks, you get a robust development environment without sensitive code ever leaving your network.