Installing and Configuring Jan AI Locally on Windows

Along the path of choosing → installing → using → connecting → managing this article picks up directly at the installation and configuration phase on Windows systems. Before we set up the interface, it's advisable to first check what hardware local LLMs require so that your system memory and graphics card align well with the desired model size. Additionally, the general Windows installation guide offers a broader overview of architectures and alternative runtimes on this operating system.

Jan AI is an open-source desktop application that mimics the functionality of cloud-based AI chatbots but performs all computation and storage locally. Anyone looking for an overview of similar programs will find in the overview of local LLM tools a direct comparison between Jan, LM Studio, and desktop clients built around Ollama.

1. Architecture and Operation of Jan AI on Windows

Jan AI combines a modern desktop interface with a modular C++ backend. Under the hood, the application uses an optimized inference engine built on top of llama.cpp. This allows the program to read GGUF files directly and route computations to CPU instruction sets or graphics APIs such as Nvidia CUDA and Khronos Vulkan.

When the local API server is enabled, Jan starts a background process that acts as an OpenAI-compatible proxy. All conversation history, model configurations, and prompts are structurally stored as plain JSON files and Markdown in the Windows user folder. No heavy external database engine is required, so backing up or moving sessions comes down to simple file management.

2. Setting Up System Requirements and Hardware Acceleration

The response speed of local models depends directly on how the graphics adapter is addressed. The software supports two important acceleration paths on Windows:

For systems with an Nvidia GPU, the CUDA backend is the most efficient choice. For systems with AMD Radeon cards or Intel graphics cards, the application includes a Vulkan runtime. Although Vulkan carries a fraction more CPU overhead when processing long prompts, it enables hardware acceleration across a wide range of graphics hardware on Windows.

3. Installation Steps via Installer and Windows Package Manager

The application can be downloaded manually or installed automatically using the Windows Package Manager (winget). Using winget enables consistent management and updates via PowerShell.

# Installatie via Windows Package Manager
winget install Jan.Jan

# Controleren of het pakket geregistreerd staat
winget list Jan.Jan

For a manual installation, run the installer and select the desired installation path. If Windows Defender SmartScreen shows a warning for a newly released installer file, verify the file's SHA256 hash against the source's official checksum before continuing the installation.

4. Downloading Models, Choosing Quantization, and Importing

In the built-in model library, models can be selected directly in GGUF format. To understand how compression allows models to fit within the available video memory, the overview on quantization explains how 4-bit and 8-bit formats reduce memory usage while preserving text quality.

For common consumer hardware, models from well-known open families such as Llama, Mistral, and Qwen are suitable for a wide range of tasks, from general text analysis to programming. Already downloaded GGUF files can also be imported manually via the local folder structure:

# Standaardlocatie voor modellen in de gebruikersmap
%USERPROFILE%\jan\models\

# Voorbeeld van de mappenstructuur per model:
# C:\Users\<gebruiker>\jan\models\aangepast-model\
#   ├── model.gguf
#   └── model.json

In the configuration file model.json parameters such as the context size and the structure of the instruction template are defined, so that the backend formats the input correctly.

5. Runtime Configuration: Optimizing GPU Layers and Context Length

To achieve maximum computation speed, as many model layers as possible must be moved to the GPU's video memory (GPU offloading). In the settings, you can switch between CPU, Vulkan, and CUDA.

With GPU acceleration active, the number of layers sent to the graphics card can be precisely set via the GPU layers slider. If the model fits entirely in video memory, all layers are assigned to the GPU. If memory is insufficient, part of the layers can be moved to system memory via the CPU.

The context window determines how many tokens of prior context and document text can be processed at once. Consult the guide to optimizing context length to prevent an overly large context buffer setting from leading to memory errors. Within Jan, the configuration can be adjusted per model:

{
  "sources": [
    {
      "filename": "model.gguf"
    }
  ],
  "id": "lokaal-model",
  "object": "model",
  "name": "Lokaal Instructiemodel",
  "format": "gguf",
  "settings": {
    "ctx_len": 8192,
    "ngl": 33,
    "temperature": 0.7,
    "top_p": 0.95,
    "stream": true
  },
  "parameters": {
    "max_tokens": 4096
  }
}

6. Activating the Local API Server and Integrating with Windows Apps

The application includes a local HTTP server compatible with OpenAI endpoints. This allows Jan to function as a local engine for editors such as VS Code, note-taking applications, or automation scripts.

After enabling the server in the settings, it listens by default on port 1337 of localhost. Its operation can easily be checked via PowerShell:

# Lokale API testen via PowerShell
curl.exe -X POST http://127.0.0.1:1337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lokaal-model",
    "messages": [
      {"role": "system", "content": "Je bent een behulpzame assistent."},
      {"role": "user", "content": "Wat is het voordeel van een lokaal model?"}
    ],
    "temperature": 0.7
  }'

7. Privacy, Telemetry, and Data Security on Windows

The main motive for running language models on your own Windows PC is safeguarding data privacy. Anyone processing sensitive business information, source code, or personal data can, via the guide to privacy-friendly AI find out which measures help keep data flows entirely local.

During response generation, data does not leave the local machine. However, a desktop application may include options for anonymized crash reporting. To set up an isolated working environment, the following steps are recommended:

1. Open the privacy settings within the program and disable optional telemetry and automatic crash reporting.
2. In Windows Defender Firewall, make sure the local API port only listens on the loopback adapter (127.0.0.1) and does not accept incoming connections from external networks.
3. All conversation data and model settings remain stored in the local user folder and are not synchronized to cloud services.

8. Limitations and Performance Trade-offs in Practice

Although a graphical desktop interface significantly lowers the barrier to getting started locally, it also brings some functional trade-offs compared to purely command-line management via tools such as llama.cpp or Ollama:

A graphical user interface continuously consumes some extra memory for rendering windows and interactive elements. In addition, advanced parameters for sampling and caching are often more accessible in dedicated server environments than in a simplified consumer interface. For purely interactive chatting and basic local API integrations, however, the desktop setup offers the most ease of use.

9. Troubleshooting Common Windows Errors

Specific errors can occur while setting up or loading models. Below are the most common situations and recovery steps:

Model does not load or immediately crashes with a memory error

When a model crashes as soon as it is loaded, the number of assigned GPU layers is usually too high for the available video memory. In the model's settings, gradually reduce the number of GPU layers until the model fits stably in memory, or choose a more compact context length.

Missing C++ runtime libraries

The underlying C++ inference module requires the presence of the Microsoft Visual C++ Redistributable libraries. If error messages appear about missing DLL files, install the current runtime package:

# Visual C++ runtimes installeren via winget
winget install Microsoft.VCRedist.2015+.x64

No GPU activity during generation

If response time is very slow and the processor is heavily loaded while the graphics card remains idle, hardware acceleration is likely set to CPU mode or the correct driver support is missing. Check that the correct acceleration path (CUDA or Vulkan) is active in the runtime settings and restart the program.

Conclusion and Next Steps

Jan AI offers an accessible, privacy-focused way to run open language models locally on a Windows system. Thanks to flexible support for both CUDA and Vulkan, existing hardware can be optimally utilized for text processing and analysis. Once the basic installation is operational, the local REST API can easily be linked to external productivity applications and development tools within your own work environment.