Setting up AnythingLLM as a complete desktop workplace

Anyone who wants to use local language models to search their own documents quickly discovers that a bare chat interface like the Ollama CLI falls short. Manually applying chunking to PDF files, computing vectors, and injecting context calls for a streamlined RAG pipeline (Retrieval-Augmented Generation). In the learning path from choosing to managing, this article is in the phase of using and connecting. Before we look at the application setup, it's wise to check whether the computer has enough compute power; see the analysis on what hardware local LLMs require so you don't run into memory shortages.

AnythingLLM offers a complete desktop solution in which document parsing, vector storage, LLM orchestration, and a modern user interface come together in a single package. Where a separate backend configuration often requires multiple active processes, AnythingLLM's desktop app bundles a built-in vector store (LanceDB) and a document processor. In this guide, we walk through the full configuration of AnythingLLM as a daily working environment on Windows, macOS, and Linux, with an emphasis on local privacy, workspace management, and performance optimization.

What makes AnythingLLM suitable as a desktop environment?

Traditional interfaces focus primarily on sending and receiving text prompts. AnythingLLM is fundamentally built differently, around the concept of Workspaces (workspaces). Each workspace has its own collection of documents, its own system prompt, configurable temperature settings, and a separate vector index. This allows legal documents, programming manuals, and personal notes to remain strictly separated without contexts contaminating each other.

An important advantage is the modular design under the hood. The desktop version runs as an Electron shell with an integrated Node.js/Express backend and a local LanceDB vector store. Users can, however, choose to decouple specific components: an external Ollama instance for generation, a remote embedding model, or a central vector database. For anyone who wants to run a local model on an Apple system, the installation guide on installing Ollama on macOS offers a solid starting point for getting the underlying inference engine running.

In addition, AnythingLLM supports so-called agent skills and web scraping directly from the interface. The program can not only read local PDF, DOCX, and TXT files, but also fetch web pages, index GitHub repositories, and transcribe audio recordings, provided a speech model has been made available.

Installation and initial setup per operating system

The simplest way to get started is the official AnythingLLM desktop installer. It installs all the necessary supporting services (document engine, LanceDB, and the UI) without requiring Docker.

During the initial startup, the configuration wizard asks for three choices: the LLM provider, the embedding provider, and the vector database. For a fully isolated work environment, we select local components in all three steps:

• LLM Provider: Choose Ollama or LocalAIAs the base URL, enter http://127.0.0.1:11434 when Ollama is running on the same computer.
• Embedding Provider: Likewise, choose Ollama or the built-in Native Embedder.
• Vector Database: Select LanceDB (built in by default, no further configuration needed).

Connecting to local inference engines (Ollama and LM Studio)

To provide AnythingLLM with text generation, we connect the application to a local engine. Ollama is the most commonly used option for this because of its stable API. Make sure the desired model has already been pulled beforehand via the command line:

# Haal het LLM en het embeddingmodel binnen via de terminal
ollama pull llama3.1:8b-instruct-q4_K_M
ollama pull nomic-embed-text

Then open the settings in AnythingLLM (the gear icon in the bottom left) and navigate to AI Providers > LLM. Select Ollama in the dropdown menu. The application immediately checks the connection via http://127.0.0.1:11434 and shows a list of locally available models. Select llama3.1:8b-instruct-q4_K_Mhere. Set the context window (Token Context Window) to a value that fits the available memory, for example 8192 tokens.

If LM Studio is preferred, start the Local Server in it on port 1234 and in AnythingLLM choose the provider LM Studio with URL http://127.0.0.1:1234/v1. This protocol follows the OpenAI standard, allowing AnythingLLM to communicate seamlessly with virtually any engine that provides a compatible endpoint.

Setting up the vector engine: LanceDB versus external databases

The default vector store in AnythingLLM is LanceDB, an embedded vector engine that writes directly to local files on the hard drive. For an individual workplace with a few hundred documents, LanceDB performs excellently: no separate database process is needed, memory usage stays low, and making backups is a matter of copying a single folder.

When the document collection grows to tens of thousands of pages, or when multiple applications need to query the same vectors, an external database can help. The article on setting up a local vector database with Qdrant and Chroma describes in detail how such engines can run via Docker. In AnythingLLM, under AI Providers > Vector Database you can easily switch to an external Qdrant instance at http://127.0.0.1:6333.

Structuring document management and workspaces

Organizing files in AnythingLLM happens via Workspaces. Create a separate workspace for each well-defined project or knowledge area using the New Workspace button on the home screen. Give the workspace a clear name, such as Onderzoek-Energiecontracten or Documentatie-Softwareproject.

Within the workspace, click the upload icon to add documents. AnythingLLM supports, among others:

• Text and Formatting: PDF, DOCX, TXT, MD, CSV, JSON.
• Web Sources: Direct URLs (scraped locally and converted into clean text).
• Codebases: ZIP files with source code or linked Git repositories.

After documents are uploaded, they end up in the central Document Storage of the application. To actually make them searchable within the chosen workspace, the files need to be selected and linked using the Move to Workspace > Save and Embedbutton. At that point, AnythingLLM reads out the text, splits it into fragments (chunks), and the embedding model computes the corresponding vectors.

Selecting embedding models for Dutch-language documents

The quality of information retrieval stands or falls with the chosen embedding model. Many default English-language models perform poorly on compound Dutch words or specialized jargon. Anyone wanting to dig deeper into the specific performance differences can read the overview on which embedding model to choose for Dutch documents.

For general use, nomic-embed-text via Ollama provides a stable foundation with a context length of 8192 tokens. For strictly Dutch-language corporate documents and policy texts, multilingual models such as bge-m3 or paraphrase-multilingual-mpnet-base-v2 often provide a more accurate semantic match. In AnythingLLM, the active embedding model can be changed per workspace or globally under AI Providers > Embedding.

Fine-tuning RAG parameters: chunking, similarity, and context window

If a model cites irrelevant fragments or misses essential facts, the cause usually lies in poorly configured RAG parameters. Within AnythingLLM, these can be optimized per workspace via Workspace Settings > Vector Database Settings.

Chunk Size and Chunk Overlap

By default, AnythingLLM splits documents into blocks of about 1000 characters with an overlap of 20 characters. For structured reports with long paragraphs, a chunk size of 1500 characters with 150 characters of overlap works better, because sentences and reasoning are then less likely to be cut in half. For short question-and-answer documents or tables, a smaller chunk of 500 characters is recommended.

Similarity Threshold and Top-K

The Similarity Threshold determines how closely a document fragment must match the question to be sent along to the language model. A value between 0.60 and 0.75 (for cosine similarity) effectively filters out noise. The Max Context Snippets (Top-K) sets how many text fragments are injected at most. For a model with an 8k context window, 4 to 6 snippets is a safe setting that offers sufficient depth without overfilling the prompt.

// Richtlijnen voor RAG-instellingen in AnythingLLM:
{
  "chunk_size": 1200,
  "chunk_overlap": 120,
  "similarity_threshold": 0.65,
  "top_k_snippets": 5,
  "system_prompt_mode": "strict_context"
}

Chat versus Query Mode: when do you use which mode?

In the top right of the chat window of each workspace there's a toggle between Chat and Query mode. This is a fundamental distinction in how AnythingLLM builds prompts:

• Chat Mode (Standard Conversation): The model answers questions using the linked documents, but may draw on its own general training data if the answer isn't in the documents. The model also remembers earlier messages in the conversation.
• Query Mode (Strict Document Analysis): The model acts purely as a summarizer of the retrieved context. If the requested facts aren't literally present in the indexed documents, the model refuses to answer. This prevents hallucinations and is essential for legal and financial document verification.

Privacy, network traffic, and GDPR safeguards

A primary reason to run AnythingLLM locally is to guarantee data sovereignty. With a standard cloud solution, uploaded files are processed on external servers. By combining AnythingLLM with a local Ollama instance, all data processing stays on your own physical device.

To verify that no data is leaking out, we can analyze the process's network traffic. When starting, AnythingLLM checks for software updates via GitHub and optionally fetches telemetry pings. This telemetry can be fully disabled in the settings under Privacy & Data > Anonymous Telemetry. For a broad overview of organizational privacy measures, see the practical guide on using AI in a privacy-friendly way.

When loading confidential personal data or internal company reports, it's advisable to use the built-in web scraper with care: avoid fetching public URLs that contain trackers. Also make sure the AnythingLLM storage folder is included in your local encrypted backup protocol (such as FileVault on macOS or BitLocker on Windows).

For a broader perspective on deploying compact architectures that run directly on the workstation, the background article on small on-device models offers useful insights into how 3B and 8B models hold up against heavy cloud services.

Weak points and known limitations

Although AnythingLLM offers a very complete environment, there are clear technical limitations to keep in mind:

• Complex PDF Tables: The built-in document parser primarily reads PDFs as flat text streams. Complex tables with merged cells are sometimes read incorrectly, resulting in the loss of column relationships in the vector store.
• Memory Footprint During Large Bulk Ingestions: When indexing hundreds of documents simultaneously, the embedding process can cause peak loads on the CPU or GPU. It's wise to import large document collections in batches of 20 to 30 at a time.
• No Advanced Re-ranking: By default, the desktop version doesn't include a cross-encoder re-ranking step after the vector search, so sometimes less relevant chunks still end up in the context if the top-k is set too high.

Practical workflow: from document to reliable answer

A well-thought-out setup of AnythingLLM transforms a computer into a powerful, fully isolated research assistant. By organizing workspaces thematically, connecting a stable local model such as Llama 3.1 8B via Ollama, and carefully tuning the RAG parameters to the document structure, you get a working environment that operates fast, reliably, and 100% confidentially.