Running local vision models for image analysis

Within the roadmap for local AI — from selecting basic components to advanced automation — visual processing falls under the pillar applications and modalities. Once text-based models run smoothly, many organizations immediately feel the need to digitize their document flows. Important information is rarely held in plain text; it sits in scanned PDFs, handwritten notes, charts, diagrams, screenshots, and invoices.

Self-hosting visual models raises specific infrastructure questions around memory allocation, image resolutions, and context management. If you still need to set up the underlying workstation, the guide on hardware for local LLMs offers a thorough comparison of graphics cards, memory bandwidth, and system configurations.

Vision-language models (VLMs) bridge the gap between computer vision and natural language processing. Rather than simply assigning a label to an image, they combine a visual encoder with a decoding autoregressive language model. That lets them reason directly about visual relationships, parse complex document structures, and answer questions about images. A conceptual exploration of this shift can be found in the analysis of why multimodal models are the next step.

The architecture of a local vision model

A local vision-language model is built from three fundamental components: the visual encoder, the projection layer, and the autoregressive language model. The visual encoder (usually a Vision Transformer such as ViT or a SigLIP network) splits an incoming image into a two-dimensional grid of small image fragments known as patches (often 14x14 or 16x16 pixels). Each of these patches is converted into a continuous numerical vector through linear projections and attention mechanisms.

The projection layer (such as a multilayer perceptron or a cross-attention bridge) then translates these visual vectors into the language model's embedding space. The language model treats these transformed image vectors exactly the same way it treats regular text tokens. Feeding in a 1024x1024-pixel image produces hundreds to thousands of visual tokens. Those tokens immediately fill a substantial portion of the context window before the user has even asked a question.

When quantization techniques are applied to VLMs, the weights of the language model and the projection layers are compressed to 4-bit or 8-bit precision, for example. You can read more about the technical workings of weight compression and bit reduction in the overview article on quantization and GGUF file formats. In most local architectures the visual encoder itself keeps running at 16-bit precision (FP16 or BF16), because compressing the encoder leads directly to a loss of fine detail in small type and complex line patterns.

Hardware requirements and VRAM calculation for image analysis

Calculating the video memory a vision model needs differs fundamentally from a traditional text model. Beyond the static storage of the model weights in VRAM, you have to account for three dynamic factors: the base memory of the vision encoder, the memory for the patches, and the KV cache for the resulting sequence of visual tokens.

Modern models use dynamic resolution techniques. Instead of rigidly shrinking a document to a fixed size (which renders small type unreadable), architectures such as Qwen2-VL and MiniCPM-V slice large images into multiple sub-segments, or tiles. An A4 document scanned at 300 DPI quickly yields four to nine separate tiles, plus one downscaled overview image for global context.

As a result, a single page can claim between 2,000 and 6,000 context tokens. Anyone analyzing several pages at once will see the KV cache grow exponentially. A generous margin on top of the model's base weight is therefore essential to avoid runtime out-of-memory (OOM) errors during processing.

Installation and configuration with Ollama

For individual workstations and quick local integrations, Ollama offers the most accessible route to working with vision models. The software bundles the language model weights, the vision encoder, and the accompanying projection layer into a single manifest file. Downloading and starting up therefore runs through familiar commands.

# Download en start het 8B MiniCPM-V model
ollama run minicpm-v

# Of kies voor Llama 3.2 Vision met 11 miljard parameters
ollama run llama3.2-vision:11b

For direct interaction on the command line, a local image path can simply be appended to the input command. The runtime reads the file, processes the binary data, and passes the extracted vectors straight into the inference chain:

ollama run minicpm-v "Wat zijn de belangrijkste posten op deze factuur? /pad/naar/factuur.png"

In development environments and automated workflows, communication runs through the built-in REST API. The image is converted to a base64 string and sent inside the JSON structure to the chat endpoint:

curl http://localhost:11434/api/chat -d '{
  "model": "minicpm-v",
  "messages": [
    {
      "role": "user",
      "content": "Beschrijf de meetwaarden in dit screenshot:",
      "images": ["iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY44YAAAAASUVORK5CYII="]
    }
  ],
  "stream": false
}'

This simple interface makes it possible to integrate vision functionality quickly into local scripts, internal dashboards, and file monitors without complicated dependencies.

Production setups with vLLM

When large batches of documents have to be processed, or when several users submit requests at the same time, a sequential engine reaches its limits. In production environments a specialized inference server such as vLLM offers substantial advantages through PagedAttention and continuous dynamic batching.

# Start een OpenAI-compatibele vLLM server met een vision-model
vllm serve Qwen/Qwen2-VL-7B-Instruct \
  --trust-remote-code \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --limit-mm-per-prompt image=4

The configuration option --limit-mm-per-prompt image=4 reserves memory slots so that a single prompt may contain at most four images at once. This is particularly valuable for tasks that compare before-and-after situations, or where a multi-page contract has to be assessed in a single context.

Because the server mimics an OpenAI-compatible API, existing applications, libraries, and orchestration layers can switch to the local server by changing only the base URL and the model key. That makes a seamless transition from commercial cloud APIs to internal hosting possible.

Extracting structured data with JSON schemas

The most important business application of local vision models is converting unstructured visual documents — such as bills of lading, identity documents, inspection reports, and receipts — into validated JSON objects. Without guidance, vision models often produce long-winded textual explanations that are hard to parse reliably in backend systems.

By enforcing formal grammars and JSON schemas, model output can be tightly constrained. Technical details on configuring restrictive output filters are covered at length in the guide on structured JSON outputs with local LLMs.

The Python example below demonstrates how a document image is analyzed locally and validated directly against a strict data model using Pydantic:

import base64
import json
import urllib.request
from pydantic import BaseModel, Field

class FactuurRegel(BaseModel):
    omschrijving: str
    aantal: int
    bedrag_excl_btw: float

class FactuurData(BaseModel):
    leverancier: str
    factuurnummer: str
    regels: list[FactuurRegel]
    totaalbedrag: float

def analyseer_document(afbeelding_pad: str) -> FactuurData:
    with open(afbeelding_pad, "rb") as f:
        afbeelding_base64 = base64.b64encode(f.read()).decode("utf-8")

    payload = {
        "model": "minicpm-v",
        "messages": [
            {
                "role": "user",
                "content": "Extraheer alle gegevens van deze factuur exact volgens het schema.",
                "images": [afbeelding_base64]
            }
        ],
        "format": FactuurData.model_json_schema(),
        "stream": False
    }

    req = urllib.request.Request(
        "http://localhost:11434/api/chat",
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"}
    )

    with urllib.request.urlopen(req) as resp:
        resultaat = json.loads(resp.read().decode("utf-8"))
        gevalideerd = FactuurData.model_validate_json(resultaat["message"]["content"])
        return gevalideerd

Performance on Dutch-language documents

Many common open-weight vision models are trained primarily on large English and Asian datasets. Processing specifically Dutch documents therefore requires extra attention. Think of legal documents with typically Dutch jargon, notarial deeds, or documents with compound words and abbreviations such as 't.a.v.', 'i.o.m.', 'KVK', and 'BSN'.

Older multimodal architectures tend to automatically 'correct' vague or unclear Dutch text into phonetically similar English words. Modern multilingual models such as Qwen2-VL and MiniCPM-V 2.6 perform considerably more consistently here. They preserve the exact literal transcription and also recognize Dutch date formats (such as '14 augustus 2026') and number notations, where the roles of commas and periods for decimals and thousands are reversed.

To further improve accuracy on complex Dutch text, it helps to design the system prompt and the instructions carefully. See the article on getting AI to perform better in Dutch for practical strategies around prompting, terminology, and stylistic consistency.

Privacy, compliance, and the GDPR in visual processing

Visual data carries considerably greater compliance and privacy risks than regular text files. A scan or photo often unintentionally contains sensitive side information, such as passport photos (biometric data under GDPR Article 9), handwritten signatures, national identification numbers, barcodes, or personal data captured incidentally in the background.

When such images are sent to external cloud services, they cross network boundaries and are temporarily stored on third-party systems. That complicates compliance with privacy legislation and requires extensive data processing agreements. Running the models locally within your own infrastructure keeps the entire data flow inside your own network.

To make sure the processing meets every legal standard, it is advisable to work through the steps in the GDPR privacy checklist for organizations. It describes how temporary image caches in RAM should be cleared, how log files are anonymized, and how retention periods are strictly enforced.

Limitations, failure modes, and optimizations

Despite the impressive capabilities of modern vision models, developers have to account for a number of specific failure mechanisms when designing automated processing pipelines:

Hallucinations at small font sizes: Once characters in an image are less than 12 to 14 pixels tall, the vision encoder can no longer distinguish the visual features unambiguously. The model then starts guessing statistically, which is dangerous for digit sequences such as account numbers or amounts. Optimization: Crop documents into logical zones beforehand, or raise the scan resolution to 300 DPI before submitting the file.

Tilted or rotated images: Many models struggle with text that is rotated 90 or 180 degrees, or with documents photographed at an angle. Optimization: Add a lightweight preprocessing step using a computer vision library (such as OpenCV) to correct orientation and rotation automatically (deskewing) based on document edges or EXIF metadata.

Token saturation from excessive resolutions: Passing raw 4K photos to a model unfiltered leads to enormous context sizes and slow response times, without adding any informational value. Optimization: Scale images down before processing to the model's effective resolution limit (1024x1024 or 1344x1344 pixels, for example) to keep processing speed high and VRAM use predictable.

Implementation strategy

Self-hosting vision models gives organizations and developers a powerful instrument for automating document processes, quality checks, and data extraction entirely in-house. The step from text to image calls for a considered trade-off between resolution, context tokens, and available video memory, but thanks to modern quantization techniques it no longer requires unaffordable data center infrastructure.

The soundest approach is a phased rollout: start with a compact model inside Ollama to evaluate recognition quality on representative documents. Once the extraction results meet your accuracy requirements, the pipeline can be extended with formal JSON schemas for reliable backend integration. For scalable bulk processing, moving to vLLM is then the logical final piece of a performant, privacy-friendly architecture.