# natlibfi/annif architecture

> Architecture diagram of the GitHub repository natlibfi/annif, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/natlibfi/annif
- Repository: https://github.com/natlibfi/annif

## Overview

##### Project type and purpose
- Annif is a modular, full-stack toolkit for automated subject indexing.
- It provides:
- A command‐line interface (CLI built with Click)
- A REST API and web UI (Flask + Jinja2 templates + static assets)
- A plugin architecture for “analyzers” (text preprocessing/lemmatization) and “backends” (indexing/classification algorithms)

##### High-level Architectural Components
- Clients
- CLI (annif/cli.py)
- REST API / Web UI (annif/rest.py + annif/views.py + annif/templates + annif/static)
- Core Application
- Request dispatcher & API layer (Flask app in rest.py, OpenAPI spec in annif/openapi/annif.yaml, validation.py)
- Project manager (annif/project.py, annif/registry.py, projects.cfg)
- Data directory abstraction (annif/datadir.py)
- Processing Pipeline
- Analyzers (annif/analyzer/*.py): spaCy, EstNLTK, Voikko, simple, etc.
- Transformation filters (annif/transform/*.py): input limiting, language filtering
- Lexical utilities (annif/lexical/*.py): token sets, LLM helpers
- Backends (annif/backend/*.py): TF–IDF, fastText, SVC, neural, ensemble, HTTP remote, Omikuji, YAKE, hyperopt, etc.
- Vocabulary & corpus loaders (annif/vocab/*.py, annif/corpus/*.py)
- Evaluation & suggestion logic (annif/suggestion.py, annif/eval.py)
- Storage & Configuration
- File-based projects configuration (projects.cfg(.dist), pyproject.toml, setup.cfg)
- Model and vocabulary files under data directory or Hugging Face cache
- Testing & CI/CD
- Extensive unit tests (tests/)
- GitHub workflows for CI, CodeQL, Docker rebuild, etc.

##### Component Interactions & Data Flow
1. Client issues a request
– CLI or HTTP request → CLI handler or Flask endpoint
2. Routing & Validation
– REST endpoints validated against OpenAPI spec → call service functions
3. Project Initialization
– Project registry loads project settings, vocabularies, models
4. Input Processing
– Document text → chosen Analyzer(s) → preprocessed token stream
– Token stream → Transform filters (lang, length)
5. Feature Extraction & Classification
– Lexical utilities generate features or prompt text
– Backends consume features (or send remote requests in HTTP backend) → produce subject candidates + scores
6. Post-processing
– Suggestions filtered, scored, formatted
7. Output
– Suggestions returned as CLI text, JSON (API), or rendered HTML in Web UI

##### Architectural Patterns & Design Principles
- Plugin Pattern for Analyzers & Backends: each analyzer/backend implements a common interface (base classes in annif/analyzer/analyzer.py and annif/backend/backend.py)
- Separation of Concerns:
- CLI vs. REST vs. core logic
- Text processing vs. model inference vs. I/O
- Configuration‐driven: new projects and backends can be added via config files—no code changes needed
- Stateless micro‐request design: each indexing request is independent

##### Technologies & Frameworks
- Python 3.10–3.12, Click (CLI), Flask (REST), Jinja2 (templates)
- Optional dependencies: spaCy, EstNLTK, Voikko, fastText, Scikit-learn, Hugging Face transformers
- Docker for containerization (Dockerfile + docker-compose.yml)
- Poetry for development environment (pyproject.toml)
- OpenAPI / Swagger for API schema

##### Essential Details
- Do not omit the registry/project loader—it’s key to how Annif discovers plugins and models
- Both synchronous (in‐process Python backends) and asynchronous/remote (HTTP backend) paths
- Detail static asset serving for the web UI (CSS, JS, templates)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Tier
    subgraph "Client Tier"
        CLI["CLI Interface"]:::client
        Browser["Web UI Browser"]:::client
    end

    %% API Layer
    subgraph "API Layer"
        REST["REST API Entry Point"]:::api
        Views["Web UI Views"]:::api
        OpenAPI["OpenAPI Spec & Validation"]:::api
    end

    %% Core Engine
    subgraph "Core Engine"
        Dispatcher["Dispatcher & Validation"]:::core
        ProjectMgr["Project Manager & Registry"]:::core
        DataDir["Data Directory Abstraction"]:::data
        Suggestion["Suggestion & Evaluation Logic"]:::core
    end

    %% Plugin Modules
    subgraph "Plugin Modules"
        subgraph "Analyzers & Transforms"
            Analyzers["Analyzer Plugins"]:::plugin
            Transforms["Transformation Filters"]:::plugin
            Lexical["Lexical Utilities"]:::plugin
        end
        subgraph "Backends"
            Backends["Backend Plugins"]:::plugin
        end
    end

    %% Data Stores
    subgraph "Data Stores"
        Config["Configuration & Defaults"]:::data
        ProjectsCfg["projects.cfg(.dist)"]:::data
        Models["Model & Vocab Files"]:::data
        HFcache["Hugging Face Cache"]:::data
    end

    %% CI & Testing
    subgraph "CI & Testing"
        Tests["Testing Suite"]:::data
        Docker["Docker & CI/CD Config"]:::data
    end

    %% Data Flow
    CLI -->|"raw text"| Dispatcher
    Browser -->|HTTP request| REST
    REST -->|"validated request"| Dispatcher
    Dispatcher -->|"load project"| ProjectMgr
    ProjectMgr -->|"proj settings"| DataDir
    Dispatcher -->|"document text"| Analyzers
    Analyzers -->|"tokens"| Transforms
    Transforms -->|"filtered tokens"| Lexical
    Lexical -->|"features"| Backends
    Backends -->|"candidates & scores"| Suggestion
    Suggestion -->|"formatted suggestions"| Dispatcher
    Dispatcher -->|"JSON/HTML"| REST
    REST -->|JSON/HTML| Browser
    Dispatcher -->|CLI output| CLI

    %% Configuration Links
    Config --> ProjectMgr
    ProjectsCfg --> ProjectMgr
    DataDir --> Models
    DataDir --> HFcache

    %% Testing & CI Links
    Tests --> Dispatcher
    Docker --> REST

    %% Click Events
    click CLI "https://github.com/natlibfi/annif/blob/main/annif/cli.py"
    click CLI "https://github.com/natlibfi/annif/blob/main/annif/cli_util.py"
    click REST "https://github.com/natlibfi/annif/blob/main/annif/rest.py"
    click Views "https://github.com/natlibfi/annif/blob/main/annif/views.py"
    click Views "https://github.com/natlibfi/annif/blob/main/annif/templates/home.html"
    click Views "https://github.com/natlibfi/annif/tree/main/annif/static/"
    click OpenAPI "https://github.com/natlibfi/annif/blob/main/annif/openapi/annif.yaml"
    click OpenAPI "https://github.com/natlibfi/annif/blob/main/annif/openapi/validation.py"
    click ProjectMgr "https://github.com/natlibfi/annif/blob/main/annif/project.py"
    click ProjectMgr "https://github.com/natlibfi/annif/blob/main/annif/registry.py"
    click ProjectsCfg "https://github.com/natlibfi/annif/blob/main/projects.cfg.dist"
    click DataDir "https://github.com/natlibfi/annif/blob/main/annif/datadir.py"
    click Analyzers "https://github.com/natlibfi/annif/blob/main/annif/analyzer/analyzer.py"
    click Transforms "https://github.com/natlibfi/annif/blob/main/annif/transform/transform.py"
    click Lexical "https://github.com/natlibfi/annif/blob/main/annif/lexical/mllm.py"
    click Backends "https://github.com/natlibfi/annif/blob/main/annif/backend/backend.py"
    click Suggestion "https://github.com/natlibfi/annif/blob/main/annif/suggestion.py"
    click Suggestion "https://github.com/natlibfi/annif/blob/main/annif/eval.py"
    click Config "https://github.com/natlibfi/annif/blob/main/annif/config.py"
    click Config "https://github.com/natlibfi/annif/blob/main/annif/default_config.py"
    click Config "https://github.com/natlibfi/annif/blob/main/pyproject.toml"
    click Config "https://github.com/natlibfi/annif/blob/main/setup.cfg"
    click Tests "https://github.com/natlibfi/annif/tree/main/tests/"
    click Docker "https://github.com/natlibfi/annif/blob/main/docker-compose.yml"

    %% Styles
    classDef client fill:#D0E8FF,stroke:#0284C7,color:#000;
    classDef api fill:#D4FADF,stroke:#059669,color:#000;
    classDef core fill:#E0F2FE,stroke:#0369A1,color:#000;
    classDef data fill:#F3F4F6,stroke:#6B7280,color:#000;
    classDef plugin fill:#FEF3C7,stroke:#F59E0B,color:#000;
```
