How natlibfi/annif works
Project type and purpose
• Annif is a modular, full-stack toolkit for automated subject indexing.
• It provides:
• A command‐line interface (CLI built with Click)
• A REST API and web UI (Flask + Jinja2 templates + static assets)
• A plugin architecture for “analyzers” (text preprocessing/lemmatization) and “backends” (indexing/classification algorithms)
High-level Architectural Components
• Clients
• CLI (annif/cli.py)
• REST API / Web UI (annif/rest.py + annif/views.py + annif/templates + annif/static)
• Core Application
• Request dispatcher & API layer (Flask app in rest.py, OpenAPI spec in annif/openapi/annif.yaml, validation.py)
• Project manager (annif/project.py, annif/registry.py, projects.cfg)
• Data directory abstraction (annif/datadir.py)
• Processing Pipeline
• Analyzers (annif/analyzer/*.py): spaCy, EstNLTK, Voikko, simple, etc.
• Transformation filters (annif/transform/*.py): input limiting, language filtering
• Lexical utilities (annif/lexical/*.py): token sets, LLM helpers
• Backends (annif/backend/*.py): TF–IDF, fastText, SVC, neural, ensemble, HTTP remote, Omikuji, YAKE, hyperopt, etc.
• Vocabulary & corpus loaders (annif/vocab/*.py, annif/corpus/*.py)
• Evaluation & suggestion logic (annif/suggestion.py, annif/eval.py)
• Storage & Configuration
• File-based projects configuration (projects.cfg(.dist), pyproject.toml, setup.cfg)
• Model and vocabulary files under data directory or Hugging Face cache
• Testing & CI/CD
• Extensive unit tests (tests/)
• GitHub workflows for CI, CodeQL, Docker rebuild, etc.
Component Interactions & Data Flow
1. Client issues a request
– CLI or HTTP request → CLI handler or Flask endpoint
2. Routing & Validation
– REST endpoints validated against OpenAPI spec → call service functions
3. Project Initialization
– Project registry loads project settings, vocabularies, models
4. Input Processing
– Document text → chosen Analyzer(s) → preprocessed token stream
– Token stream → Transform filters (lang, length)
5. Feature Extraction & Classification
– Lexical utilities generate features or prompt text
– Backends consume features (or send remote requests in HTTP backend) → produce subject candidates + scores
6. Post-processing
– Suggestions filtered, scored, formatted
7. Output
– Suggestions returned as CLI text, JSON (API), or rendered HTML in Web UI
Architectural Patterns & Design Principles
• Plugin Pattern for Analyzers & Backends: each analyzer/backend implements a common interface (base classes in annif/analyzer/analyzer.py and annif/backend/backend.py)
• Separation of Concerns:
• CLI vs. REST vs. core logic
• Text processing vs. model inference vs. I/O
• Configuration‐driven: new projects and backends can be added via config files—no code changes needed
• Stateless micro‐request design: each indexing request is independent
Technologies & Frameworks
• Python 3.10–3.12, Click (CLI), Flask (REST), Jinja2 (templates)
• Optional dependencies: spaCy, EstNLTK, Voikko, fastText, Scikit-learn, Hugging Face transformers
• Docker for containerization (Dockerfile + docker-compose.yml)
• Poetry for development environment (pyproject.toml)
• OpenAPI / Swagger for API schema
Essential Details
• Do not omit the registry/project loader—it’s key to how Annif discovers plugins and models
• Both synchronous (in‐process Python backends) and asynchronous/remote (HTTP backend) paths
• Detail static asset serving for the web UI (CSS, JS, templates)