# crispae/botannotator architecture

> Architecture diagram of the GitHub repository crispae/botannotator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/crispae/botannotator
- Repository: https://github.com/crispae/botannotator

## Overview

##### Project Type and Purpose
- BotAnnotator is an open-source, modular text-mining component specialized in biomedical concept extraction (NER) for AOP development.
- It integrates multiple NER engines (BERN2, REACH, Huner, BioBERT, ABNER, BANNER, TEES, SPARSER) behind a common Python API/CLI and provides normalization, parsing, and optional web UIs for results.

##### File-Structure Analysis
- Top-level:
- app.py / config.py / config.json – core orchestrator and configuration files
- README.md – project overview and instructions
- NER/ – submodules for each NER engine
- sources/ – model/data resources for BIONER
- NER/ subfolders:
- Bern – a Flask-based mini-app (static/html templates) and wrapper scripts for BERN2
- bern2, multi_ner – Python packages to launch and communicate with the BERN2 server, handle conversion and normalization
- normalizers/ – entity-specific normalization modules
- preprocess/ – scripts preparing ontologies and vocabularies
- scripts/ – shell wrappers to start/stop local NER servers
- The presence of Docker instructions in README signals that most NER engines live in separate containers/services.

##### Key Insights from README
- Plugins are external services (Docker or Java-based) exposing REST or RPC endpoints.
- BotAnnotator orchestrates: reads text, dispatches to each NER service in parallel, normalizes and merges results, and outputs annotated data.

a. Main Components
1. Client Layer
- CLI or REST API consumer (invokes app.py)
2. Orchestrator (Core Python service)
- app.py/controller
- config loader (config.json / config.py)
- Preprocessor (NER/preprocess)
- Normalizers (NER/normalizers)
- Result Parser (NER/Bern/app/result_parser.py)
3. NER Engines (external microservices/containers)
- BERN2 (multi_ner/ner_server.py)
- REACH Docker container
- Huner Docker container
- BioBERT model (in-process or separate service)
- ABNER, BANNER, TEES, SPARSER (each in Docker or Java)
4. Model/Data Repositories
- sources/Bioner
- Preprocessed ontology files (NER/preprocess)
5. Optional UI for BERN results (NER/Bern/app/templates + static assets)
b. Relationships & Interactions
- Client → Orchestrator (HTTP/CLI invocation)
- Orchestrator loads config → identifies active NER plugins
- Orchestrator → Preprocessor (if required) → sends cleaned text
- Orchestrator → For each NER Engine (concurrently):
- Calls REST/RPC endpoint
- Receives raw JSON/XML response
- Orchestrator → Normalizers (map raw entities to standard IDs)
- Orchestrator → Result Parser (merge, dedupe, format)
- Orchestrator → Returns unified annotated output to Client
- Optional: Orchestrator → BERN Web UI (serving via Flask blueprint)
c. Architectural Patterns & Design Principles
- Plugin-based architecture: each NER engine implements a common interface (convert, call_server, parse)
- Microservices: external engines live in separate containers/processes
- Pipeline processing: sequential stages (preprocess → NER → normalize → parse → output)
- Configuration-driven: config.json enables/disables engines, sets endpoints/ports
d. Key Technologies
- Python 3.7+, Flask (for BERN UI and core API)
- Docker (for REACH, Huner, ABNER, etc.)
- Java (BERN2’s backend)
- Templates (Jinja2) + static (CSS/JS) for documentation UI
- External storage: Google Drive–hosted BERN2 resources

##### Essential Details to Capture
- Plugin lifecycle: load config → start/stop shells or HTTP connections (scripts/run_bern2.sh, docker run)
- Data flow through preprocessors, normalizers, parser modules
- Optional UI layer specific to BERN
- Ability to toggle engines via config.json without code changes

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Client Layer"
        Client["CLI / REST Client"]:::ui
    end

    subgraph "Core Orchestrator (Python Modules)"
        App["Orchestrator (app.py)"]:::core
        ConfigJson["config.json"]:::doc
        ConfigPy["config.py"]:::core
        Preprocessor["Preprocessor Modules"]:::core
        Normalizers["Entity Normalizers"]:::core
        Parser["Result Parser"]:::core
        README["README.md"]:::doc
        Init["__init__.py"]:::core
    end

    subgraph "Data Repositories"
        BionerRepo["sources/Bioner"]:::data
        Ontology["Preprocessed Ontologies"]:::data
    end

    subgraph "BERN Web UI"
        UI_Templates["Templates"]:::ui
        UI_Static["Static Assets"]:::ui
    end

    subgraph "BERN2 Wrapper"
        bern2_core["bern2.py"]:::core
        bern2_convert["convert.py"]:::core
        bern2_normalizer["normalizer.py"]:::core
    end

    subgraph "BERN2 Multi-NER Server"
        multi_server["ner_server.py"]:::service
        multi_main["main.py"]:::service
        multi_convert["convert.py"]:::service
        multi_ops["ops.py"]:::service
        training["training/"]:::service
    end

    subgraph "BERN2 Control Scripts"
        Scripts["scripts/"]:::service
    end

    subgraph "Standalone BERN2 Servers"
        B2p["Bern2.py"]:::service
        B2s["Bern2server.py"]:::service
    end

    subgraph "Other NER Engines"
        Reach["REACH"]:::service
        HuNER["HuNER"]:::service
        BioBERT["BioBERT"]:::service
        ABNER["ABNER"]:::service
        BANNER["BANNER"]:::service
        TEES["TEES"]:::service
        SPARSER["SPARSER"]:::service
        BionerEngine["BIONER Wrapper"]:::service
    end

    Client -->|"invoke"| App
    App -->|"load config"| ConfigJson
    App --> ConfigPy
    App --> Preprocessor
    App --> Normalizers
    App --> Parser
    App --> UI_Templates
    App --> UI_Static
    App --> BionerRepo
    App --> Ontology
    App --> bern2_core
    App --> bern2_convert
    App --> bern2_normalizer
    App --> multi_server
    App --> multi_main
    App --> multi_convert
    App --> multi_ops
    App --> training
    App --> Scripts
    App --> B2p
    App --> B2s
    App --> Reach
    App --> HuNER
    App --> BioBERT
    App --> ABNER
    App --> BANNER
    App --> TEES
    App --> SPARSER
    App --> BionerEngine

    multi_server -->|response| App
    Reach -->|response| App
    HuNER -->|response| App
    BioBERT -->|response| App
    ABNER -->|response| App
    BANNER -->|response| App
    TEES -->|response| App
    SPARSER -->|response| App
    BionerEngine -->|response| App

    App -->|"normalized"| Normalizers
    Normalizers --> Parser
    Parser -->|output| Client

    click App "https://github.com/crispae/botannotator/blob/main/app.py"
    click ConfigJson "https://github.com/crispae/botannotator/blob/main/config.json"
    click ConfigPy "https://github.com/crispae/botannotator/blob/main/config.py"
    click Preprocessor "https://github.com/crispae/botannotator/tree/main/NER/Bern/preprocess/"
    click Normalizers "https://github.com/crispae/botannotator/tree/main/NER/Bern/normalizers/"
    click Parser "https://github.com/crispae/botannotator/blob/main/NER/Bern/app/result_parser.py"
    click UI_Templates "https://github.com/crispae/botannotator/tree/main/NER/Bern/app/templates/"
    click UI_Static "https://github.com/crispae/botannotator/tree/main/NER/Bern/app/static/"
    click bern2_core "https://github.com/crispae/botannotator/blob/main/NER/Bern/bern2/bern2.py"
    click bern2_convert "https://github.com/crispae/botannotator/blob/main/NER/Bern/bern2/convert.py"
    click bern2_normalizer "https://github.com/crispae/botannotator/blob/main/NER/Bern/bern2/normalizer.py"
    click multi_server "https://github.com/crispae/botannotator/blob/main/NER/Bern/multi_ner/ner_server.py"
    click multi_main "https://github.com/crispae/botannotator/blob/main/NER/Bern/multi_ner/main.py"
    click multi_convert "https://github.com/crispae/botannotator/blob/main/NER/Bern/multi_ner/convert.py"
    click multi_ops "https://github.com/crispae/botannotator/blob/main/NER/Bern/multi_ner/ops.py"
    click training "https://github.com/crispae/botannotator/tree/main/NER/Bern/multi_ner/training/"
    click Scripts "https://github.com/crispae/botannotator/tree/main/NER/Bern/scripts/"
    click B2p "https://github.com/crispae/botannotator/blob/main/NER/Bern2.py"
    click B2s "https://github.com/crispae/botannotator/blob/main/NER/Bern2server.py"
    click BionerEngine "https://github.com/crispae/botannotator/blob/main/NER/Bioner.py"
    click BionerRepo "https://github.com/crispae/botannotator/tree/main/sources/Bioner/"
    click Reach "https://github.com/crispae/botannotator/blob/main/NER/Reach.py"
    click HuNER "https://github.com/crispae/botannotator/blob/main/NER/Huner.py"
    click ABNER "https://github.com/crispae/botannotator/blob/main/NER/Abner.py"
    click README "https://github.com/crispae/botannotator/blob/main/README.md"
    click Init "https://github.com/crispae/botannotator/blob/main/__init__.py"

    classDef core fill:#E0F7FA,stroke:#006064
    classDef service fill:#E8F5E9,stroke:#2E7D32
    classDef data fill:#FFF3E0,stroke:#E65100
    classDef ui fill:#F5F5F5,stroke:#616161
    classDef doc fill:#ECEFF1,stroke:#455A64
```
