How crispae/botannotator works
Project Type and Purpose
• BotAnnotator is an open-source, modular text-mining component specialized in biomedical concept extraction (NER) for AOP development.
• It integrates multiple NER engines (BERN2, REACH, Huner, BioBERT, ABNER, BANNER, TEES, SPARSER) behind a common Python API/CLI and provides normalization, parsing, and optional web UIs for results.
File-Structure Analysis
• Top-level:
• app.py / config.py / config.json – core orchestrator and configuration files
• README.md – project overview and instructions
• NER/ – submodules for each NER engine
• sources/ – model/data resources for BIONER
• NER/ subfolders:
• Bern – a Flask-based mini-app (static/html templates) and wrapper scripts for BERN2
• bern2, multi_ner – Python packages to launch and communicate with the BERN2 server, handle conversion and normalization
• normalizers/ – entity-specific normalization modules
• preprocess/ – scripts preparing ontologies and vocabularies
• scripts/ – shell wrappers to start/stop local NER servers
• The presence of Docker instructions in README signals that most NER engines live in separate containers/services.
Key Insights from README
• Plugins are external services (Docker or Java-based) exposing REST or RPC endpoints.
• BotAnnotator orchestrates: reads text, dispatches to each NER service in parallel, normalizes and merges results, and outputs annotated data.
a. Main Components
1. Client Layer
• CLI or REST API consumer (invokes app.py)
2. Orchestrator (Core Python service)
• app.py/controller
• config loader (config.json / config.py)
• Preprocessor (NER/preprocess)
• Normalizers (NER/normalizers)
• Result Parser (NER/Bern/app/result_parser.py)
3. NER Engines (external microservices/containers)
• BERN2 (multi_ner/ner_server.py)
• REACH Docker container
• Huner Docker container
• BioBERT model (in-process or separate service)
• ABNER, BANNER, TEES, SPARSER (each in Docker or Java)
4. Model/Data Repositories
• sources/Bioner
• Preprocessed ontology files (NER/preprocess)
5. Optional UI for BERN results (NER/Bern/app/templates + static assets)
b. Relationships & Interactions
• Client → Orchestrator (HTTP/CLI invocation)
• Orchestrator loads config → identifies active NER plugins
• Orchestrator → Preprocessor (if required) → sends cleaned text
• Orchestrator → For each NER Engine (concurrently):
• Calls REST/RPC endpoint
• Receives raw JSON/XML response
• Orchestrator → Normalizers (map raw entities to standard IDs)
• Orchestrator → Result Parser (merge, dedupe, format)
• Orchestrator → Returns unified annotated output to Client
• Optional: Orchestrator → BERN Web UI (serving via Flask blueprint)
c. Architectural Patterns & Design Principles
• Plugin-based architecture: each NER engine implements a common interface (convert, call_server, parse)
• Microservices: external engines live in separate containers/processes
• Pipeline processing: sequential stages (preprocess → NER → normalize → parse → output)
• Configuration-driven: config.json enables/disables engines, sets endpoints/ports
d. Key Technologies
• Python 3.7+, Flask (for BERN UI and core API)
• Docker (for REACH, Huner, ABNER, etc.)
• Java (BERN2’s backend)
• Templates (Jinja2) + static (CSS/JS) for documentation UI
• External storage: Google Drive–hosted BERN2 resources
Essential Details to Capture
• Plugin lifecycle: load config → start/stop shells or HTTP connections (scripts/run_bern2.sh, docker run)
• Data flow through preprocessors, normalizers, parser modules
• Optional UI layer specific to BERN
• Ability to toggle engines via config.json without code changes