# drmoumenai/neurolit-miner architecture

> Architecture diagram of the GitHub repository drmoumenai/neurolit-miner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/drmoumenai/neurolit-miner
- Repository: https://github.com/drmoumenai/neurolit-miner

## Overview

1. **Overall shape**
   NeuroLit Miner is a local Python evidence-infrastructure app with two entry points: a command-line workflow and a Flask-based web UI. Both front ends talk to the same core services and the same SQLite database, which is the single source of truth for literature, trials, taxonomy tags, and search history.

2. **External data acquisition**
   The ingestion layer is split across two API modules:
   - `src/pubmed_api.py` for NCBI E-utilities PubMed retrieval
   - `src/trials_api.py` for ClinicalTrials.gov API v2  
   These modules are the only boundary that reaches out to external biomedical services.

3. **Parsing and semantic enrichment**
   `src/parser.py` is the normalization and classification boundary. It turns PubMed XML into structured article records and applies the V3.1 three-axis taxonomy:
   - clinical topic
   - methodology
   - domain  
   It also uses MeSH terms as the preferred semantic anchor and assigns confidence levels, so classification is not just keyword-based.

4. **Persistence layer**
   `src/database.py` owns the SQLite schema, writes new records, supports deduplication, and handles backfill/migration behavior. This is the core stateful layer of the system. The README makes clear that the local database is the authoritative corpus; ingestion appends rather than overwriting.

5. **CLI orchestration**
   `src/main.py` is the general command-line entry point for retrieval, database search, taxonomy backfill, and export operations. It orchestrates calls into the API, parser, database, and exporter modules depending on flags. `run_app.py` is a small convenience launcher for starting the web interface.

6. **Web application boundary**
   `src/app.py` is the Flask server and API layer for local interaction. It serves the UI template in `templates/index.html` and exposes the web workflow described in the README: PubMed search, local database browsing, analytics, and clinical trials views. This is a local-only presentation layer, not a production web service.

7. **Analytics subsystem**
   `src/analyze.py` reads from SQLite and generates publication-style outputs, mainly PNG figures and related summaries. The README indicates five analysis types, including year trends, journal distributions, MeSH frequency, and MeSH co-occurrence. This module is read-heavy and does not own data; it transforms stored corpus data into visual analytics artifacts.

8. **Evidence synthesis subsystem**
   `src/summarize.py` is the synthesis CLI, and `src/summarizer.py` provides the provider abstraction. This layer selects a corpus from SQLite using taxonomy filters, then generates narrative evidence summaries. It supports mock mode by default and can route to Ollama, Anthropic, OpenAI, or OpenRouter through the same interface.

9. **Export and interoperability**
   `src/exporter.py` handles structured export of stored records, especially CSV output for reference managers and downstream review workflows. This is a boundary from the database into portable artifacts, separate from both analytics figures and synthesis reports.

10. **Data flow through the system**
   The main flow is: external PubMed / ClinicalTrials.gov data → parsing and taxonomy enrichment → SQLite persistence → downstream retrieval for analytics, export, or synthesis. The web UI and CLI are just different entry points into that same pipeline. Backfill support means taxonomy logic can be reapplied to already stored records without re-ingesting them.

11. **Important architectural constraints**
   The repo is intentionally local-first:
   - Python 3.10+ runtime
   - SQLite for storage
   - Flask for the UI
   - requests/XML parsing for API ingestion
   - matplotlib/numpy-style analysis outputs
   - optional LLM backends, with mock fallback  
   The architecture is resilient to missing model providers because synthesis can degrade to mock mode rather than failing.

12. **Repository boundaries to show in a diagram**
   A clear architecture diagram should separate these subsystems:
   - user entry points: CLI and Flask UI
   - external services: PubMed, ClinicalTrials.gov, optional LLM providers
   - core processing: retrieval, parsing/taxonomy, persistence, analytics, synthesis
   - local artifacts: SQLite database, generated figures, CSVs, summary files, HTML template  
   The key design idea is that classification and synthesis are built on top of a persistent, queryable corpus rather than performed ad hoc at search time.

Main entry points: [`src/database.py`](https://github.com/drmoumenai/neurolit-miner/blob/main/src/database.py) (SQLite store), [`src/main.py`](https://github.com/drmoumenai/neurolit-miner/blob/main/src/main.py) (CLI workflow), [`src/app.py`](https://github.com/drmoumenai/neurolit-miner/blob/main/src/app.py) (Flask app), [`src/parser.py`](https://github.com/drmoumenai/neurolit-miner/blob/main/src/parser.py) (Parser)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry points"]
  node_cli_main["CLI workflow<br/>command entry<br/>[main.py]"]
  node_web_app["Flask app<br/>web entry<br/>[app.py]"]
  node_launcher["Launcher<br/>startup helper<br/>[run_app.py]"]
end

subgraph group_external["External services"]
  node_pubmed["PubMed API<br/>ingestion client<br/>[pubmed_api.py]"]
  node_trials["Trials API<br/>ingestion client<br/>[trials_api.py]"]
  node_llm["LLM providers<br/>optional synthesis backends<br/>[summarizer.py]"]
end

subgraph group_core["Core pipeline"]
  node_parser["Parser<br/>normalization layer<br/>[parser.py]"]
  node_db[("SQLite store<br/>persistence layer<br/>[database.py]")]
  node_analyze["Analytics<br/>read-only analysis<br/>[analyze.py]"]
  node_summarize["Synthesis CLI<br/>summary workflow<br/>[summarize.py]"]
  node_exporter["Exporter<br/>data export<br/>[exporter.py]"]
end

subgraph group_artifacts["Local artifacts"]
  node_template["UI template<br/>presentation asset<br/>[index.html]"]
  node_figures["Figures<br/>generated output"]
  node_csvs["CSV exports<br/>generated output"]
  node_summaries["Summary files<br/>generated output"]
end

node_cli_main -->|"retrieve"| node_pubmed
node_cli_main -->|"retrieve"| node_trials
node_cli_main -->|"normalize"| node_parser
node_cli_main -->|"store/query"| node_db
node_cli_main -->|"analyze"| node_analyze
node_cli_main -->|"summarize"| node_summarize
node_cli_main -->|"export"| node_exporter
node_web_app -->|"browse/query"| node_db
node_web_app -->|"analytics"| node_analyze
node_web_app -->|"trials view"| node_trials
node_web_app -->|"render"| node_template
node_launcher -->|"start"| node_web_app
node_pubmed -->|"xml records"| node_parser
node_trials -->|"trial data"| node_db
node_parser -->|"write records"| node_db
node_parser -.->|"backfill"| node_db
node_db -->|"read corpus"| node_analyze
node_db -->|"select corpus"| node_summarize
node_db -->|"export source"| node_exporter
node_analyze -->|"write figures"| node_figures
node_exporter -->|"write csv"| node_csvs
node_summarize -->|"generate text"| node_llm
node_summarize -->|"write summaries"| node_summaries

click node_cli_main "https://github.com/drmoumenai/neurolit-miner/blob/main/src/main.py"
click node_web_app "https://github.com/drmoumenai/neurolit-miner/blob/main/src/app.py"
click node_launcher "https://github.com/drmoumenai/neurolit-miner/blob/main/run_app.py"
click node_pubmed "https://github.com/drmoumenai/neurolit-miner/blob/main/src/pubmed_api.py"
click node_trials "https://github.com/drmoumenai/neurolit-miner/blob/main/src/trials_api.py"
click node_llm "https://github.com/drmoumenai/neurolit-miner/blob/main/src/summarizer.py"
click node_parser "https://github.com/drmoumenai/neurolit-miner/blob/main/src/parser.py"
click node_db "https://github.com/drmoumenai/neurolit-miner/blob/main/src/database.py"
click node_analyze "https://github.com/drmoumenai/neurolit-miner/blob/main/src/analyze.py"
click node_summarize "https://github.com/drmoumenai/neurolit-miner/blob/main/src/summarize.py"
click node_exporter "https://github.com/drmoumenai/neurolit-miner/blob/main/src/exporter.py"
click node_template "https://github.com/drmoumenai/neurolit-miner/blob/main/templates/index.html"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli_main,node_web_app,node_launcher toneBlue
class node_pubmed,node_trials,node_llm toneAmber
class node_parser,node_db,node_analyze,node_summarize,node_exporter toneMint
class node_template,node_figures,node_csvs,node_summaries toneRose
```

## Components

### Entry points

User-facing ways to run the local app.

- [CLI workflow](https://github.com/drmoumenai/neurolit-miner/blob/main/src/main.py): command entry · Main command-line entry point for retrieval, search, backfill, and export. · `src/main.py`
- [Flask app](https://github.com/drmoumenai/neurolit-miner/blob/main/src/app.py): web entry · Local Flask server for browsing, search, analytics, and trials views. · `src/app.py`
- [Launcher](https://github.com/drmoumenai/neurolit-miner/blob/main/run_app.py): startup helper · Convenience script for starting the web interface. · `run_app.py`

### External services

Remote biomedical and optional model providers.

- [PubMed API](https://github.com/drmoumenai/neurolit-miner/blob/main/src/pubmed_api.py): ingestion client · NCBI E-utilities access for literature retrieval. · `src/pubmed_api.py`
- [Trials API](https://github.com/drmoumenai/neurolit-miner/blob/main/src/trials_api.py): ingestion client · ClinicalTrials.gov API v2 access. · `src/trials_api.py`
- [LLM providers](https://github.com/drmoumenai/neurolit-miner/blob/main/src/summarizer.py): optional synthesis backends · Mock mode plus Ollama, Anthropic, OpenAI, and OpenRouter interfaces. · `src/summarizer.py`

### Core pipeline

Ingestion, normalization, persistence, querying, and synthesis.

- [Parser](https://github.com/drmoumenai/neurolit-miner/blob/main/src/parser.py): normalization layer · Transforms PubMed XML into structured records and applies the V3.1 taxonomy. · `src/parser.py`
- [SQLite store](https://github.com/drmoumenai/neurolit-miner/blob/main/src/database.py): persistence layer · Authoritative local corpus with schema, deduplication, writes, and backfill support. · `src/database.py`
- [Analytics](https://github.com/drmoumenai/neurolit-miner/blob/main/src/analyze.py): read-only analysis · Generates publication-style figures and corpus summaries from stored data. · `src/analyze.py`
- [Synthesis CLI](https://github.com/drmoumenai/neurolit-miner/blob/main/src/summarize.py): summary workflow · Selects corpus slices and generates narrative evidence summaries. · `src/summarize.py`
- [Exporter](https://github.com/drmoumenai/neurolit-miner/blob/main/src/exporter.py): data export · Produces structured exports such as CSV for downstream workflows. · `src/exporter.py`

### Local artifacts

Files produced or served locally by the system.

- [UI template](https://github.com/drmoumenai/neurolit-miner/blob/main/templates/index.html): presentation asset · HTML template served by the Flask interface. · `templates/index.html`
- **Figures**: generated output · Rendered analysis artifacts such as PNG charts.
- **CSV exports**: generated output · Portable exported records for reference managers and reviews.
- **Summary files**: generated output · Narrative evidence synthesis outputs produced from stored corpora.

## Connections

- CLI workflow → PubMed API: retrieve · Fetches literature records from PubMed.
- CLI workflow → Trials API: retrieve · Fetches clinical trial records.
- CLI workflow → Parser: normalize · Parses and classifies retrieved records.
- CLI workflow → SQLite store: store/query · Writes to and queries the local corpus.
- CLI workflow → Analytics: analyze · Runs corpus analysis workflows.
- CLI workflow → Synthesis CLI: summarize · Runs evidence synthesis workflows.
- CLI workflow → Exporter: export · Exports stored records.
- Flask app → SQLite store: browse/query · Serves local search and database views.
- Flask app → Analytics: analytics · Drives analytics views from stored data.
- Flask app → Trials API: trials view · Displays trial data fetched through the local workflow.
- Flask app → UI template: render · Serves the HTML interface.
- Launcher → Flask app: start · Starts the Flask application.
- PubMed API → Parser: xml records · Provides raw PubMed content for normalization.
- Trials API → SQLite store: trial data · Feeds trial records into local storage workflows.
- Parser → SQLite store: write records · Persists normalized, taxonomized records.
- Parser → SQLite store: backfill · Reapplies taxonomy and enrichment to stored records.
- SQLite store → Analytics: read corpus · Supplies stored evidence for analysis.
- SQLite store → Synthesis CLI: select corpus · Provides filtered datasets for synthesis.
- SQLite store → Exporter: export source · Provides records for CSV and other exports.
- Analytics → Figures: write figures · Outputs charts and summary artifacts.
- Exporter → CSV exports: write csv · Produces portable export files.
- Synthesis CLI → LLM providers: generate text · Uses optional provider backends or mock mode.
- Synthesis CLI → Summary files: write summaries · Outputs narrative evidence summaries.
