How drmoumenai/neurolit-miner works
1. Overall shape
NeuroLit Miner is a local Python evidence-infrastructure app with two entry points: a command-line workflow and a Flask-based web UI. Both front ends talk to the same core services and the same SQLite database, which is the single source of truth for literature, trials, taxonomy tags, and search history.
2. External data acquisition
The ingestion layer is split across two API modules:
- src/pubmed_api.py for NCBI E-utilities PubMed retrieval
- src/trials_api.py for ClinicalTrials.gov API v2
These modules are the only boundary that reaches out to external biomedical services.
3. Parsing and semantic enrichment
src/parser.py is the normalization and classification boundary. It turns PubMed XML into structured article records and applies the V3.1 three-axis taxonomy:
- clinical topic
- methodology
- domain
It also uses MeSH terms as the preferred semantic anchor and assigns confidence levels, so classification is not just keyword-based.
4. Persistence layer
src/database.py owns the SQLite schema, writes new records, supports deduplication, and handles backfill/migration behavior. This is the core stateful layer of the system. The README makes clear that the local database is the authoritative corpus; ingestion appends rather than overwriting.
5. CLI orchestration
src/main.py is the general command-line entry point for retrieval, database search, taxonomy backfill, and export operations. It orchestrates calls into the API, parser, database, and exporter modules depending on flags. run_app.py is a small convenience launcher for starting the web interface.
6. Web application boundary
src/app.py is the Flask server and API layer for local interaction. It serves the UI template in templates/index.html and exposes the web workflow described in the README: PubMed search, local database browsing, analytics, and clinical trials views. This is a local-only presentation layer, not a production web service.
7. Analytics subsystem
src/analyze.py reads from SQLite and generates publication-style outputs, mainly PNG figures and related summaries. The README indicates five analysis types, including year trends, journal distributions, MeSH frequency, and MeSH co-occurrence. This module is read-heavy and does not own data; it transforms stored corpus data into visual analytics artifacts.
8. Evidence synthesis subsystem
src/summarize.py is the synthesis CLI, and src/summarizer.py provides the provider abstraction. This layer selects a corpus from SQLite using taxonomy filters, then generates narrative evidence summaries. It supports mock mode by default and can route to Ollama, Anthropic, OpenAI, or OpenRouter through the same interface.
9. Export and interoperability
src/exporter.py handles structured export of stored records, especially CSV output for reference managers and downstream review workflows. This is a boundary from the database into portable artifacts, separate from both analytics figures and synthesis reports.
10. Data flow through the system
The main flow is: external PubMed / ClinicalTrials.gov data → parsing and taxonomy enrichment → SQLite persistence → downstream retrieval for analytics, export, or synthesis. The web UI and CLI are just different entry points into that same pipeline. Backfill support means taxonomy logic can be reapplied to already stored records without re-ingesting them.
11. Important architectural constraints
The repo is intentionally local-first:
- Python 3.10+ runtime
- SQLite for storage
- Flask for the UI
- requests/XML parsing for API ingestion
- matplotlib/numpy-style analysis outputs
- optional LLM backends, with mock fallback
The architecture is resilient to missing model providers because synthesis can degrade to mock mode rather than failing.
12. Repository boundaries to show in a diagram
A clear architecture diagram should separate these subsystems:
- user entry points: CLI and Flask UI
- external services: PubMed, ClinicalTrials.gov, optional LLM providers
- core processing: retrieval, parsing/taxonomy, persistence, analytics, synthesis
- local artifacts: SQLite database, generated figures, CSVs, summary files, HTML template
The key design idea is that classification and synthesis are built on top of a persistent, queryable corpus rather than performed ad hoc at search time.
Main entry points: database.py SQLite store · main.py CLI workflow · app.py Flask app · parser.py Parser
Components 15
Entry points
User-facing ways to run the local app.
- CLI workflow · Main command-line entry point for retrieval, search, backfill, and export.
src/main.py - Flask app · Local Flask server for browsing, search, analytics, and trials views.
src/app.py - Launcher · Convenience script for starting the web interface.
run_app.py
External services
Remote biomedical and optional model providers.
- PubMed API · NCBI E-utilities access for literature retrieval.
src/pubmed_api.py - Trials API · ClinicalTrials.gov API v2 access.
src/trials_api.py - LLM providers · Mock mode plus Ollama, Anthropic, OpenAI, and OpenRouter interfaces.
src/summarizer.py
Core pipeline
Ingestion, normalization, persistence, querying, and synthesis.
- Parser · Transforms PubMed XML into structured records and applies the V3.1 taxonomy.
src/parser.py - SQLite store · Authoritative local corpus with schema, deduplication, writes, and backfill support.
src/database.py - Analytics · Generates publication-style figures and corpus summaries from stored data.
src/analyze.py - Synthesis CLI · Selects corpus slices and generates narrative evidence summaries.
src/summarize.py - Exporter · Produces structured exports such as CSV for downstream workflows.
src/exporter.py
Local artifacts
Files produced or served locally by the system.
- UI template · HTML template served by the Flask interface.
templates/index.html - Figures · Rendered analysis artifacts such as PNG charts.
- CSV exports · Portable exported records for reference managers and reviews.
- Summary files · Narrative evidence synthesis outputs produced from stored corpora.
Connections 23
- CLI workflow → PubMed API: retrieve · Fetches literature records from PubMed.
- CLI workflow → Trials API: retrieve · Fetches clinical trial records.
- CLI workflow → Parser: normalize · Parses and classifies retrieved records.
- CLI workflow → SQLite store: store/query · Writes to and queries the local corpus.
- CLI workflow → Analytics: analyze · Runs corpus analysis workflows.
- CLI workflow → Synthesis CLI: summarize · Runs evidence synthesis workflows.
- CLI workflow → Exporter: export · Exports stored records.
- Flask app → SQLite store: browse/query · Serves local search and database views.
- Flask app → Analytics: analytics · Drives analytics views from stored data.
- Flask app → Trials API: trials view · Displays trial data fetched through the local workflow.
- Flask app → UI template: render · Serves the HTML interface.
- Launcher → Flask app: start · Starts the Flask application.
- PubMed API → Parser: xml records · Provides raw PubMed content for normalization.
- Trials API → SQLite store: trial data · Feeds trial records into local storage workflows.
- Parser → SQLite store: write records · Persists normalized, taxonomized records.
- Parser → SQLite store: backfill · Reapplies taxonomy and enrichment to stored records.
- SQLite store → Analytics: read corpus · Supplies stored evidence for analysis.
- SQLite store → Synthesis CLI: select corpus · Provides filtered datasets for synthesis.
- SQLite store → Exporter: export source · Provides records for CSV and other exports.
- Analytics → Figures: write figures · Outputs charts and summary artifacts.
- Exporter → CSV exports: write csv · Produces portable export files.
- Synthesis CLI → LLM providers: generate text · Uses optional provider backends or mock mode.
- Synthesis CLI → Summary files: write summaries · Outputs narrative evidence summaries.