# el-bid/smartreader architecture

> Architecture diagram of the GitHub repository el-bid/smartreader, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/el-bid/smartreader
- Repository: https://github.com/el-bid/smartreader

## Overview

SmartReader helps researchers build topic-and-keyword models from Google search results, apply those models to uploaded text corpora, and inspect or download ranked summaries and extracted information. The workflow runs through a Flask interface and two polling workers, with MongoDB tracking queued jobs and local files holding models and results. The README documents the major user-facing stages; some implementation details for UI templates and several indexed modules were not sampled, so their internal links are omitted.

Main entry points: [`main.py`](https://github.com/el-bid/smartreader/blob/master/main.py) (Flask application), [`src/process.py`](https://github.com/el-bid/smartreader/blob/master/src/process.py) (Paragraph scoring and extraction), [`summary_processor.py`](https://github.com/el-bid/smartreader/blob/master/summary_processor.py) (Summary worker), [`src/create_model.py`](https://github.com/el-bid/smartreader/blob/master/src/create_model.py) (TF-IDF model builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_web["Research interface"]
  node_webapp["Flask application<br/>[main.py]"]
  node_ui["Web interfaces"]
end

subgraph group_modeling["Model creation"]
  node_model_worker["Model worker<br/>[processor.py]"]
  node_search["Web search and extraction<br/>[search_data.py]"]
  node_google["Google search client<br/>[__init__.py]"]
  node_model_builder["TF-IDF model builder<br/>[create_model.py]"]
  node_tokenizer["Lemma tokenizer<br/>[LemmaTokenizer.py]"]
end

subgraph group_analysis["Corpus analysis"]
  node_summary_worker["Summary worker"]
  node_dataset_reader["Dataset reader<br/>[dataset_reader.py]"]
  node_document["Document and paragraphs<br/>[document.py]"]
  node_analyzer["Paragraph scoring and extraction<br/>[process.py]"]
  node_summarizer["Sentence summarization<br/>[sumytest.py]"]
  node_spellcheck["Spell checking<br/>[spell_check.py]"]
  node_converter["Annotated document output<br/>[convert_dataset.py]"]
end

subgraph group_state["Jobs and results"]
  node_db[("MongoDB job store")]
  node_model_files[("Saved models")]
end

node_researcher(("Researcher"))

node_researcher -->|"submits topics and corpus"| node_webapp
node_webapp -->|"renders pages"| node_ui
node_webapp -->|"queues and reads jobs"| node_db
node_model_worker -->|"polls and updates"| node_db
node_model_worker -->|"retrieves topic text"| node_search
node_search -->|"searches for URLs"| node_google
node_search -->|"supplies topic text"| node_model_builder
node_model_builder -->|"configures vectorizer"| node_tokenizer
node_model_builder -->|"saves model"| node_model_files
node_webapp -->|"accepts selected model"| node_model_files
node_webapp -->|"queues corpus job"| node_summary_worker
node_summary_worker -->|"polls and updates"| node_db
node_summary_worker -->|"creates summaries"| node_analyzer
node_analyzer -->|"loads corpus"| node_dataset_reader
node_dataset_reader -->|"creates documents"| node_document
node_analyzer -->|"loads saved model"| node_model_builder
node_analyzer -->|"selects sentences"| node_summarizer
node_analyzer -->|"checks selected text"| node_spellcheck
node_summary_worker -->|"annotates source documents"| node_converter
node_researcher -->|"views or downloads results"| node_webapp

click node_webapp "https://github.com/el-bid/smartreader/blob/master/main.py"
click node_ui "https://github.com/el-bid/smartreader/tree/master/templates"
click node_db "https://github.com/el-bid/smartreader/blob/master/src/database_connectivity.py"
click node_model_worker "https://github.com/el-bid/smartreader/blob/master/processor.py"
click node_search "https://github.com/el-bid/smartreader/blob/master/src/search_data.py"
click node_google "https://github.com/el-bid/smartreader/blob/master/_google_/__init__.py"
click node_model_builder "https://github.com/el-bid/smartreader/blob/master/src/create_model.py"
click node_tokenizer "https://github.com/el-bid/smartreader/blob/master/src/LemmaTokenizer.py"
click node_model_files "https://github.com/el-bid/smartreader/tree/master/models"
click node_summary_worker "https://github.com/el-bid/smartreader/blob/master/summary_processor.py"
click node_dataset_reader "https://github.com/el-bid/smartreader/blob/master/src/dataset_reader.py"
click node_document "https://github.com/el-bid/smartreader/blob/master/src/document.py"
click node_analyzer "https://github.com/el-bid/smartreader/blob/master/src/process.py"
click node_summarizer "https://github.com/el-bid/smartreader/blob/master/src/sumytest.py"
click node_spellcheck "https://github.com/el-bid/smartreader/blob/master/src/spell_check.py"
click node_converter "https://github.com/el-bid/smartreader/blob/master/src/convert_dataset.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_webapp,node_ui toneBlue
class node_model_worker,node_search,node_google,node_model_builder,node_tokenizer toneAmber
class node_summary_worker,node_dataset_reader,node_document,node_analyzer,node_summarizer,node_spellcheck,node_converter toneMint
class node_db,node_model_files toneRose
class node_researcher toneIndigo
```

## Components

### Research interface

- [Flask application](https://github.com/el-bid/smartreader/blob/master/main.py): `main.py`
- [Web interfaces](https://github.com/el-bid/smartreader/tree/master/templates): `templates`

### Model creation

- [Model worker](https://github.com/el-bid/smartreader/blob/master/processor.py): `processor.py`
- [Web search and extraction](https://github.com/el-bid/smartreader/blob/master/src/search_data.py): `src/search_data.py`
- [Google search client](https://github.com/el-bid/smartreader/blob/master/_google_/__init__.py): `_google_/__init__.py`
- [TF-IDF model builder](https://github.com/el-bid/smartreader/blob/master/src/create_model.py): `src/create_model.py`
- [Lemma tokenizer](https://github.com/el-bid/smartreader/blob/master/src/LemmaTokenizer.py): `src/LemmaTokenizer.py`

### Corpus analysis

- [Summary worker](https://github.com/el-bid/smartreader/blob/master/summary_processor.py): `summary_processor.py`
- [Dataset reader](https://github.com/el-bid/smartreader/blob/master/src/dataset_reader.py): `src/dataset_reader.py`
- [Document and paragraphs](https://github.com/el-bid/smartreader/blob/master/src/document.py): `src/document.py`
- [Paragraph scoring and extraction](https://github.com/el-bid/smartreader/blob/master/src/process.py): `src/process.py`
- [Sentence summarization](https://github.com/el-bid/smartreader/blob/master/src/sumytest.py): `src/sumytest.py`
- [Spell checking](https://github.com/el-bid/smartreader/blob/master/src/spell_check.py): `src/spell_check.py`
- [Annotated document output](https://github.com/el-bid/smartreader/blob/master/src/convert_dataset.py): `src/convert_dataset.py`

### Jobs and results

- [MongoDB job store](https://github.com/el-bid/smartreader/blob/master/src/database_connectivity.py): `src/database_connectivity.py`
- [Saved models](https://github.com/el-bid/smartreader/tree/master/models): `models`

### Other components

- **Researcher**

## Connections

- Researcher → Flask application: submits topics and corpus (evidence: `README.md`)
- Flask application → Web interfaces: renders pages (evidence: `main.py`)
- Flask application → MongoDB job store: queues and reads jobs (evidence: `main.py`)
- Model worker → MongoDB job store: polls and updates (evidence: `processor.py`)
- Model worker → Web search and extraction: retrieves topic text (evidence: `processor.py`)
- Web search and extraction → Google search client: searches for URLs (evidence: `src/search_data.py`)
- Web search and extraction → TF-IDF model builder: supplies topic text (evidence: `processor.py`)
- TF-IDF model builder → Lemma tokenizer: configures vectorizer (evidence: `src/create_model.py`)
- TF-IDF model builder → Saved models: saves model
- Flask application → Saved models: accepts selected model (evidence: `main.py`)
- Flask application → Summary worker: queues corpus job (evidence: `README.md`)
- Summary worker → MongoDB job store: polls and updates (evidence: `summary_processor.py`)
- Summary worker → Paragraph scoring and extraction: creates summaries (evidence: `summary_processor.py`)
- Paragraph scoring and extraction → Dataset reader: loads corpus (evidence: `src/process.py`)
- Dataset reader → Document and paragraphs: creates documents (evidence: `src/dataset_reader.py`)
- Paragraph scoring and extraction → TF-IDF model builder: loads saved model (evidence: `src/process.py`)
- Paragraph scoring and extraction → Sentence summarization: selects sentences (evidence: `src/process.py`)
- Paragraph scoring and extraction → Spell checking: checks selected text (evidence: `src/process.py`)
- Summary worker → Annotated document output: annotates source documents (evidence: `summary_processor.py`)
- Researcher → Flask application: views or downloads results (evidence: `README.md`)
