How el-bid/smartreader works
SmartReader helps researchers build topic-and-keyword models from Google search results, apply those models to uploaded text corpora, and inspect or download ranked summaries and extracted information. The workflow runs through a Flask interface and two polling workers, with MongoDB tracking queued jobs and local files holding models and results. The README documents the major user-facing stages; some implementation details for UI templates and several indexed modules were not sampled, so their internal links are omitted.
Main entry points: main.py Flask application · process.py Paragraph scoring and extraction · summary_processor.py Summary worker · create_model.py TF-IDF model builder
Components 17
Research interface
- Flask application
main.py - Web interfaces
templates
Model creation
- Model worker
processor.py - Web search and extraction
src/search_data.py - Google search client
_google_/__init__.py - TF-IDF model builder
src/create_model.py - Lemma tokenizer
src/LemmaTokenizer.py
Corpus analysis
- Summary worker
summary_processor.py - Dataset reader
src/dataset_reader.py - Document and paragraphs
src/document.py - Paragraph scoring and extraction
src/process.py - Sentence summarization
src/sumytest.py - Spell checking
src/spell_check.py - Annotated document output
src/convert_dataset.py
Jobs and results
- MongoDB job store
src/database_connectivity.py - Saved models
models
Other components
- Researcher
Connections 20
- Researcher → Flask application: submits topics and corpus
README.md - Flask application → Web interfaces: renders pages
main.py - Flask application → MongoDB job store: queues and reads jobs
main.py - Model worker → MongoDB job store: polls and updates
processor.py - Model worker → Web search and extraction: retrieves topic text
processor.py - Web search and extraction → Google search client: searches for URLs
src/search_data.py - Web search and extraction → TF-IDF model builder: supplies topic text
processor.py - TF-IDF model builder → Lemma tokenizer: configures vectorizer
src/create_model.py - TF-IDF model builder → Saved models: saves model
- Flask application → Saved models: accepts selected model
main.py - Flask application → Summary worker: queues corpus job
README.md - Summary worker → MongoDB job store: polls and updates
summary_processor.py - Summary worker → Paragraph scoring and extraction: creates summaries
summary_processor.py - Paragraph scoring and extraction → Dataset reader: loads corpus
src/process.py - Dataset reader → Document and paragraphs: creates documents
src/dataset_reader.py - Paragraph scoring and extraction → TF-IDF model builder: loads saved model
src/process.py - Paragraph scoring and extraction → Sentence summarization: selects sentences
src/process.py - Paragraph scoring and extraction → Spell checking: checks selected text
src/process.py - Summary worker → Annotated document output: annotates source documents
summary_processor.py - Researcher → Flask application: views or downloads results
README.md