# panolifer/reviewclassifier architecture

> Architecture diagram of the GitHub repository panolifer/reviewclassifier, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/panolifer/reviewclassifier
- Repository: https://github.com/panolifer/reviewclassifier

## Overview

#### Purpose and entry points
ReviewClassifier is a Python batch analytics pipeline for Russian film reviews: it segments reviews, identifies technology-relevant fragments, classifies sentiment/stance (`optimism`, `skepticism`, `mixed`), and produces aggregate research tables and heatmaps. Primary entry points are `run_pipeline.py` for the baseline workflow and `run_pipeline_with_pytorch.py` for baseline processing plus PyTorch training. Focused training/prediction entry points include `run_pytorch_training.py`, `run_pytorch_polar_training.py`, and `install_5000_labels.py`.

#### Data lifecycle and boundaries
The repository is file-oriented, with Excel/CSV files as stage contracts rather than a database or service API. Raw KinoPoisk review exports enter through `data/raw/kinopoisk_reviews_full.xlsx` and `data/raw/kinopoisk_reviews_for_ML.xlsx`. Processing creates complete and technology-filtered fragment datasets at `data/processed/fragments_all.csv` and `data/processed/fragments_tech.csv`. Final predictions, reports, and aggregates are persisted under `data/results/`; visual artifacts go to `data/figures/`.

#### Preparation and relevance selection
Dataset inspection validates incoming review data before downstream work in `src/inspect_dataset.py`. Fragment construction/segmentation is handled by `src/prepare_fragments.py`, producing the complete fragment corpus. Weak-label or relevance-oriented preparation is separated in `src/build_weak_labels.py`; label selection behavior is centralized in `src/labels.py`. The key boundary is from unfiltered review fragments to the technology-relevant subset consumed by classifiers.

#### Annotation and training
Manual coding begins with `make_annotation_sample.py` / `src/make_annotation_sample.py`, generating `data/labels/annotation_sample_for_manual_coding.xlsx`; completed labels are read from `data/labels/labeled_fragments.xlsx`. Training preferentially uses the larger `data/labels/labeled_fragments_5000.xlsx`. The baseline model is trained in `src/train_baseline.py` and stored as `models/tfidf_logreg.joblib`, implying a TF-IDF plus logistic-regression path. This label dataset is the principal human/weak-supervision boundary.

#### Classification alternatives
Baseline inference is implemented by `src/predict_fragments.py` and writes `data/results/classification_fragment_level.csv`, which drives standard aggregation. An independent PyTorch path uses `src/train_pytorch_classifier.py` and `src/predict_fragments_pytorch.py`; it trains an `EmbeddingBag` classifier and stores model/vocabulary artifacts at `models/pytorch_embeddingbag_classifier.pt` and `models/pytorch_vocab.json`. Its output, `data/results/classification_fragment_level_pytorch.csv`, is intentionally isolated from the main pipeline. A binary optimism-versus-skepticism variant is defined in `src/train_pytorch_polar_classifier.py` and saved separately.

#### Aggregation, interpretation, and outputs
Classified fragment records flow into `src/aggregate_results.py`, producing reviewer-, film-, and year-level analysis such as `data/results/reviewer_tables.xlsx`, `data/results/yearly_polarity.csv`, and `data/results/film_polarity.csv`. `src/make_heatmaps.py` converts aggregate polarity tables into figure outputs. `src/lexical_analysis.py` produces class-oriented lexeme results, including `data/results/lexemes_optimism.csv` and `data/results/lexemes_skepticism.csv`.

#### Runtime and external context
The system runs locally under Python with dependencies pinned in `requirements.txt`; PyTorch is a material optional runtime for neural training and inference. There is no deployed application or persistent infrastructure. Historical collection from KinoPoisk is kept outside the active pipeline in `legacy_collection_scripts/`, including offline extraction at `legacy_collection_scripts/export_reviews_offline.py` and archived source HTML under `legacy_collection_scripts/html_dump/`; active analysis consumes the exported Excel data instead.

Main entry points: [`data/processed/fragments_all.csv`](https://github.com/panolifer/reviewclassifier/blob/main/data/processed/fragments_all.csv) (All fragments), [`data/processed/fragments_tech.csv`](https://github.com/panolifer/reviewclassifier/blob/main/data/processed/fragments_tech.csv) (Technology fragments), [`data/labels/labeled_fragments_5000.xlsx`](https://github.com/panolifer/reviewclassifier/blob/main/data/labels/labeled_fragments_5000.xlsx) (Labeled fragments), [`data/results/classification_fragment_level.csv`](https://github.com/panolifer/reviewclassifier/blob/main/data/results/classification_fragment_level.csv) (Baseline classifications)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Preparation"]
  node_raw_full["Raw review export<br/>Excel input"]
  node_inspection["Dataset inspection<br/>Python stage<br/>[inspect_dataset.py]"]
  node_fragment_prep["Fragment preparation<br/>Python stage"]
  node_all_fragments["All fragments<br/>CSV contract<br/>[fragments_all.csv]"]
  node_weak_labels["Relevance selection<br/>Python stage"]
  node_tech_fragments["Technology fragments<br/>CSV contract<br/>[fragments_tech.csv]"]
  node_annotation_sample["Manual annotation sample<br/>Python stage"]
  node_human_labels["Labeled fragments<br/>Excel labels"]
end

subgraph group_baseline["Baseline workflow"]
  node_baseline_train["Baseline training<br/>TF-IDF + logistic regression<br/>[train_baseline.py]"]
  node_baseline_model["Baseline model<br/>Joblib artifact"]
  node_baseline_predict["Baseline prediction<br/>Python stage"]
  node_baseline_results["Baseline classifications<br/>CSV results"]
end

subgraph group_pytorch["PyTorch alternative"]
  node_pytorch_train["EmbeddingBag training<br/>PyTorch stage"]
  node_pytorch_model["PyTorch model<br/>PyTorch artifact"]
  node_pytorch_predict["PyTorch prediction<br/>PyTorch stage"]
  node_pytorch_results["PyTorch classifications<br/>CSV results"]
end

subgraph group_analysis["Analysis outputs"]
  node_aggregation["Result aggregation<br/>Python stage"]
  node_aggregate_tables["Aggregate polarity tables<br/>CSV/XLSX results"]
  node_heatmaps["Heatmap generation<br/>Python stage<br/>[make_heatmaps.py]"]
  node_figures["Figures<br/>Visual outputs"]
  node_lexical_analysis["Lexical analysis<br/>Python stage"]
end

node_legacy_export["Legacy KinoPoisk export<br/>Offline collection"]

node_legacy_export -.->|"historical export"| node_raw_full
node_raw_full -->|"validate"| node_inspection
node_inspection -->|"approved reviews"| node_fragment_prep
node_fragment_prep -->|"writes"| node_all_fragments
node_all_fragments -->|"select relevance"| node_weak_labels
node_weak_labels -->|"writes"| node_tech_fragments
node_all_fragments -->|"sample"| node_annotation_sample
node_annotation_sample -->|"manual coding"| node_human_labels
node_human_labels -->|"training labels"| node_baseline_train
node_baseline_train -->|"stores"| node_baseline_model
node_tech_fragments -->|"classify"| node_baseline_predict
node_baseline_model -->|"load model"| node_baseline_predict
node_baseline_predict -->|"writes"| node_baseline_results
node_human_labels -->|"training labels"| node_pytorch_train
node_pytorch_train -->|"stores"| node_pytorch_model
node_tech_fragments -->|"classify"| node_pytorch_predict
node_pytorch_model -->|"load model"| node_pytorch_predict
node_pytorch_predict -->|"writes"| node_pytorch_results
node_baseline_results -->|"main input"| node_aggregation
node_aggregation -->|"writes"| node_aggregate_tables
node_aggregate_tables -->|"polarity tables"| node_heatmaps
node_heatmaps -->|"renders"| node_figures
node_baseline_results -->|"classified text"| node_lexical_analysis

click node_raw_full "https://github.com/panolifer/reviewclassifier/blob/main/data/raw/kinopoisk_reviews_full.xlsx"
click node_inspection "https://github.com/panolifer/reviewclassifier/blob/main/src/inspect_dataset.py"
click node_fragment_prep "https://github.com/panolifer/reviewclassifier/blob/main/src/prepare_fragments.py"
click node_all_fragments "https://github.com/panolifer/reviewclassifier/blob/main/data/processed/fragments_all.csv"
click node_weak_labels "https://github.com/panolifer/reviewclassifier/blob/main/src/build_weak_labels.py"
click node_tech_fragments "https://github.com/panolifer/reviewclassifier/blob/main/data/processed/fragments_tech.csv"
click node_annotation_sample "https://github.com/panolifer/reviewclassifier/blob/main/src/make_annotation_sample.py"
click node_human_labels "https://github.com/panolifer/reviewclassifier/blob/main/data/labels/labeled_fragments_5000.xlsx"
click node_baseline_train "https://github.com/panolifer/reviewclassifier/blob/main/src/train_baseline.py"
click node_baseline_model "https://github.com/panolifer/reviewclassifier/blob/main/models/tfidf_logreg.joblib"
click node_baseline_predict "https://github.com/panolifer/reviewclassifier/blob/main/src/predict_fragments.py"
click node_baseline_results "https://github.com/panolifer/reviewclassifier/blob/main/data/results/classification_fragment_level.csv"
click node_pytorch_train "https://github.com/panolifer/reviewclassifier/blob/main/src/train_pytorch_classifier.py"
click node_pytorch_model "https://github.com/panolifer/reviewclassifier/blob/main/models/pytorch_embeddingbag_classifier.pt"
click node_pytorch_predict "https://github.com/panolifer/reviewclassifier/blob/main/src/predict_fragments_pytorch.py"
click node_pytorch_results "https://github.com/panolifer/reviewclassifier/blob/main/data/results/classification_fragment_level_pytorch.csv"
click node_aggregation "https://github.com/panolifer/reviewclassifier/blob/main/src/aggregate_results.py"
click node_aggregate_tables "https://github.com/panolifer/reviewclassifier/blob/main/data/results/yearly_polarity.csv"
click node_heatmaps "https://github.com/panolifer/reviewclassifier/blob/main/src/make_heatmaps.py"
click node_lexical_analysis "https://github.com/panolifer/reviewclassifier/blob/main/src/lexical_analysis.py"
click node_legacy_export "https://github.com/panolifer/reviewclassifier/blob/main/legacy_collection_scripts/export_reviews_offline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_full,node_inspection,node_fragment_prep,node_all_fragments,node_weak_labels,node_tech_fragments,node_annotation_sample,node_human_labels toneBlue
class node_baseline_train,node_baseline_model,node_baseline_predict,node_baseline_results toneAmber
class node_pytorch_train,node_pytorch_model,node_pytorch_predict,node_pytorch_results toneMint
class node_aggregation,node_aggregate_tables,node_heatmaps,node_figures,node_lexical_analysis toneRose
class node_legacy_export toneNeutral
```

## Components

### Preparation

- [Raw review export](https://github.com/panolifer/reviewclassifier/blob/main/data/raw/kinopoisk_reviews_full.xlsx): Excel input · `data/raw/kinopoisk_reviews_full.xlsx`
- [Dataset inspection](https://github.com/panolifer/reviewclassifier/blob/main/src/inspect_dataset.py): Python stage · Validates incoming review data. · `src/inspect_dataset.py`
- [Fragment preparation](https://github.com/panolifer/reviewclassifier/blob/main/src/prepare_fragments.py): Python stage · Segments reviews into a complete fragment corpus. · `src/prepare_fragments.py`
- [All fragments](https://github.com/panolifer/reviewclassifier/blob/main/data/processed/fragments_all.csv): CSV contract · `data/processed/fragments_all.csv`
- [Relevance selection](https://github.com/panolifer/reviewclassifier/blob/main/src/build_weak_labels.py): Python stage · Builds weak labels and selects technology-relevant fragments. · `src/build_weak_labels.py`
- [Technology fragments](https://github.com/panolifer/reviewclassifier/blob/main/data/processed/fragments_tech.csv): CSV contract · `data/processed/fragments_tech.csv`
- [Manual annotation sample](https://github.com/panolifer/reviewclassifier/blob/main/src/make_annotation_sample.py): Python stage · `src/make_annotation_sample.py`
- [Labeled fragments](https://github.com/panolifer/reviewclassifier/blob/main/data/labels/labeled_fragments_5000.xlsx): Excel labels · Human and larger training label sets. · `data/labels/labeled_fragments_5000.xlsx`

### Baseline workflow

- [Baseline training](https://github.com/panolifer/reviewclassifier/blob/main/src/train_baseline.py): TF-IDF + logistic regression · `src/train_baseline.py`
- [Baseline model](https://github.com/panolifer/reviewclassifier/blob/main/models/tfidf_logreg.joblib): Joblib artifact · `models/tfidf_logreg.joblib`
- [Baseline prediction](https://github.com/panolifer/reviewclassifier/blob/main/src/predict_fragments.py): Python stage · `src/predict_fragments.py`
- [Baseline classifications](https://github.com/panolifer/reviewclassifier/blob/main/data/results/classification_fragment_level.csv): CSV results · `data/results/classification_fragment_level.csv`

### PyTorch alternative

- [EmbeddingBag training](https://github.com/panolifer/reviewclassifier/blob/main/src/train_pytorch_classifier.py): PyTorch stage · `src/train_pytorch_classifier.py`
- [PyTorch model](https://github.com/panolifer/reviewclassifier/blob/main/models/pytorch_embeddingbag_classifier.pt): PyTorch artifact · Vocabulary is stored alongside the classifier. · `models/pytorch_embeddingbag_classifier.pt`
- [PyTorch prediction](https://github.com/panolifer/reviewclassifier/blob/main/src/predict_fragments_pytorch.py): PyTorch stage · `src/predict_fragments_pytorch.py`
- [PyTorch classifications](https://github.com/panolifer/reviewclassifier/blob/main/data/results/classification_fragment_level_pytorch.csv): CSV results · Isolated from the main aggregation path. · `data/results/classification_fragment_level_pytorch.csv`

### Analysis outputs

- [Result aggregation](https://github.com/panolifer/reviewclassifier/blob/main/src/aggregate_results.py): Python stage · Creates reviewer, film, and year-level polarity analysis. · `src/aggregate_results.py`
- [Aggregate polarity tables](https://github.com/panolifer/reviewclassifier/blob/main/data/results/yearly_polarity.csv): CSV/XLSX results · `data/results/yearly_polarity.csv`
- [Heatmap generation](https://github.com/panolifer/reviewclassifier/blob/main/src/make_heatmaps.py): Python stage · `src/make_heatmaps.py`
- **Figures**: Visual outputs
- [Lexical analysis](https://github.com/panolifer/reviewclassifier/blob/main/src/lexical_analysis.py): Python stage · Produces class-oriented lexeme tables. · `src/lexical_analysis.py`

### Other components

- [Legacy KinoPoisk export](https://github.com/panolifer/reviewclassifier/blob/main/legacy_collection_scripts/export_reviews_offline.py): Offline collection · Historical collection outside the active pipeline. · `legacy_collection_scripts/export_reviews_offline.py`

## Connections

- Legacy KinoPoisk export → Raw review export: historical export
- Raw review export → Dataset inspection: validate
- Dataset inspection → Fragment preparation: approved reviews
- Fragment preparation → All fragments: writes
- All fragments → Relevance selection: select relevance
- Relevance selection → Technology fragments: writes
- All fragments → Manual annotation sample: sample
- Manual annotation sample → Labeled fragments: manual coding
- Labeled fragments → Baseline training: training labels
- Baseline training → Baseline model: stores
- Technology fragments → Baseline prediction: classify
- Baseline model → Baseline prediction: load model
- Baseline prediction → Baseline classifications: writes
- Labeled fragments → EmbeddingBag training: training labels
- EmbeddingBag training → PyTorch model: stores
- Technology fragments → PyTorch prediction: classify
- PyTorch model → PyTorch prediction: load model
- PyTorch prediction → PyTorch classifications: writes
- Baseline classifications → Result aggregation: main input
- Result aggregation → Aggregate polarity tables: writes
- Aggregate polarity tables → Heatmap generation: polarity tables
- Heatmap generation → Figures: renders
- Baseline classifications → Lexical analysis: classified text