panolifer/reviewclassifier

Purpose and entry points

ReviewClassifier is a Python batch analytics pipeline for Russian film reviews: it segments reviews, identifies technology-relevant fragments, classifies sentiment/stance (optimism, skepticism, mixed), and produces aggregate research tables and heatmaps. Primary entry points are run_pipeline.py for the baseline workflow and run_pipeline_with_pytorch.py for baseline processing plus PyTorch training. Focused training/prediction entry points include run_pytorch_training.py, run_pytorch_polar_training.py, and install_5000_labels.py.

Data lifecycle and boundaries

The repository is file-oriented, with Excel/CSV files as stage contracts rather than a database or service API. Raw KinoPoisk review exports enter through data/raw/kinopoisk_reviews_full.xlsx and data/raw/kinopoisk_reviews_for_ML.xlsx. Processing creates complete and technology-filtered fragment datasets at data/processed/fragments_all.csv and data/processed/fragments_tech.csv. Final predictions, reports, and aggregates are persisted under data/results/; visual artifacts go to data/figures/.

Preparation and relevance selection

Dataset inspection validates incoming review data before downstream work in src/inspect_dataset.py. Fragment construction/segmentation is handled by src/prepare_fragments.py, producing the complete fragment corpus. Weak-label or relevance-oriented preparation is separated in src/build_weak_labels.py; label selection behavior is centralized in src/labels.py. The key boundary is from unfiltered review fragments to the technology-relevant subset consumed by classifiers.

Annotation and training

Manual coding begins with make_annotation_sample.py / src/make_annotation_sample.py, generating data/labels/annotation_sample_for_manual_coding.xlsx; completed labels are read from data/labels/labeled_fragments.xlsx. Training preferentially uses the larger data/labels/labeled_fragments_5000.xlsx. The baseline model is trained in src/train_baseline.py and stored as models/tfidf_logreg.joblib, implying a TF-IDF plus logistic-regression path. This label dataset is the principal human/weak-supervision boundary.

Classification alternatives

Baseline inference is implemented by src/predict_fragments.py and writes data/results/classification_fragment_level.csv, which drives standard aggregation. An independent PyTorch path uses src/train_pytorch_classifier.py and src/predict_fragments_pytorch.py; it trains an EmbeddingBag classifier and stores model/vocabulary artifacts at models/pytorch_embeddingbag_classifier.pt and models/pytorch_vocab.json. Its output, data/results/classification_fragment_level_pytorch.csv, is intentionally isolated from the main pipeline. A binary optimism-versus-skepticism variant is defined in src/train_pytorch_polar_classifier.py and saved separately.

Aggregation, interpretation, and outputs

Classified fragment records flow into src/aggregate_results.py, producing reviewer-, film-, and year-level analysis such as data/results/reviewer_tables.xlsx, data/results/yearly_polarity.csv, and data/results/film_polarity.csv. src/make_heatmaps.py converts aggregate polarity tables into figure outputs. src/lexical_analysis.py produces class-oriented lexeme results, including data/results/lexemes_optimism.csv and data/results/lexemes_skepticism.csv.

Runtime and external context

The system runs locally under Python with dependencies pinned in requirements.txt; PyTorch is a material optional runtime for neural training and inference. There is no deployed application or persistent infrastructure. Historical collection from KinoPoisk is kept outside the active pipeline in legacy_collection_scripts/, including offline extraction at legacy_collection_scripts/export_reviews_offline.py and archived source HTML under legacy_collection_scripts/html_dump/; active analysis consumes the exported Excel data instead.