How bvijayabhaskarreddy/lexicalconnections works
1. Project purpose
- This repository is a batch analytics pipeline for studying how a news caption relates to its parent article.
- The core output is a set of caption/article relationship labels and quantitative features used for statistical analysis and publication-ready tables/figures.
2. High-level architecture
- The repo is organized into three clear layers:
- Data layer: data/ holds the input sample and intermediate CSV outputs.
- Processing layer: code/ contains the NLP and feature-extraction pipeline.
- Reporting layer: results/ stores generated figures and tables.
- There is no server, UI, or service boundary; this is an offline, reproducible research workflow.
3. Primary data flow
- The pipeline starts from a gzipped CSV sample of 30,000 caption-article pairs: data/sample_30k.csv.gz.
- run_pipeline.py appears to orchestrate the end-to-end processing, producing a results CSV that carries computed labels and features forward.
- generate_figures.py then consumes that processed dataset to create charts and summary artifacts in results/figures/ and results/tables/.
4. Core NLP preprocessing boundary
- preprocessing.py is the normalization and text-cleaning stage before any comparison is made.
- This is where raw article/caption text is likely prepared for downstream NLP tasks such as tokenization, sentence handling, and feature extraction.
- This stage is important because all later similarity, entity, and structural measures depend on consistent text preprocessing.
5. Semantic similarity subsystem
- similarity.py is the semantic comparison layer.
- The README indicates it combines SBERT embeddings (all-MiniLM-L6-v2) with traditional lexical measures such as TF-IDF and Jaccard similarity.
- This subsystem provides the “how close are these texts?” signal used to distinguish extractive captions from more independent ones.
6. Entity alignment subsystem
- entity_alignment.py focuses on named-entity overlap and correspondence between caption and article.
- The key entity types are PERSON, GPE, ORG, and DATE, with fuzzy matching used to handle slight variations.
- This creates a second semantic boundary: not just overall similarity, but whether the same named entities are referenced across the two texts.
7. Structural feature subsystem
- structural_features.py extracts linguistic form features rather than meaning-based features.
- The README points to clausal complexity, dependency depth, and POS distributions.
- These features help characterize whether a caption is terse, descriptive, syntactically complex, or context-expanding.
8. Classification layer
- caption_classifier.py is the decision layer that maps the engineered features into the caption relationship categories described in the README:
- Extractive
- Descriptive
- Expansive
- Independent
- In architecture terms, this is the point where feature extraction becomes an interpretable analytical label.
9. Sampling and experiment management
- downsample.py suggests a preprocessing utility for creating a manageable stratified subset from a larger corpus.
- This is a data-preparation boundary rather than a core analytic step, but it matters because the project’s reported results are based on a 30k sample from VisualNews (Guardian and BBC).
10. Reporting and publication outputs
- generate_figures.py is the post-processing/reporting stage.
- The results/tables/ directory already contains descriptive statistics, statistical tests, and regression outputs in both CSV and LaTeX formats, indicating the pipeline is designed to support academic reporting.
- results/figures/ contains the visualization outputs generated from the processed dataset.
11. Technology stack that materially shapes the design
- Python 3.10+ is the runtime.
- spaCy provides linguistic parsing and NER, including the transformer and small English models referenced in the README.
- Sentence-Transformers provides embedding-based semantic comparison.
- pandas, scikit-learn, and statsmodels support tabular processing, similarity/statistical analysis, and regression.
- rapidfuzz supports approximate entity matching.
- matplotlib and seaborn handle visual reporting.
12. What an architecture diagram should show
- A single linear analytical pipeline:
- raw sampled CSV → preprocessing → similarity/entity/structural feature extraction → classification/statistical analysis → tables and figures.
- Around that pipeline, show three external dependencies:
- spaCy NLP models,
- SBERT embeddings,
- statistical/plotting libraries.
- The main boundary is between input data, feature computation/classification, and report generation.
Main entry points: preprocessing.py Preprocess · caption_classifier.py Classifier · similarity.py Similarity · generate_figures.py Generate figures
Components 17
Data layer
Input sample and intermediate/output datasets.
- Sample CSV · Gzipped caption-article sample used as pipeline input.
data/sample_30k.csv.gz - Processed data · Intermediate results carrying computed labels and features forward.
Processing layer
NLP preprocessing, feature extraction, and classification pipeline.
- Downsample · Creates a manageable stratified subset from a larger corpus.
code/downsample.py - Run pipeline · Coordinates the end-to-end analytics workflow.
code/run_pipeline.py - Preprocess · Cleans and normalizes caption and article text before comparison.
code/preprocessing.py - Similarity · Computes SBERT, TF-IDF, and Jaccard similarity signals.
code/similarity.py - Entity alignment · Measures overlap and fuzzy correspondence of named entities.
code/entity_alignment.py - Structure features · Extracts clausal, dependency, and POS-based form features.
code/structural_features.py - Classifier · Maps engineered features to caption relationship categories.
code/caption_classifier.py
Reporting layer
Generated tables and figures for analysis and publication.
- Generate figures · Consumes processed data to create charts and summary artifacts.
code/generate_figures.py - Tables · Descriptive statistics, tests, and regression tables.
results/tables - Figures · Generated plots and visual summaries.
results/figures
External dependencies
Core libraries and models that shape the pipeline.
- spaCy · Provides parsing and named-entity recognition.
- SBERT · Sentence-Transformers model used for semantic similarity.
- Stats tools · pandas, scikit-learn, and statsmodels support tabular analysis and regression.
- Plotting libs · matplotlib and seaborn support figure generation.
- RapidFuzz · Supports approximate entity matching.
Connections 20
- Sample CSV → Run pipeline: input · Raw sampled pairs enter the pipeline.
- Downsample → Sample CSV: prepares · Optional sampling step used to build the working dataset.
- Run pipeline → Preprocess: orchestrates · Pipeline begins with text normalization.
- Preprocess → Similarity: feeds · Cleaned text is compared semantically.
- Preprocess → Entity alignment: feeds · Normalized text is used for entity overlap checks.
- Preprocess → Structure features: feeds · Preprocessed text supports structural analysis.
- Similarity → Classifier: signal · Similarity metrics contribute to classification.
- Entity alignment → Classifier: signal · Entity overlap contributes to classification.
- Structure features → Classifier: signal · Syntactic features contribute to classification.
- Classifier → Processed data: writes · Labels and features are persisted for analysis.
- Run pipeline → Processed data: produces · End-to-end execution emits the processed dataset.
- Processed data → Generate figures: input · Reporting stage consumes the analysis dataset.
- Generate figures → Tables: writes · Creates statistical tables and summaries.
- Generate figures → Figures: writes · Creates publication figures.
- Preprocess → spaCy: uses · Linguistic parsing and NER support.
- Entity alignment → RapidFuzz: uses · Approximate matching for entity correspondence.
- Similarity → SBERT: uses · Embedding-based semantic comparison.
- Similarity → Stats tools: uses · Tabular similarity analysis and metrics.
- Classifier → Stats tools: uses · Classification and statistical modeling support.
- Generate figures → Plotting libs: uses · Visualization libraries render outputs.