# bvijayabhaskarreddy/lexicalconnections architecture

> Architecture diagram of the GitHub repository bvijayabhaskarreddy/lexicalconnections, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bvijayabhaskarreddy/lexicalconnections
- Repository: https://github.com/bvijayabhaskarreddy/lexicalconnections

## Overview

1. **Project purpose**
   - This repository is a batch analytics pipeline for studying how a news caption relates to its parent article.
   - The core output is a set of caption/article relationship labels and quantitative features used for statistical analysis and publication-ready tables/figures.

2. **High-level architecture**
   - The repo is organized into three clear layers:
     - **Data layer:** `data/` holds the input sample and intermediate CSV outputs.
     - **Processing layer:** `code/` contains the NLP and feature-extraction pipeline.
     - **Reporting layer:** `results/` stores generated figures and tables.
   - There is no server, UI, or service boundary; this is an offline, reproducible research workflow.

3. **Primary data flow**
   - The pipeline starts from a gzipped CSV sample of 30,000 caption-article pairs: `data/sample_30k.csv.gz`.
   - `run_pipeline.py` appears to orchestrate the end-to-end processing, producing a results CSV that carries computed labels and features forward.
   - `generate_figures.py` then consumes that processed dataset to create charts and summary artifacts in `results/figures/` and `results/tables/`.

4. **Core NLP preprocessing boundary**
   - `preprocessing.py` is the normalization and text-cleaning stage before any comparison is made.
   - This is where raw article/caption text is likely prepared for downstream NLP tasks such as tokenization, sentence handling, and feature extraction.
   - This stage is important because all later similarity, entity, and structural measures depend on consistent text preprocessing.

5. **Semantic similarity subsystem**
   - `similarity.py` is the semantic comparison layer.
   - The README indicates it combines **SBERT** embeddings (`all-MiniLM-L6-v2`) with traditional lexical measures such as **TF-IDF** and **Jaccard** similarity.
   - This subsystem provides the “how close are these texts?” signal used to distinguish extractive captions from more independent ones.

6. **Entity alignment subsystem**
   - `entity_alignment.py` focuses on named-entity overlap and correspondence between caption and article.
   - The key entity types are **PERSON, GPE, ORG, and DATE**, with fuzzy matching used to handle slight variations.
   - This creates a second semantic boundary: not just overall similarity, but whether the same named entities are referenced across the two texts.

7. **Structural feature subsystem**
   - `structural_features.py` extracts linguistic form features rather than meaning-based features.
   - The README points to clausal complexity, dependency depth, and POS distributions.
   - These features help characterize whether a caption is terse, descriptive, syntactically complex, or context-expanding.

8. **Classification layer**
   - `caption_classifier.py` is the decision layer that maps the engineered features into the caption relationship categories described in the README:
     - Extractive
     - Descriptive
     - Expansive
     - Independent
   - In architecture terms, this is the point where feature extraction becomes an interpretable analytical label.

9. **Sampling and experiment management**
   - `downsample.py` suggests a preprocessing utility for creating a manageable stratified subset from a larger corpus.
   - This is a data-preparation boundary rather than a core analytic step, but it matters because the project’s reported results are based on a 30k sample from VisualNews (Guardian and BBC).

10. **Reporting and publication outputs**
   - `generate_figures.py` is the post-processing/reporting stage.
   - The `results/tables/` directory already contains descriptive statistics, statistical tests, and regression outputs in both CSV and LaTeX formats, indicating the pipeline is designed to support academic reporting.
   - `results/figures/` contains the visualization outputs generated from the processed dataset.

11. **Technology stack that materially shapes the design**
   - Python 3.10+ is the runtime.
   - **spaCy** provides linguistic parsing and NER, including the transformer and small English models referenced in the README.
   - **Sentence-Transformers** provides embedding-based semantic comparison.
   - **pandas**, **scikit-learn**, and **statsmodels** support tabular processing, similarity/statistical analysis, and regression.
   - **rapidfuzz** supports approximate entity matching.
   - **matplotlib** and **seaborn** handle visual reporting.

12. **What an architecture diagram should show**
   - A single linear analytical pipeline:
     - raw sampled CSV → preprocessing → similarity/entity/structural feature extraction → classification/statistical analysis → tables and figures.
   - Around that pipeline, show three external dependencies:
     - spaCy NLP models,
     - SBERT embeddings,
     - statistical/plotting libraries.
   - The main boundary is between **input data**, **feature computation/classification**, and **report generation**.

Main entry points: [`code/preprocessing.py`](https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/preprocessing.py) (Preprocess), [`code/caption_classifier.py`](https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/caption_classifier.py) (Classifier), [`code/similarity.py`](https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/similarity.py) (Similarity), [`code/generate_figures.py`](https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/generate_figures.py) (Generate figures)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data_layer["Data layer"]
  node_data_sample[("Sample CSV<br/>input dataset<br/>[sample_30k.csv.gz]")]
  node_processed_output[("Processed data<br/>analysis dataset")]
end

subgraph group_processing_layer["Processing layer"]
  node_downsample["Downsample<br/>sampling utility<br/>[downsample.py]"]
  node_run_pipeline["Run pipeline<br/>orchestrator<br/>[run_pipeline.py]"]
  node_preprocessing["Preprocess<br/>text normalization<br/>[preprocessing.py]"]
  node_similarity["Similarity<br/>semantic comparison<br/>[similarity.py]"]
  node_entity_alignment["Entity alignment<br/>NER matching"]
  node_structural_features["Structure features<br/>linguistic features"]
  node_caption_classifier["Classifier<br/>labeling logic"]
end

subgraph group_reporting_layer["Reporting layer"]
  node_generate_figures["Generate figures<br/>reporting pipeline"]
  node_tables["Tables<br/>publication outputs"]
  node_figures["Figures<br/>visual outputs"]
end

subgraph group_dependencies["External dependencies"]
  node_spacy(("spaCy<br/>nlp library"))
  node_sbert(("SBERT<br/>embedding model"))
  node_stats_libs(("Stats tools<br/>analysis libraries"))
  node_viz_libs(("Plotting libs<br/>visualization libraries"))
  node_rapidfuzz(("RapidFuzz<br/>fuzzy matching library"))
end

node_data_sample -->|"input"| node_run_pipeline
node_downsample -.->|"prepares"| node_data_sample
node_run_pipeline -->|"orchestrates"| node_preprocessing
node_preprocessing -->|"feeds"| node_similarity
node_preprocessing -->|"feeds"| node_entity_alignment
node_preprocessing -->|"feeds"| node_structural_features
node_similarity -->|"signal"| node_caption_classifier
node_entity_alignment -->|"signal"| node_caption_classifier
node_structural_features -->|"signal"| node_caption_classifier
node_caption_classifier -->|"writes"| node_processed_output
node_run_pipeline -->|"produces"| node_processed_output
node_processed_output -->|"input"| node_generate_figures
node_generate_figures -->|"writes"| node_tables
node_generate_figures -->|"writes"| node_figures
node_preprocessing -.->|"uses"| node_spacy
node_entity_alignment -.->|"uses"| node_rapidfuzz
node_similarity -.->|"uses"| node_sbert
node_similarity -.->|"uses"| node_stats_libs
node_caption_classifier -.->|"uses"| node_stats_libs
node_generate_figures -.->|"uses"| node_viz_libs

click node_data_sample "https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/data/sample_30k.csv.gz"
click node_downsample "https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/downsample.py"
click node_run_pipeline "https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/run_pipeline.py"
click node_preprocessing "https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/preprocessing.py"
click node_similarity "https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/similarity.py"
click node_entity_alignment "https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/entity_alignment.py"
click node_structural_features "https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/structural_features.py"
click node_caption_classifier "https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/caption_classifier.py"
click node_generate_figures "https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/generate_figures.py"
click node_tables "https://github.com/bvijayabhaskarreddy/lexicalconnections/tree/main/results/tables"
click node_figures "https://github.com/bvijayabhaskarreddy/lexicalconnections/tree/main/results/figures"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_data_sample,node_processed_output toneBlue
class node_downsample,node_run_pipeline,node_preprocessing,node_similarity,node_entity_alignment,node_structural_features,node_caption_classifier toneAmber
class node_generate_figures,node_tables,node_figures toneMint
class node_spacy,node_sbert,node_stats_libs,node_viz_libs,node_rapidfuzz toneRose
```

## Components

### Data layer

Input sample and intermediate/output datasets.

- [Sample CSV](https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/data/sample_30k.csv.gz): input dataset · Gzipped caption-article sample used as pipeline input. · `data/sample_30k.csv.gz`
- **Processed data**: analysis dataset · Intermediate results carrying computed labels and features forward.

### Processing layer

NLP preprocessing, feature extraction, and classification pipeline.

- [Downsample](https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/downsample.py): sampling utility · Creates a manageable stratified subset from a larger corpus. · `code/downsample.py`
- [Run pipeline](https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/run_pipeline.py): orchestrator · Coordinates the end-to-end analytics workflow. · `code/run_pipeline.py`
- [Preprocess](https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/preprocessing.py): text normalization · Cleans and normalizes caption and article text before comparison. · `code/preprocessing.py`
- [Similarity](https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/similarity.py): semantic comparison · Computes SBERT, TF-IDF, and Jaccard similarity signals. · `code/similarity.py`
- [Entity alignment](https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/entity_alignment.py): NER matching · Measures overlap and fuzzy correspondence of named entities. · `code/entity_alignment.py`
- [Structure features](https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/structural_features.py): linguistic features · Extracts clausal, dependency, and POS-based form features. · `code/structural_features.py`
- [Classifier](https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/caption_classifier.py): labeling logic · Maps engineered features to caption relationship categories. · `code/caption_classifier.py`

### Reporting layer

Generated tables and figures for analysis and publication.

- [Generate figures](https://github.com/bvijayabhaskarreddy/lexicalconnections/blob/main/code/generate_figures.py): reporting pipeline · Consumes processed data to create charts and summary artifacts. · `code/generate_figures.py`
- [Tables](https://github.com/bvijayabhaskarreddy/lexicalconnections/tree/main/results/tables): publication outputs · Descriptive statistics, tests, and regression tables. · `results/tables`
- [Figures](https://github.com/bvijayabhaskarreddy/lexicalconnections/tree/main/results/figures): visual outputs · Generated plots and visual summaries. · `results/figures`

### External dependencies

Core libraries and models that shape the pipeline.

- **spaCy**: nlp library · Provides parsing and named-entity recognition.
- **SBERT**: embedding model · Sentence-Transformers model used for semantic similarity.
- **Stats tools**: analysis libraries · pandas, scikit-learn, and statsmodels support tabular analysis and regression.
- **Plotting libs**: visualization libraries · matplotlib and seaborn support figure generation.
- **RapidFuzz**: fuzzy matching library · Supports approximate entity matching.

## Connections

- Sample CSV → Run pipeline: input · Raw sampled pairs enter the pipeline.
- Downsample → Sample CSV: prepares · Optional sampling step used to build the working dataset.
- Run pipeline → Preprocess: orchestrates · Pipeline begins with text normalization.
- Preprocess → Similarity: feeds · Cleaned text is compared semantically.
- Preprocess → Entity alignment: feeds · Normalized text is used for entity overlap checks.
- Preprocess → Structure features: feeds · Preprocessed text supports structural analysis.
- Similarity → Classifier: signal · Similarity metrics contribute to classification.
- Entity alignment → Classifier: signal · Entity overlap contributes to classification.
- Structure features → Classifier: signal · Syntactic features contribute to classification.
- Classifier → Processed data: writes · Labels and features are persisted for analysis.
- Run pipeline → Processed data: produces · End-to-end execution emits the processed dataset.
- Processed data → Generate figures: input · Reporting stage consumes the analysis dataset.
- Generate figures → Tables: writes · Creates statistical tables and summaries.
- Generate figures → Figures: writes · Creates publication figures.
- Preprocess → spaCy: uses · Linguistic parsing and NER support.
- Entity alignment → RapidFuzz: uses · Approximate matching for entity correspondence.
- Similarity → SBERT: uses · Embedding-based semantic comparison.
- Similarity → Stats tools: uses · Tabular similarity analysis and metrics.
- Classifier → Stats tools: uses · Classification and statistical modeling support.
- Generate figures → Plotting libs: uses · Visualization libraries render outputs.
