# shalinvachheta017/text_summarization_project architecture

> Architecture diagram of the GitHub repository shalinvachheta017/text_summarization_project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shalinvachheta017/text_summarization_project
- Repository: https://github.com/shalinvachheta017/text_summarization_project

## Overview

##### Project Type and Purpose
- This is an end-to-end machine-learning application for dialogue summarization. It spans a research phase (Jupyter notebooks), a production-grade training pipeline (modular Python code under `src/textSummarizer/`), and a model-serving API (`app.py`), all containerized with Docker.

##### High-Level Component Breakdown
A. Data Layer
- Configuration Files (`config/config.yaml`, `params.yaml`)
B. Pipeline Layer
1. Data Ingestion (`components/data_ingestion.py` / `stage_01_data_ingestion.py`)
2. Data Validation (`components/data_validation.py` / `stage_02_data_validation.py`)
3. Data Transformation (`components/data_transformation.py` / `stage_03_data_transformation.py`)
4. Model Training (`components/model_trainer.py` / `stage_04_model_trainer.py`)
5. Model Evaluation (`components/model_evaluation.py` / `stage_05_model_evaluation.py`)
– These stages form a linear pipeline.
C. Artifacts Storage
- Trained Model Checkpoints
- Evaluation Metrics & Reports
D. Serving Layer
- API Server (`app.py` using Flask/FastAPI) exposing endpoints for summarization
- Main Runner (`main.py`) orchestrates pipeline execution
E. DevOps / Infra
- Docker Container (via `Dockerfile`)
- CI/CD Workflows (`.github/workflows/.gitkeep`)
- Dependency Management (`requirements.txt`, `setup.py`)

##### Interactions & Data Flow
- Raw Data → Data Ingestion → Validated Data → Feature-Engineered Data →
→ Model Training → Model Artifact → Model Evaluation → Best Model Artifact
- Best Model Artifact + Config Files → Packaged into Docker Image →
→ Deployed API Server → Client Requests (POST /summarize) → Model Inference → Response
- Pipeline stages read/write intermediate data on the local filesystem or Hugging Face Datasets cache.
- External calls: Hugging Face Transformers & Datasets hubs for downloading pretrained PEGASUS.

##### Key Architectural Patterns & Principles
- Modular “pipeline” pattern: each pipeline stage is a standalone component.
- Configuration-driven: YAML files control paths, hyperparameters, and settings.
- Separation of concerns: research (notebooks) is kept distinct from production code (`src/textSummarizer/`).
- Containerization for environment consistency and easy deployment.

##### Technologies & Frameworks
- Python 3.8+
- PyTorch & Hugging Face Transformers/Datasets
- Flask or FastAPI (in `app.py`)
- Docker
- YAML for configuration
- GitHub Actions (placeholder for CI/CD)

1. Data Ingestion Block
- Inputs: `research/samsum_dataset/`, `config/config.yaml`
- Outputs: cleaned raw data file → next stage
2. Validation → Transformation → Training → Evaluation
- As a sequential block
3. Model Artifact Store
- Where checkpoints and metrics are written
4. API Server
- Shows loading artifact + `app.py` logic + `requirements.txt`
5. Docker + CI/CD
- `Dockerfile` building the image and `.github/workflows/` as future CI triggers
6. External Services
- Hugging Face model hub for pretrained weights
- (Optional) Monitoring or logging sinks

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data & Config
    RawData["Raw Dataset\n(SAMSum & HF cache)"]:::storage
    Config1["config/config.yaml"]:::storage
    Config2["params.yaml"]:::storage

    %% Pipeline Stages
    subgraph "Pipeline" 
        direction TB
        Ingestion["Data Ingestion"]:::pipeline
        Validation["Data Validation"]:::pipeline
        Transformation["Data Transformation"]:::pipeline
        Training["Model Training"]:::pipeline
        Evaluation["Model Evaluation"]:::pipeline
    end

    %% Artifact Store
    ArtifactStore["Model Artifacts\n(Checkpoints & Metrics)"]:::storage

    %% Serving Layer
    API["API Server"]:::serving

    %% DevOps / Infra
    Docker["Docker Container"]:::devops
    CI["CI/CD Workflows"]:::devops

    %% External Service
    HF["Hugging Face Hub"]:::external

    %% Data Flow
    RawData -->|"raw_data.csv"| Ingestion
    Config1 -.->|config| Ingestion
    Config2 -.->|params| Ingestion

    Ingestion -->|"cleaned_data.csv"| Validation
    Validation -->|"validated_data.csv"| Transformation
    Transformation -->|"features.csv"| Training
    Training -->|"model_checkpoint.pt"| Evaluation
    Evaluation -->|"metrics_report.json"| ArtifactStore

    %% External pretrained weights
    HF -->|"pretrained PEGASUS"| Training

    %% Serving flow
    ArtifactStore -->|"best_model"| API
    Config1 -.->|config| API
    Config2 -.->|params| API

    %% DevOps flows
    CI -.->|triggers| Pipeline
    CI -.->|triggers| Docker
    Docker -->|packages API + model| API

    %% Click Events
    click RawData "https://github.com/shalinvachheta017/text_summarization_project/tree/main/research/samsum_dataset/"
    click Config1 "https://github.com/shalinvachheta017/text_summarization_project/blob/main/config/config.yaml"
    click Config2 "https://github.com/shalinvachheta017/text_summarization_project/blob/main/params.yaml"
    click Ingestion "https://github.com/shalinvachheta017/text_summarization_project/blob/main/src/textSummarizer/components/data_ingestion.py"
    click Ingestion "https://github.com/shalinvachheta017/text_summarization_project/blob/main/src/textSummarizer/pipeline/stage_01_data_ingestion.py"
    click Validation "https://github.com/shalinvachheta017/text_summarization_project/blob/main/src/textSummarizer/components/data_validation.py"
    click Validation "https://github.com/shalinvachheta017/text_summarization_project/blob/main/src/textSummarizer/pipeline/stage_02_data_validation.py"
    click Transformation "https://github.com/shalinvachheta017/text_summarization_project/blob/main/src/textSummarizer/components/data_transformation.py"
    click Transformation "https://github.com/shalinvachheta017/text_summarization_project/blob/main/src/textSummarizer/pipeline/stage_03_data_transformation.py"
    click Training "https://github.com/shalinvachheta017/text_summarization_project/blob/main/src/textSummarizer/components/model_trainer.py"
    click Training "https://github.com/shalinvachheta017/text_summarization_project/blob/main/src/textSummarizer/pipeline/stage_04_model_trainer.py"
    click Evaluation "https://github.com/shalinvachheta017/text_summarization_project/blob/main/src/textSummarizer/components/model_evaluation.py"
    click Evaluation "https://github.com/shalinvachheta017/text_summarization_project/blob/main/src/textSummarizer/pipeline/stage_05_model_evaluation.py"
    click API "https://github.com/shalinvachheta017/text_summarization_project/blob/main/app.py"
    click Docker "https://github.com/shalinvachheta017/text_summarization_project/tree/main/Dockerfile"
    click CI "https://github.com/shalinvachheta017/text_summarization_project/blob/main/.github/workflows/.gitkeep"

    %% Styles
    classDef storage fill:#cce5ff,stroke:#224a7a;
    classDef pipeline fill:#d4edda,stroke:#255d32;
    classDef serving fill:#ffe5b4,stroke:#cc8400;
    classDef devops fill:#f2f2f2,stroke:#7a7a7a;
    classDef external fill:#fff3cd,stroke:#856404;
```
