How shalinvachheta017/text_summarization_project works
Project Type and Purpose
• This is an end-to-end machine-learning application for dialogue summarization. It spans a research phase (Jupyter notebooks), a production-grade training pipeline (modular Python code under src/textSummarizer/), and a model-serving API (app.py), all containerized with Docker.
High-Level Component Breakdown
A. Data Layer
• Configuration Files (config/config.yaml, params.yaml)
B. Pipeline Layer
1. Data Ingestion (components/data_ingestion.py / stage_01_data_ingestion.py)
2. Data Validation (components/data_validation.py / stage_02_data_validation.py)
3. Data Transformation (components/data_transformation.py / stage_03_data_transformation.py)
4. Model Training (components/model_trainer.py / stage_04_model_trainer.py)
5. Model Evaluation (components/model_evaluation.py / stage_05_model_evaluation.py)
– These stages form a linear pipeline.
C. Artifacts Storage
• Trained Model Checkpoints
• Evaluation Metrics & Reports
D. Serving Layer
• API Server (app.py using Flask/FastAPI) exposing endpoints for summarization
• Main Runner (main.py) orchestrates pipeline execution
E. DevOps / Infra
• Docker Container (via Dockerfile)
• CI/CD Workflows (.github/workflows/.gitkeep)
• Dependency Management (requirements.txt, setup.py)
Interactions & Data Flow
• Raw Data → Data Ingestion → Validated Data → Feature-Engineered Data →
→ Model Training → Model Artifact → Model Evaluation → Best Model Artifact
• Best Model Artifact + Config Files → Packaged into Docker Image →
→ Deployed API Server → Client Requests (POST /summarize) → Model Inference → Response
• Pipeline stages read/write intermediate data on the local filesystem or Hugging Face Datasets cache.
• External calls: Hugging Face Transformers & Datasets hubs for downloading pretrained PEGASUS.
Key Architectural Patterns & Principles
• Modular “pipeline” pattern: each pipeline stage is a standalone component.
• Configuration-driven: YAML files control paths, hyperparameters, and settings.
• Separation of concerns: research (notebooks) is kept distinct from production code (src/textSummarizer/).
• Containerization for environment consistency and easy deployment.
Technologies & Frameworks
• Python 3.8+
• PyTorch & Hugging Face Transformers/Datasets
• Flask or FastAPI (in app.py)
• Docker
• YAML for configuration
• GitHub Actions (placeholder for CI/CD)
1. Data Ingestion Block
• Inputs: research/samsum_dataset/, config/config.yaml
• Outputs: cleaned raw data file → next stage
2. Validation → Transformation → Training → Evaluation
• As a sequential block
3. Model Artifact Store
• Where checkpoints and metrics are written
4. API Server
• Shows loading artifact + app.py logic + requirements.txt
5. Docker + CI/CD
• Dockerfile building the image and .github/workflows/ as future CI triggers
6. External Services
• Hugging Face model hub for pretrained weights
• (Optional) Monitoring or logging sinks