How jkcreddy/text-summarizer works
Project Type and Purpose
• This is an end-to-end machine-learning pipeline and serving application for text summarization.
• It ingests raw data, validates and transforms it, trains and evaluates a model, then exposes a REST endpoint (app.py) for on-demand summarization.
• It also includes CI/CD via GitHub Actions to build a Docker image, push it to AWS ECR, and deploy to an EC2 instance.
High-Level File Structure Analysis
• Top-level scripts:
• main.py – orchestrates the pipeline (CLI).
• app.py – runs the web service (Flask/FastAPI) for inference.
• Configuration:
• config/config.yaml, params.yaml – hyperparameters, file paths.
• src/textSummarizer/config/configuration.py – loads and provides config objects.
• Pipeline code under src/textSummarizer:
• components/ – modular stages (data_ingestion, validation, transformation, trainer, evaluator).
• pipeline/ – stage orchestration scripts (stage_01…05) plus prediction.py.
• utils/, logging/, entity/, constants/ – shared helpers, data models, logging setup.
• Artifacts folder:
• artifacts/data_ingestion/… – raw and Hugging Face–formatted dataset files.
• Deployment & CI:
• Dockerfile – containerizes the app.
• .github/workflows/main.yaml – CI/CD pipeline.
• AWS secrets, ECR/EC2 instructions in README.
Main Components & Relationships
• Component: Configuration Manager
• Reads config/config.yaml & params.yaml → supplies paths & hyperparameters.
• Component: Data Storage
• Raw data (artifacts/data_ingestion/data.zip) → Hugging Face dataset files under artifacts/data_ingestion/samsum_dataset.
• Component: Pipeline Stages
1. Data Ingestion
2. Data Validation
3. Data Transformation
4. Model Training
5. Model Evaluation
• Each implemented in src/textSummarizer/components and invoked by src/textSummarizer/pipeline/stage_* scripts.
• Component: Model Artifacts
• Trained model files (not explicitly shown, but saved via trainer component).
• Component: Prediction Service
• prediction.py provides inference logic.
• app.py wraps it in a REST API.
• CI/CD Workflow
1. Developer commits code to GitHub.
2. GitHub Actions builds Docker image (Dockerfile + requirements.txt).
3. Pushes image to AWS ECR.
4. EC2 self-hosted runner pulls image from ECR.
5. Launches the container, exposing the API endpoint.
Architectural Patterns & Design Principles
• Modular separation: each pipeline stage is self-contained.
• Configuration-driven: YAML files define behavior without code changes.
• Clear layering: components → pipeline orchestrator → CLI (main.py) or service (app.py).
• CI/CD automation for reproducible builds and deployments.