How entbappy/end-to-end-youtube-sentiment works
Main Components
• ML Pipeline and Experimentation:
• Data Ingestion and Preprocessing (found under src/data and notebooks).
• Model Building, Evaluation, and Registration (in src/model and pickled model files like lgbm_model.pkl and tfidf_vectorizer.pkl).
• Experiment tracking and reproducibility via DVC (dvc.yaml, dvc.lock, and the .dvc folder).
• API Layer:
• Main server scripts (app.py and flask_app/app.py) exposing a REST endpoint (/predict) for sentiment analysis.
• Frontend:
• A Chrome extension (yt-chrome-plugin-frontend) that interacts with the backend service.
• Deployment & Infrastructure:
• Docker containerization (Dockerfile).
• CI/CD through GitHub Actions (.github/workflows/cicd.yaml).
• AWS services for deployment (instructions in README regarding EC2 and ECR).
Relationships and Interactions
• The ML Pipeline (data ingestion, model building, evaluation) produces predictive models and vectorizers used by the API.
• The API layer receives JSON POST requests (see the sample in the README) from either a client or directly from the Chrome extension and uses the trained models to process inputs.
• The Chrome extension is a frontend client that sends user requests to the Flask API.
• DVC manages the data and experiment pipelines, ensuring that the correct versioned data and models are used.
• CI/CD pipelines build a Docker image, push it to AWS ECR, and deploy it on AWS EC2 – ensuring smooth integration and continuous deployment.
• The infrastructure orchestration (Dockerfile, GitHub Actions, and AWS deployment instructions) connects the dots between code, container image, and production deployment.
Architectural Patterns and Design Principles
• Modularity: Separation between the data pipeline (in src), API (Flask app), and the frontend (Chrome extension).
• Reproducibility: DVC tracks data, experiments, and dependencies.
• Containerization & CI/CD: Docker is used for packaging the application; GitHub Actions automate build and deployment to AWS.
• RESTful Design: The API follows REST principles for accepting and processing prediction requests.
• Integration: There is a clear boundary between ML model building and serving, making it easier to update either side independently.
Incorporate Relevant Technologies
• Programming Language & Frameworks: Python (Flask for API, ML libraries referenced in notebooks).
• Version Control and Experiment Tracking: Git and DVC.
• Deployment Tools: Docker, GitHub Actions, AWS (EC2, ECR).
• Frontend: Basic HTML/JS for the Chrome plugin (manifest.json, popup.html, popup.js).
• Model Artifacts: Pickled models (lgbm_model.pkl, tfidf_vectorizer.pkl).