How bharaniabhishek123/final_project_cs229 works
File structure
• Top-level directories:
• data/ – raw and cleaned CSV datasets for US and GB videos and comments
• youtube_sentiment/ – Python modules for each pipeline stage (preprocess.py, sentiment.py, precision_recall.py, etc.)
• proposal/, progress/, final/, poster/ – LaTeX and PPT artifacts documenting the research progress and results
• Configuration files: LICENSE, README.md, .idea/
• No explicit microservice or web layer—this is a batch data pipeline.
README for additional insights
• The README only confirms that it’s the Fall 2017 CS229 project. There is no pre-drawn architecture, so we must infer the pipeline structure from folder names and script names.
a. Main components
1. Raw Data Storage
• Raw CSVs and ZIPs under data/youtube/ (GBComments-train.csv, USvideos_clean_train.csv, etc.)
2. Data Ingestion & Cleaning
• youtube_sentiment/preprocess.py reads raw CSVs, handles missing values, text normalization, splits train/dev/test, writes cleaned CSVs back to data/youtube/*.
3. Feature Extraction & Tagging
• youtube_sentiment/tags.py and category.py extract metadata features (video category, tags)
• youtube_sentiment/generateWordCloud.py creates visual word clouds for exploratory analysis
4. Model Training & Sentiment Classification
• youtube_sentiment/sentiment.py encapsulates model training (e.g., TF-IDF or bag-of-words → classifier), cross-validation
5. Evaluation & Metrics
• youtube_sentiment/precision_recall.py calculates accuracy, precision/recall, generates performance tables
6. Reporting & Visualization
• Final LaTeX pipeline under final/ generates figures, tables, writes final.tex
• poster/ holds PPT slides
• proposal/ and progress/ folders track earlier drafts
b. Relationships and interactions
• Raw Data → Data Ingestion & Cleaning → Clean Data
• Clean Data → Feature Extraction → Feature Store
• Feature Store → Model Training → Trained Model
• Trained Model + Test Data → Evaluation → Metrics & Plots
• Metrics & Plots + Narrative → Reporting artifacts (LaTeX, PPT)
c. Architectural patterns and design principles
• Linear batch pipeline (ETL → ML training → evaluation → reporting)
• Modular script organization: each stage in a dedicated Python module
• Data persisted at each stage as CSV artifacts, enabling reproducibility
d. Key technologies/frameworks
• Python (pandas, scikit-learn, matplotlib/wordcloud)
• LaTeX (LLNCS template) for final write-up
• PPT (PowerPoint) for poster