How daniel27110/ciencia-de-datos-aplicada-proyecto-final works
Project Type and Purpose
• This is a full-stack data analytics and machine learning solution that processes raw student data, trains a predictive model, and exposes it via a web interface to identify at-risk students.
• It comprises three major phases: offline data ingestion & modeling, model artifact management, and a lightweight REST + UI deployment.
High-Level Components
a. Data Storage Layer
• Raw Data Directory (“data/raw”) – source CSV files (DIVIPOLA, programas.csv, Demograficos.csv.gz).
• Staging Directory (“data/stage”) – cleaned and partitioned CSVs (clean.csv, train.csv, ultimo_semestre.csv).
b. Analytics Pipeline
• Jupyter Notebooks under “analytics” and “data/analytics”: EDA, data preparation, model selection, training.
• Outputs: clean.csv in data/stage and serialized models (rf_model.joblib, lr_model.joblib, rf_pipeline.joblib) in “deploy”.
c. Model Artifact Repository
• Serialized models and pipelines stored in “deploy” alongside versioned artifacts (clf_model.joblib, rf_model.joblib, etc.).
d. API / Service Layer
• FastAPI application (“deploy/main.py”, run via uvicorn)
• Pydantic schema definitions (“deploy/datamodel.py”)
• Model wrappers (“deploy/model.py”, “deploy/rf_model.py”, “deploy/lr_model.py”)
• Endpoints:
– GET /inicio serves HTML form (index.html)
– POST /predict returns single or batch predictions
e. Frontend Layer
• Static HTML templates (“deploy/index.html”, “deploy/student-details.html”)
• JavaScript client logic (“deploy/charts.js”) for rendering charts and calling API
f. Supporting Files
• Configuration / dependencies (requirements.txt)
• Documentation (docs/, README.md)
Component Interactions and Data Flow
1. Offline phase: Raw CSVs → Data preparation notebook → clean.csv (staging) → model selection notebook → trained model artifacts (.joblib)
2. Deployment phase:
• At startup, FastAPI loads model artifacts into memory.
• User opens index.html in browser → fills form or uploads estudiantes.csv → client JS invokes /predict.
• FastAPI endpoint processes JSON or file, invokes model wrapper → returns prediction response.
• Frontend renders results and drill-down chart via charts.js and student-details.html.
Architectural Patterns and Design Principles
• Separation of Concerns: distinct offline analytics pipeline vs. online serving components.
• Micro-service pattern: stateless, single-responsibility FastAPI application.
• File-based “database”: CSVs and joblib serializations replace a traditional RDBMS.
• Modular code: Pydantic for input validation, separate model wrappers, static templates for UI.