How renansserpa/fiap_datathon works
Project type and purpose
• This is a containerized, ML-powered web service built with FastAPI that serves as an end-to-end candidate-selection system.
• It ingests JSON data (candidates & jobs), computes text embeddings, cleans and engineers features, trains an XGBoost model, and exposes endpoints for data-processing, training, and prediction.
File structure
• Top-level “app/” folder → Python package containing:
• auth.py, security.py, users.py → user management & JWT authorization
• data_pipeline/ → ETL & feature-engineering pipeline (config.yml, data_cleaning.py, loading, model training, unify_feature_engineering)
• ml_endpoints.py → route definitions for train/predict
• database.py → ORM or raw-SQL wrapper for SQLite
• models.py & schemas.py → Pydantic and DB models
• webscrapper.py → data acquisition helper
• main.py → FastAPI application bootstrap
• Dockerfile, .github/workflows → CI/CD and container orchestration
• tests/ → unit tests for core logic
README for additional insights
• Technical stack: Python 3.11, FastAPI, Pydantic, XGBoost, SQLite, Docker, GitHub Actions for CI/CD
• Execution flow: ingest JSON → generate embeddings → clean/normalize → engineer features → train XGBoost → expose REST endpoints
Elements
• JSON input → pipeline → model → API response
• API calls → DB reads/writes
• CI → Docker build → Registry → Deployment
Summary
• Client → FastAPI (Auth, Users, Data Pipeline, ML Endpoints)
• Internal modules (cleaning, feature engineering, training, prediction)
• Persistence (SQLite, file-based model artifacts)
• DevOps flows (CI, CD, Docker)