# renansserpa/fiap_datathon architecture

> Architecture diagram of the GitHub repository renansserpa/fiap_datathon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/renansserpa/fiap_datathon
- Repository: https://github.com/renansserpa/fiap_datathon

## Overview

##### Project type and purpose
- This is a containerized, ML-powered web service built with FastAPI that serves as an end-to-end candidate-selection system.
- It ingests JSON data (candidates & jobs), computes text embeddings, cleans and engineers features, trains an XGBoost model, and exposes endpoints for data-processing, training, and prediction.

##### File structure
- Top-level “app/” folder → Python package containing:
- auth.py, security.py, users.py → user management & JWT authorization
- data_pipeline/ → ETL & feature-engineering pipeline (config.yml, data_cleaning.py, loading, model training, unify_feature_engineering)
- ml_endpoints.py → route definitions for train/predict
- database.py → ORM or raw-SQL wrapper for SQLite
- models.py & schemas.py → Pydantic and DB models
- webscrapper.py → data acquisition helper
- main.py → FastAPI application bootstrap
- Dockerfile, .github/workflows → CI/CD and container orchestration
- tests/ → unit tests for core logic

##### README for additional insights
- Technical stack: Python 3.11, FastAPI, Pydantic, XGBoost, SQLite, Docker, GitHub Actions for CI/CD
- Execution flow: ingest JSON → generate embeddings → clean/normalize → engineer features → train XGBoost → expose REST endpoints

##### Elements
- JSON input → pipeline → model → API response
- API calls → DB reads/writes
- CI → Docker build → Registry → Deployment

**Summary**
- Client → FastAPI (Auth, Users, Data Pipeline, ML Endpoints)
- Internal modules (cleaning, feature engineering, training, prediction)
- Persistence (SQLite, file-based model artifacts)
- DevOps flows (CI, CD, Docker)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Clients and External
    Client["Client / Swagger UI"]:::external
    External["External Data Source\n(Web Scrapper)"]:::external
    EmbeddingAPI["Embedding API"]:::external

    %% API Layer
    subgraph "API Layer"
        direction TB
        Auth["Auth Service"]:::api
        Users["User Management Service"]:::api
    end

    %% Application Core
    subgraph "Application Core"
        direction TB
        Main["FastAPI Server\n(main.py)"]:::api
        Exception["Exception Handling"]:::api
        ML["ML Endpoints\n(Train & Predict)"]:::api
        subgraph "Data Pipeline" 
            direction TB
            Conf["Config\n(config.yml)"]:::pipeline
            Clean["Data Cleaning Module"]:::pipeline
            Load["Data Loading Module"]:::pipeline
            FE["Feature Engineering Module"]:::pipeline
            Train["Model Training & Persistence"]:::pipeline
        end
    end

    %% Database Layer
    subgraph "Database Layer"
        direction TB
        DBConn["DB Connector\n(database.py)"]:::storage
        DBModels["DB Models\n(models.py)"]:::storage
        Schemas["Pydantic Schemas\n(schemas.py)"]:::storage
        DBFile["SQLite File\n(identifier.sqlite)"]:::storage
    end

    FileStore["File Storage\n(model artifacts, CSVs)"]:::storage

    %% DevOps Pipeline
    subgraph "DevOps Pipeline"
        direction TB
        CI["CI Pipeline\n(CI.yaml)"]:::devops
        Tests["Test Suite\n(tests/)"]:::devops
        CD["CD Pipeline\n(CD.yml)"]:::devops
        Docker["Docker Image\n(Dockerfile)"]:::devops
        Registry["Container Registry"]:::devops
        Deployment["Deployment"]:::devops
    end

    %% Connections
    Client -->|HTTP requests| Main
    Main --> Auth
    Main --> Users
    Main --> ML
    Main --> Exception

    Main --> DBConn
    DBConn --> DBModels
    DBConn --> Schemas
    DBConn --> DBFile

    ML --> Conf
    ML --> Clean
    ML --> Load
    ML --> FE
    ML --> Train

    Clean --> Load
    Load --> FE
    FE -->|calls Embedding| EmbeddingAPI
    FE --> Train
    Train --> FileStore
    ML --> FileStore

    Main --> External
    External --> Clean

    CI --> Tests
    Tests --> CD
    CD --> Docker
    Docker --> Registry
    Registry --> Deployment

    %% Click Events
    click Auth "https://github.com/renansserpa/fiap_datathon/blob/main/app/auth.py"
    click Auth "https://github.com/renansserpa/fiap_datathon/blob/main/app/security.py"
    click Users "https://github.com/renansserpa/fiap_datathon/blob/main/app/users.py"
    click Main "https://github.com/renansserpa/fiap_datathon/blob/main/app/main.py"
    click ML "https://github.com/renansserpa/fiap_datathon/blob/main/app/ml_endpoints.py"
    click Exception "https://github.com/renansserpa/fiap_datathon/blob/main/app/exceptions.py"
    click DBConn "https://github.com/renansserpa/fiap_datathon/blob/main/app/database.py"
    click DBModels "https://github.com/renansserpa/fiap_datathon/blob/main/app/models.py"
    click Schemas "https://github.com/renansserpa/fiap_datathon/blob/main/app/schemas.py"
    click DBFile "https://github.com/renansserpa/fiap_datathon/blob/main/identifier.sqlite"
    click Conf "https://github.com/renansserpa/fiap_datathon/blob/main/app/data_pipeline/config.yml"
    click Clean "https://github.com/renansserpa/fiap_datathon/blob/main/app/data_pipeline/data_cleaning.py"
    click Load "https://github.com/renansserpa/fiap_datathon/blob/main/app/data_pipeline/data_loading.py"
    click FE "https://github.com/renansserpa/fiap_datathon/blob/main/app/data_pipeline/unify_feature_engineering.py"
    click Train "https://github.com/renansserpa/fiap_datathon/blob/main/app/data_pipeline/model.py"
    click External "https://github.com/renansserpa/fiap_datathon/blob/main/app/webscrapper.py"
    click CI "https://github.com/renansserpa/fiap_datathon/blob/main/.github/workflows/CI.yaml"
    click Tests "https://github.com/renansserpa/fiap_datathon/tree/main/tests/"
    click CD "https://github.com/renansserpa/fiap_datathon/blob/main/.github/workflows/CD.yml"
    click Docker "https://github.com/renansserpa/fiap_datathon/tree/main/Dockerfile"

    %% Styles
    classDef api fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef pipeline fill:#98FB98,stroke:#333,stroke-width:1px
    classDef storage fill:#FFD700,stroke:#333,stroke-width:1px
    classDef devops fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px
```
