# improve-ai/tracker-trainer architecture

> Architecture diagram of the GitHub repository improve-ai/tracker-trainer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/improve-ai/tracker-trainer
- Repository: https://github.com/improve-ai/tracker-trainer

## Overview

a. Client → Tracking Endpoint → Kinesis Firehose
b. Firehose → S3 Raw Data
c. Ingest Lambda → Processed Data in S3
d. Training Pipeline:
- Processed Data → SageMaker Training Job
- Training Job → Model Artifacts
- Model Artifacts → Model Serving Layer

##### Key Components to Highlight

a. XGBoost Training Pipeline:
- Feature encoding (feature_encoder.py)
- Decision training (decision_trainer.py)
- Model checkpointing (checkpoint.py)

b. Data Processing:
- Firehose record processing
- Parquet conversions
- Feature selection and flattening

##### Technical Details

- Serverless components (Lambda, Firehose)
- File formats (Parquet, XGB, MLModel)
- Public endpoints vs private infrastructure

##### Architecture Notes

- Serverless nature of the system
- Multi-region capability if applicable
- Scaling points
- Integration points for different SDK clients (iOS, Android, Python)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Data Ingestion Layer"
        Client[Client Applications]
        TrackAPI["HTTP Tracking API"]:::api
        Firehose[("Kinesis Firehose")]:::aws
        IngestLambda["Ingest Lambda"]:::lambda
    end

    subgraph "Storage Layer"
        RawS3[("Raw Data S3")]:::s3
        ProcessedS3[("Processed Data S3")]:::s3
        ModelS3[("Model Storage S3")]:::s3
    end

    subgraph "Training Pipeline"
        SageMaker["SageMaker Training Job"]:::aws
        FeatureProcessing["Feature Processing"]:::process
        ModelTraining["XGBoost Training"]:::process
        Checkpoint["Model Checkpoint"]:::process
    end

    subgraph "Model Serving Layer"
        CDN["CDN Distribution"]:::aws
        ModelAPI["Model Serving API"]:::api
    end

    subgraph "Configuration & Benchmarking"
        Config["Configuration Management"]:::config
        Benchmark["Benchmark System"]:::test
    end

    %% Data Flow
    Client --> TrackAPI
    TrackAPI --> Firehose
    Firehose --> RawS3
    RawS3 --> IngestLambda
    IngestLambda --> ProcessedS3
    ProcessedS3 --> FeatureProcessing
    FeatureProcessing --> ModelTraining
    ModelTraining --> Checkpoint
    Checkpoint --> ModelS3
    ModelS3 --> CDN
    CDN --> ModelAPI
    Config --> SageMaker
    Config --> IngestLambda
    Benchmark --> ModelTraining

    %% Click Events
    click TrackAPI "https://github.com/improve-ai/tracker-trainer/blob/master/src/track/http_api.js"
    click IngestLambda "https://github.com/improve-ai/tracker-trainer/blob/master/src/ingest/ingest_firehose.py"
    click FeatureProcessing "https://github.com/improve-ai/tracker-trainer/blob/master/src/trainer/code/feature_encoder.py"
    click ModelTraining "https://github.com/improve-ai/tracker-trainer/blob/master/src/trainer/code/decision_trainer.py"
    click Checkpoint "https://github.com/improve-ai/tracker-trainer/blob/master/src/trainer/code/checkpoint.py"
    click Config "https://github.com/improve-ai/tracker-trainer/blob/master/config/config.yml"
    click Benchmark "https://github.com/improve-ai/tracker-trainer/blob/master/src/trainer/benchmark/benchmark.py"

    %% Styling
    classDef aws fill:#FF9900,stroke:#232F3E,color:white
    classDef s3 fill:#3F8624,stroke:#232F3E,color:white
    classDef api fill:#1E88E5,stroke:#232F3E,color:white
    classDef process fill:#E65100,stroke:#232F3E,color:white
    classDef lambda fill:#6741D9,stroke:#232F3E,color:white
    classDef config fill:#795548,stroke:#232F3E,color:white
    classDef test fill:#607D8B,stroke:#232F3E,color:white
```
