# arezkisaba/french-id-detection-training architecture

> Architecture diagram of the GitHub repository arezkisaba/french-id-detection-training, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arezkisaba/french-id-detection-training
- Repository: https://github.com/arezkisaba/french-id-detection-training

## Overview

##### Main components
- Data‑sources
- Document templates (src/dataset/templates)
- OpenImages backgrounds (src/openimages via backgrounds‑download scripts)
- Dataset generation pipeline
- Python scripts:
– backgrounds-download(.py) & backgrounds-download-openimages.py
– dataset-generate.py (combines templates + backgrounds + augmentations)
- Model training pipeline
- dataset-train.py (Ultralytics YOLO11 in TensorFlow, albumentations, tensorboard)
- args.yaml for hyperparameters (src/results/args.yaml)
- Model artifacts storage
- Local filesystem (src/results/weights: .pt, web_model_original, web_model_quantized)
- Model conversion & testing
- model-convert.py (convert .pt → TF SavedModel / web model)
- model-test.py (validation/inference on samples)
- Auxiliary
- utils.py (shared routines)
- default.ipynb (exploration / visualization)
- AWS integration via boto3 (optional for uploading/downloading)

##### Relationships & data‑flows
b. “Dataset Generator” → output folder (src/dataset/… or results/…) → input into “Model Trainer”
c. “Model Trainer” writes to “Results/Weights” and “Results/args.yaml”
d. “Model Converter” consumes weights → produces TF/web artifacts
e. “Model Tester” consumes converted model + test samples (src/tests/samples)
f. (Optional) upload/download from S3 via boto3 at each stage

##### Architectural patterns
- Pipeline pattern: clear stages (ETL → training → post‑processing → evaluation)
- Modularity: each stage is a standalone Python script, enabling ad‑hoc execution and replacement
- Config‑driven: args.yaml, metadata.yaml in web models
- Data versioning: results folder holds multiple weight variants (best.pt, last.pt, quantized, original, etc.)

##### Technologies/frameworks to components
- Dataset Generator: Python3.11, albumentations
- Model Trainer: Ultralytics YOLO (TF backend), tensorboard
- Conversion: TensorFlow SavedModel, Web‑model JSON/BIN
- Testing: boto3 (if retrieving data/models), pure Python inference

##### Final checklist
- Clear component labels (script name + responsibility)
- File system storage illustrated for inputs/outputs

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preparation Stage
    subgraph "Data Preparation" 
        direction TB
        Templates["Document Templates"]:::dataPrep
        OpenImagesAPI["OpenImages API"]:::external
        BD1["backgrounds-download.py"]:::dataPrep
        BD2["backgrounds-download-openimages.py"]:::dataPrep
        DatasetGen["Dataset Generator (dataset-generate.py)"]:::dataPrep
        ArchiveGen["Archived Generator (archives/dataset-generate.py)"]:::dataPrep
        Utils["Shared Utilities (utils.py)"]:::dataPrep
    end

    %% Training Stage
    subgraph "Model Training" 
        direction TB
        Trainer["Model Trainer (dataset-train.py)"]:::training
        Args["Hyperparameters (args.yaml)"]:::training
        WeightsDir["Weights Directory"]:::training
        BestPT["best.pt"]:::training
        LastPT["last.pt"]:::training
        WebOriginal["best_web_model_original"]:::training
        WebQuant["best_web_model_quantized"]:::training
        ResultsCSV["Results CSV (results.csv)"]:::training
    end

    %% Conversion Stage
    subgraph "Model Conversion" 
        direction TB
        Converter["Model Converter (model-convert.py)"]:::conversion
        SavedModel["TensorFlow SavedModel"]:::conversion
        WebModel["Web Model (.json + .bin)"]:::conversion
        Metadata["metadata.yaml"]:::conversion
    end

    %% Testing Stage
    subgraph "Model Testing" 
        direction TB
        Tester["Model Tester (model-test.py)"]:::testing
        Samples["Test Samples"]:::testing
    end

    %% External Services
    OpenAI["OpenImages API"]:::external
    S3["AWS S3"]:::external

    %% Data Flows
    Templates -->|"templates"| DatasetGen
    OpenImagesAPI -->|"backgrounds metadata"| BD2
    BD1 -->|"download backgrounds"| DatasetGen
    BD2 -->|"download backgrounds"| DatasetGen
    DatasetGen -->|".jpg + .xml annotations"| Trainer
    ArchiveGen -->|"alternate pipeline"| Trainer
    Utils --> DatasetGen
    Utils --> Trainer
    Utils -.-> Converter
    Utils -.-> Tester

    Trainer -->|".pt weights"| WeightsDir
    Trainer -->|"args.yaml"| Args
    WeightsDir --> BestPT
    WeightsDir --> LastPT
    WeightsDir --> WebOriginal
    WeightsDir --> WebQuant
    WeightsDir --> ResultsCSV

    BestPT --> Converter
    LastPT --> Converter
    Converter -->|"SavedModel"| SavedModel
    Converter -->|"Web artifact"| WebModel
    WebModel --> Metadata
    SavedModel --> Tester
    WebModel --> Tester
    Samples --> Tester

    %% Optional AWS integration
    DatasetGen -.->|"upload/download"| S3
    Trainer -.->|"upload/download"| S3
    Converter -.->|"upload/download"| S3
    Tester -.->|"upload/download"| S3

    %% Click Events
    click Templates "https://github.com/arezkisaba/french-id-detection-training/tree/main/src/dataset/templates"
    click BD1 "https://github.com/arezkisaba/french-id-detection-training/blob/main/src/scripts/backgrounds-download.py"
    click BD2 "https://github.com/arezkisaba/french-id-detection-training/blob/main/src/scripts/backgrounds-download-openimages.py"
    click DatasetGen "https://github.com/arezkisaba/french-id-detection-training/blob/main/src/scripts/dataset-generate.py"
    click ArchiveGen "https://github.com/arezkisaba/french-id-detection-training/blob/main/src/scripts/archives/dataset-generate.py"
    click Trainer "https://github.com/arezkisaba/french-id-detection-training/blob/main/src/scripts/dataset-train.py"
    click Args "https://github.com/arezkisaba/french-id-detection-training/blob/main/src/results/args.yaml"
    click WeightsDir "https://github.com/arezkisaba/french-id-detection-training/tree/main/src/results/weights"
    click BestPT "https://github.com/arezkisaba/french-id-detection-training/blob/main/src/results/weights/best.pt"
    click LastPT "https://github.com/arezkisaba/french-id-detection-training/blob/main/src/results/weights/last.pt"
    click WebOriginal "https://github.com/arezkisaba/french-id-detection-training/tree/main/src/results/weights/best_web_model_original"
    click WebQuant "https://github.com/arezkisaba/french-id-detection-training/tree/main/src/results/weights/best_web_model_quantized"
    click Converter "https://github.com/arezkisaba/french-id-detection-training/blob/main/src/scripts/model-convert.py"
    click Tester "https://github.com/arezkisaba/french-id-detection-training/blob/main/src/scripts/model-test.py"
    click Samples "https://github.com/arezkisaba/french-id-detection-training/tree/main/src/tests/samples"
    click Utils "https://github.com/arezkisaba/french-id-detection-training/blob/main/src/scripts/utils.py"
    click ResultsCSV "https://github.com/arezkisaba/french-id-detection-training/blob/main/src/results/results.csv"

    %% Styles
    classDef dataPrep fill:#e0f7e9,stroke:#2e7d32,color:#2e7d32
    classDef training fill:#bbdefb,stroke:#1565c0,color:#1565c0
    classDef conversion fill:#fff9c4,stroke:#f9a825,color:#f57f17
    classDef testing fill:#ffcdd2,stroke:#c62828,color:#c62828
    classDef external fill:#f0f0f0,stroke:#666,stroke-dasharray: 5 5,color:#333
```
