# velocitatem/dapper architecture

> Architecture diagram of the GitHub repository velocitatem/dapper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/velocitatem/dapper
- Repository: https://github.com/velocitatem/dapper

## Overview

##### Main Components
- Data Processing Pipeline: Represent the components responsible for loading, processing (resizing images, splitting datasets), and uploading images to MinIO. Key files include those in classification/data, configuration files (e.g., lsnet.yaml, settings.yaml), and dataset loaders.
- Model Training Pipeline: Highlight the training process using the custom MinioImageDataset, the training scripts (train.py), model definitions (in training/models.py, training/resnet.py), and configuration files under classification/training/configs.
- Extraction Module: Although not highlighted in the README, the existence of the extraction/ directory with rule_based_extractor.py suggests a module that might be used for additional document extraction tasks.
- Dashboard and External Interfaces: The long list of scripts like dashboard.py, and potential interaction points from external services (via docker-compose and configuration settings), should be clearly drawn.

##### Analyzing the File Structure for Architectural Clues
- Folder “classification”: This is the core of the document classification workflow. Inside, the separation into subdirectories for data, training, datasets, and utils indicates a modular design.
- Subdirectories “training” and “datasets”: These indicate clear separation between dataset management and the actual model training routines.
- Configurations: Multiple YAML files under training/configs and classification/config suggest that the system is highly configurable with parameters for model and data handling.
- Extraction: The extraction subpackage (with both a base extractor and a rule-based extractor) implies an additional function layer that might process or post-process the document outputs.
- GitHub workflows and docker-compose: These files indicate CI/CD and deployment practices that can be depicted externally for integration with infrastructure.

a. Main Containers/Components:
- Frontend/Interface Layer (if applicable): Although there is a dashboard script, it might represent a lightweight UI or API.
- Backend/Processing Layer: This includes the “classification” component that is responsible for data processing and training.
▫ Data Processing Subsystem: Represent the scripts for loading and processing image data, including the augmentation and resizing operations.
▫ Model Training Subsystem: Include components for model selection (ResNet18/transfer learning), training, evaluation, and model persistence (saving the model.pth).
b. Relationships and Data Flow:
- Optionally, show external datasets and sources (RVL-CDIP, invoices, Kaggle) as inputs into the data processing component.
c. Architectural Patterns/Principles:
- Modular Design: The clear separation between data handling, training, and extraction modules.
- Pipeline Architecture: A sequential workflow moving from data ingestion and processing to model training and evaluation.
- External Integration: The use of external storage (MinIO) and external dataset sources should be visibly integrated.
d. Technologies and Frameworks:
- Mention Python as the primary programming language, MinIO for object storage, and the use of machine learning frameworks (implied from the training scripts). Also note Docker (via docker-compose) for orchestration.

## Diagram (Mermaid)

```mermaid
graph TD
  subgraph "External Layer"
    ext["External Datasets"]:::external
  end

  subgraph "Processing Layer"
    DP["Data Processing Pipeline"]:::processing
    IM["Image Processing"]:::processing
    MIH["MinIO Handler"]:::processing
    MID["MinIO Dataset"]:::processing
  end

  subgraph "Storage Layer"
    MINIO["MinIO Object Storage"]:::storage
  end

  subgraph "Training Layer"
    MT["Model Training Pipeline"]:::training
  end

  subgraph "Extraction Layer"
    EX["Extraction Module"]:::extraction
  end

  subgraph "Interface Layer"
    DASH["Dashboard"]:::interface
  end

  ext -->|"input"| DP
  DP -->|"processes"| IM
  IM -->|"integrates"| MIH
  IM -->|"integrates"| MID
  MIH -->|"uploads"| MINIO
  MID -->|"uploads"| MINIO
  MINIO -->|"fetches"| MT
  MT -->|"produces"| EX
  DP -->|"sends"| EX
  DASH -->|"monitors"| MT
  DASH -->|"views"| EX

  click DP "https://github.com/velocitatem/dapper/tree/main/classification/data"
  click MT "https://github.com/velocitatem/dapper/tree/main/classification/training"
  click MIH "https://github.com/velocitatem/dapper/blob/main/classification/data/minio_handler.py"
  click MID "https://github.com/velocitatem/dapper/blob/main/classification/data/minio_dataset.py"
  click EX "https://github.com/velocitatem/dapper/blob/main/extraction/base_extractor.py"
  click EX "https://github.com/velocitatem/dapper/blob/main/extraction/rule_based_extractor.py"
  click DASH "https://github.com/velocitatem/dapper/blob/main/dashboard.py"

  classDef processing fill:#FFEEAD,stroke:#333,stroke-width:2px;
  classDef training fill:#B3E5FC,stroke:#333,stroke-width:2px;
  classDef storage fill:#C8E6C9,stroke:#333,stroke-width:2px;
  classDef extraction fill:#FFD180,stroke:#333,stroke-width:2px;
  classDef interface fill:#D1C4E9,stroke:#333,stroke-width:2px;
  classDef external fill:#EEEEEE,stroke:#333,stroke-dasharray: 5 5,stroke-width:2px;
```
