# jabardigitalservice/datasae architecture

> Architecture diagram of the GitHub repository jabardigitalservice/datasae, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jabardigitalservice/datasae
- Repository: https://github.com/jabardigitalservice/datasae

## Overview

##### Main Components Identification
The system consists of two primary functional modules:
- Data Quality Checker
- Data Source Converter

##### Top Layer - Input/Configuration
- Configuration Files (JSON/YAML)
- CLI Interface
- Python API Interface

##### Middle Layer - Core Processing
- Config Handler (central)
- To Converter Module and Checker Module

##### Bottom Layer - Data Sources and Output
- Various data sources
- Checker Results output

Use these specific elements:

- Blue: Core Components
- Green: Data Sources
- Orange: Input/Output Interfaces
- Gray: External Services

**→ Data Flow**
**⇢ Configuration Flow**
**⇄ Two-way Communication**

##### Detailed Component Relationships

**Input Layer:**
**CLI/Python API → Config Handler**
**Config Files → Config Handler**

**Core Processing:**
**Config Handler ⇄ Converter Module**
**Config Handler ⇄ Checker Module**
**Converter Module → Data Type Validators**
**Checker Module → Data Type Validators**

**Data Sources:**
**Converter Module ⇄ Local Files**
**Converter Module ⇄ Google Spreadsheet**
**Converter Module ⇄ S3**
**Converter Module ⇄ SQL Databases**

##### Implementation Notes

- Version information and supported Python versions
- Notes about required dependencies for each connector

##### Additional Elements

- Error handling flow
- Configuration validation path
- Where tests integrate with the system
- CI/CD integration points

Remember to include:
- Version compatibility information
- Required dependencies for each connector
- Error handling flows
- Configuration validation paths
- Test integration points
- CI/CD integration points

This design should reflect the modular nature of the system while showing how all components work together to provide a comprehensive data quality framework.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Style Definitions
    classDef core fill:#b3d1ff,stroke:#333,stroke-width:2px
    classDef datasource fill:#90EE90,stroke:#333,stroke-width:2px
    classDef interface fill:#FFB366,stroke:#333,stroke-width:2px
    classDef external fill:#E0E0E0,stroke:#333,stroke-width:2px
    classDef config fill:#FFE5CC,stroke:#333,stroke-width:2px

    %% Input Layer
    subgraph Input
        CLI["CLI Interface"]:::interface
        PythonAPI["Python API"]:::interface
        ConfigFiles["Configuration Files"]:::config
    end

    %% Core Processing Layer
    subgraph CoreProcessing
        ConfigHandler["Config Handler"]:::core
        ConverterModule["Converter Module"]:::core
        CheckerModule["Checker Module"]:::core
        
        subgraph Validators["Data Type Validators"]
            Boolean["Boolean Validator"]:::core
            Float["Float Validator"]:::core
            Integer["Integer Validator"]:::core
            String["String Validator"]:::core
            Timestamp["Timestamp Validator"]:::core
        end
    end

    %% Data Sources Layer
    subgraph DataSources
        LocalFiles[("Local Files")]:::datasource
        GSheet[("Google Spreadsheet")]:::datasource
        S3[("S3 Storage")]:::datasource
        SQL[("SQL Databases")]:::datasource
    end

    %% Relationships
    CLI --> ConfigHandler
    PythonAPI --> ConfigHandler
    ConfigFiles --> ConfigHandler
    
    ConfigHandler <--> ConverterModule
    ConfigHandler <--> CheckerModule
    
    ConverterModule --> Boolean & Float & Integer & String & Timestamp
    CheckerModule --> Boolean & Float & Integer & String & Timestamp
    
    ConverterModule <--> LocalFiles
    ConverterModule <--> GSheet
    ConverterModule <--> S3
    ConverterModule <--> SQL

    %% Click Events
    click Boolean "https://github.com/jabardigitalservice/DataSae/blob/main/datasae/boolean.py"
    click Float "https://github.com/jabardigitalservice/DataSae/blob/main/datasae/float.py"
    click Integer "https://github.com/jabardigitalservice/DataSae/blob/main/datasae/integer.py"
    click String "https://github.com/jabardigitalservice/DataSae/blob/main/datasae/string.py"
    click Timestamp "https://github.com/jabardigitalservice/DataSae/blob/main/datasae/timestamp.py"
    click ConverterModule "https://github.com/jabardigitalservice/DataSae/blob/main/datasae/converter/__init__.py"
    click LocalFiles "https://github.com/jabardigitalservice/DataSae/blob/main/datasae/converter/local.py"
    click GSheet "https://github.com/jabardigitalservice/DataSae/blob/main/datasae/converter/gsheet.py"
    click S3 "https://github.com/jabardigitalservice/DataSae/blob/main/datasae/converter/s3.py"
    click SQL "https://github.com/jabardigitalservice/DataSae/blob/main/datasae/converter/sql.py"
    click ConfigFiles "https://github.com/jabardigitalservice/DataSae/tree/main/tests/data"
    
    %% Legend
    subgraph Legend
        L1["Core Components"]:::core
        L2[("Data Sources")]:::datasource
        L3["Interfaces"]:::interface
        L4["External Services"]:::external
        L5["Configuration"]:::config
    end
```
