# kuyu12/scytale-exercise architecture

> Architecture diagram of the GitHub repository kuyu12/scytale-exercise, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kuyu12/scytale-exercise
- Repository: https://github.com/kuyu12/scytale-exercise

## Overview

##### Main Components

a) Data Source Layer:
- GitHub API (external system)
- Connection with authentication (GITHUB_TOKEN)

b) ETL Pipeline Components:
- Extract Module (src/extract.py)
- Transform Module (src/transform.py)
- Load Module (src/load.py)

c) Storage Components:
- Raw Data Storage (JSON files)
- Processed Data Storage (Parquet files)

d) Processing Engine:
- Apache Spark cluster

**[GitHub API] → [Extract Module] → [Raw JSON Storage] → [Transform Module (PySpark)] → [Load Module] → [Parquet Storage]**

##### Additional Details

a) Infrastructure Layer:
- Docker container boundary
- Spark cluster configuration

b) Data Flow Annotations:
- JSON format for raw data
- Parquet format for processed data
- Schema transformation details

c) Control Flow:
- Error handling paths
- Logging paths
- Configuration management

##### Key Relationships to Highlight

- How the abstract provider pattern interfaces with the GitHub API
- Relationship between models and schemas
- Demonstrate how the ETL components interact with Spark
- Logging and error handling across all components

##### Technical Specifications to Note

- PySpark as the processing engine
- GitHub API as the data source
- JSON as intermediate storage
- Parquet as final storage
- Docker containerization
- Python 3.9+ runtime

This architecture should emphasize the modularity of the system, particularly how the provider pattern allows for potential extension to other git providers beyond GitHub, and how the ETL pipeline maintains clear separation between extraction, transformation, and loading phases.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph External
        GH[GitHub API]:::external
    end

    subgraph "Provider Layer"
        BP["Base Provider"]:::provider
        GP["Git Provider"]:::provider
        GHP["GitHub Provider"]:::provider
        
        BP --> GP
        GP --> GHP
    end

    subgraph "ETL Pipeline"
        ETL["ETL Orchestrator"]:::core
        EX["Extract Module"]:::core
        TR["Transform Module"]:::core
        LD["Load Module"]:::core
        
        ETL --> EX
        ETL --> TR
        ETL --> LD
    end

    subgraph "Data Models"
        RM["Repository Model"]:::model
        PM["Pull Request Model"]:::model
        RS["Repository Schema"]:::model
        PS["Pull Request Schema"]:::model
        
        RM --> RS
        PM --> PS
    end

    subgraph "Storage Layer"
        RD[("Raw JSON Data")]:::storage
        PD[("Processed Parquet Data")]:::storage
    end

    subgraph "Infrastructure"
        SPARK["Apache Spark Cluster"]:::infra
        DOCKER["Docker Container"]:::infra
    end

    subgraph "Utility Layer"
        LOG["Logging System"]:::util
        ERR["Error Handler"]:::util
        SU["Spark Utils"]:::util
    end

    %% Relationships
    GHP --> GH
    EX --> RD
    RD --> TR
    TR --> PD
    TR --> SPARK
    LD --> PD
    
    %% Utility Connections
    LOG -.-> ETL
    ERR -.-> ETL
    SU -.-> SPARK

    %% Click Events
    click EX "https://github.com/kuyu12/Scytale-exercise/blob/master/src/extract.py"
    click TR "https://github.com/kuyu12/Scytale-exercise/blob/master/src/transform.py"
    click LD "https://github.com/kuyu12/Scytale-exercise/blob/master/src/load.py"
    click BP "https://github.com/kuyu12/Scytale-exercise/blob/master/src/providers/abstract/base_provider.py"
    click GP "https://github.com/kuyu12/Scytale-exercise/blob/master/src/providers/abstract/git_provider.py"
    click GHP "https://github.com/kuyu12/Scytale-exercise/blob/master/src/providers/github_provider.py"
    click RM "https://github.com/kuyu12/Scytale-exercise/blob/master/src/model/repo_data_model.py"
    click PM "https://github.com/kuyu12/Scytale-exercise/blob/master/src/model/pull_request_data_model.py"
    click RS "https://github.com/kuyu12/Scytale-exercise/blob/master/src/model/schemas/repo_schema.py"
    click PS "https://github.com/kuyu12/Scytale-exercise/blob/master/src/model/schemas/pull_schema.py"
    click LOG "https://github.com/kuyu12/Scytale-exercise/blob/master/src/utils/logger_utils.py"
    click ERR "https://github.com/kuyu12/Scytale-exercise/blob/master/src/utils/errors.py"
    click SU "https://github.com/kuyu12/Scytale-exercise/blob/master/src/utils/spark_utils.py"
    click ETL "https://github.com/kuyu12/Scytale-exercise/blob/master/src/etl.py"
    click RD "https://github.com/kuyu12/Scytale-exercise/tree/master/data/"
    click DOCKER "https://github.com/kuyu12/Scytale-exercise/tree/master/src/dockerfile"

    %% Styling
    classDef external fill:#87CEEB
    classDef provider fill:#FFB6C1
    classDef core fill:#98FB98
    classDef model fill:#DDA0DD
    classDef storage fill:#F0E68C
    classDef infra fill:#D3D3D3
    classDef util fill:#E6E6FA
```
