# skalskibukowa/project-spark-scala-etl architecture

> Architecture diagram of the GitHub repository skalskibukowa/project-spark-scala-etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/skalskibukowa/project-spark-scala-etl
- Repository: https://github.com/skalskibukowa/project-spark-scala-etl

## Overview

##### Main Components
- Data Sources:
- CSV files found in the Datasource folder.
- PostgreSQL databases (with Dockerized PostgreSQL setup in the PostgreSQL directory).
- Data Ingestion Layer:
- The Ingestions package (e.g., TransactionIngests.scala) that initiates the extraction of data.
- Data Cleaning and Transformation Layers:
- The cleaners package (e.g., TweetsCleaner, ForbesCleaner, TransactionCleaner, TweetsUserCleaner) which pre-process data.
- The analysers package (e.g., TransactionAnalyzer, TweetsAnalyzer, TweetsUserAnalyzer, ForbesAnalyzer, TransactionSearch) that perform further transformation, aggregation, or complex analysis.
- Data Loaders (Sink):
- The loaders package (e.g., ForbesLoader, TweetsLoader, TweetsUserLoader, TransactionLoader) that handle output to CSV, Parquet, or PostgreSQL.
- Supporting Components:
- The Docker setup (Dockerfile and docker-compose.yml) for running PostgreSQL.
- Build and dependency management with build.sbt and sbt configuration files.
- Unit tests (for verification) located under src/test.

##### Relationships and Data Flow
- Data flows from the sources (CSV files and PostgreSQL) into the ingestion layer.
- The ingested data is then passed to the cleaning modules to ensure quality.
- Clean data moves on to the analytical layer, where transformations and computations are applied.
- Finally, the processed data is passed to the loaders, which write the output into various sinks.
- External interactions such as Docker containers for PostgreSQL and any potential API endpoints if they exist.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Clearly divide the ETL process into ingestion, cleaning, analysis, and loading stages.
- Modular Structure: Each functionality is encapsulated within its respective package, which aids in both maintainability and testing.
- Use of Docker for Database Environment: This enhances environment consistency and ease of deployment.
- Clear Data Pipeline Flow: Ensure that transformations follow a logical progression from extraction to loading.

##### Practical Steps for the Principal Software Engineer
- Zoom into each processing block (for Ingestion, Cleaning, Analysis, and Loading) if more detail is needed.
- That all components are clearly labeled with their package or module names (as evident in the file tree).
- Dependencies such as the transformation logic available in the analysers and cleaners, and the module responsible for interacting with external data sinks.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Sources
    subgraph "Data Sources"
        DS1("CSV Files"):::sources
        DS2("PostgreSQL Setup"):::sources
    end

    %% ETL Pipeline Components
    subgraph "ETL Pipeline"
        ING("Ingestion: Transaction Ingests"):::ingestion
        CLEAN("Data Cleaning (Cleaners)"):::cleaning
        ANALYSIS("Data Analysis (Analysers)"):::analysis
        LOAD("Data Loaders (Sinks)"):::loaders
    end

    %% Supporting Components
    subgraph "Supporting Components"
        BUILD("Build & Dependencies"):::support
        DOCKER("Docker Setup"):::support
        TEST("Unit Testing"):::support
    end

    %% Data Flow Connections
    DS1 -->|"data"| ING
    DS2 -->|"data"| ING
    ING -->|"transforms"| CLEAN
    CLEAN -->|"analyzes"| ANALYSIS
    ANALYSIS -->|"outputs"| LOAD

    %% Supporting Connections
    DOCKER -->|"provisions"| DS2

    %% Click events
    click DS1 "https://github.com/skalskibukowa/project-spark-scala-etl/tree/master/Datasource/"
    click DS2 "https://github.com/skalskibukowa/project-spark-scala-etl/tree/master/PostgreSQL/"
    click ING "https://github.com/skalskibukowa/project-spark-scala-etl/blob/master/src/main/scala/Ingestions/TransactionIngests.scala"
    click CLEAN "https://github.com/skalskibukowa/project-spark-scala-etl/tree/master/src/main/scala/cleaners"
    click ANALYSIS "https://github.com/skalskibukowa/project-spark-scala-etl/tree/master/src/main/scala/analysers"
    click LOAD "https://github.com/skalskibukowa/project-spark-scala-etl/tree/master/src/main/scala/loaders"
    click BUILD "https://github.com/skalskibukowa/project-spark-scala-etl/blob/master/build.sbt"
    click DOCKER "https://github.com/skalskibukowa/project-spark-scala-etl/blob/master/docker-compose.yml"
    click TEST "https://github.com/skalskibukowa/project-spark-scala-etl/tree/master/src/test/scala/UnitTests/"

    %% Styles
    classDef sources fill:#FFCC00,stroke:#333,stroke-width:2px;
    classDef ingestion fill:#99CCFF,stroke:#333,stroke-width:2px;
    classDef cleaning fill:#BBDEFB,stroke:#333,stroke-width:2px;
    classDef analysis fill:#C8E6C9,stroke:#333,stroke-width:2px;
    classDef loaders fill:#FFCDD2,stroke:#333,stroke-width:2px;
    classDef support fill:#E0E0E0,stroke:#333,stroke-width:2px;
```
