# victor-antoniassi/junior_data_analyst_test_01 architecture

> Architecture diagram of the GitHub repository victor-antoniassi/junior_data_analyst_test_01, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/victor-antoniassi/junior_data_analyst_test_01
- Repository: https://github.com/victor-antoniassi/junior_data_analyst_test_01

## Overview

##### Main Components

a) Data Sources Layer:
- External Google Drive CSV source
- Local CSV file storage (`data/BASE_DADOS.csv`)

b) ETL Pipeline Layer:
- Prefect Orchestration Server
- ETL Processing Script (`etl.py`)
- Data Transformation Components

c) Storage Layer:
- DuckDB Database (`db_games_sales.db`)
- Processed CSV (`games_sales.csv`)

d) Analysis Layer:
- Jupyter Notebook Environment
- Power BI Connection Point

**┌─────────────────┐**
- Data Sources   │ (Top Layer)
- ┬────────┘
- ↓
**┌─────────────────┐**
- ETL Pipeline   │ (Middle Layer)
- ┬────────┘
- ↓
**┌─────────────────┐**
- Storage      │ (Bottom Layer)
- ┬────────┘
- ↓
**┌─────────────────┐**
- Analysis     │ (Final Layer)
- ┘

##### Detailed Component Representation

a) Data Sources (Blue):
- ┌─────────────────┐
- Google Drive   │
- CSV File     │
- ┘

b) ETL Pipeline (Green):
- ┌─────────────────────────────┐
- Prefect Server         │
- ┌───────────────────────┐  │
- ETL Process       │  │
- (Python/etl.py)      │  │
- ┘  │
- ┘

c) Storage (Orange):
- ┌─────────────────┐ ┌─────────────────┐
- DuckDB      │ │   Processed     │
- Database     │ │      CSV        │
- ┘ └─────────────────┘

d) Analysis (Purple):
- ┌─────────────────┐ ┌─────────────────┐
- Jupyter      │ │    Power BI     │
- Notebook     │ │   Connection    │
- ┘ └─────────────────┘

##### Important Elements to Highlight

- Prefect's orchestration control over the ETL process
- Data transformation steps within the ETL process
- Dual output nature (DuckDB and CSV)
- Analysis tools' connections to their respective data sources

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and Classes
    classDef source fill:#b3d9ff,stroke:#4d94ff
    classDef etl fill:#90EE90,stroke:#3CB371
    classDef storage fill:#FFB366,stroke:#FF8C00
    classDef analysis fill:#DDA0DD,stroke:#9370DB
    classDef docs fill:#F0F0F0,stroke:#A9A9A9

    %% Data Sources Layer
    subgraph DataSources["Data Sources Layer"]
        GDrive["Google Drive CSV"]:::source
        LocalCSV["Local CSV Source"]:::source
    end
    click LocalCSV "https://github.com/victor-antoniassi/junior_data_analyst_test_01/blob/main/data/BASE_DADOS.csv"

    %% ETL Pipeline Layer
    subgraph ETLPipeline["ETL Pipeline Layer"]
        PrefectServer["Prefect Orchestration Server"]:::etl
        ETLScript["ETL Processing Script"]:::etl
        Dependencies["Dependencies Config"]:::etl
    end
    click ETLScript "https://github.com/victor-antoniassi/junior_data_analyst_test_01/blob/main/etl/etl.py"
    click Dependencies "https://github.com/victor-antoniassi/junior_data_analyst_test_01/blob/main/dependencies/requirements.txt"

    %% Storage Layer
    subgraph StorageLayer["Storage Layer"]
        DuckDB[(DuckDB Database)]:::storage
        ProcessedCSV["Processed CSV"]:::storage
    end
    click DuckDB "https://github.com/victor-antoniassi/junior_data_analyst_test_01/blob/main/data/db_games_sales.db"
    click ProcessedCSV "https://github.com/victor-antoniassi/junior_data_analyst_test_01/blob/main/data/games_sales.csv"

    %% Analysis Layer
    subgraph AnalysisLayer["Analysis Layer"]
        Jupyter["Jupyter Notebook"]:::analysis
        PowerBI["Power BI Connection"]:::analysis
    end
    click Jupyter "https://github.com/victor-antoniassi/junior_data_analyst_test_01/blob/main/notebook/analysis.ipynb"

    %% Documentation
    subgraph Docs["Documentation"]
        README["README"]:::docs
        TechSpecs["Technical Specifications"]:::docs
    end
    click README "https://github.com/victor-antoniassi/junior_data_analyst_test_01/blob/main/README.md"
    click TechSpecs "https://github.com/victor-antoniassi/junior_data_analyst_test_01/blob/main/technical_challenge_proposal.md"

    %% Connections
    GDrive --> ETLScript
    LocalCSV --> ETLScript
    PrefectServer -.-> ETLScript
    Dependencies -.-> ETLScript
    ETLScript --> DuckDB
    ETLScript --> ProcessedCSV
    DuckDB --> Jupyter
    DuckDB --> PowerBI
    ProcessedCSV --> Jupyter
    ProcessedCSV --> PowerBI

    %% Legend
    subgraph Legend
        LS["Data Source"]:::source
        LE["ETL Process"]:::etl
        LST["Storage"]:::storage
        LA["Analysis"]:::analysis
        LD["Documentation"]:::docs
    end
```
