# ggiannarakis/causal-jobs architecture

> Architecture diagram of the GitHub repository ggiannarakis/causal-jobs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ggiannarakis/causal-jobs
- Repository: https://github.com/ggiannarakis/causal-jobs

## Overview

##### Main components
- Email Source and Gmail API Integration – where the raw job alert emails reside. Note that the extract.py script reads emails from the Gmail account’s “causal-jobs” folder.
- ETL Pipeline:
– Extraction: represented by extract.py which connects to the Gmail API.
– Transformation: captured by transform.py which converts raw email data into a structured Pandas dataframe.
– Loading: load.py which pushes the transformed data into the PostgreSQL database.
- Data Store – the PostgreSQL database running inside a Docker container (including the one table that holds all job alerts).
- Analysis Component – Analysis.ipynb (Jupyter Notebook) which runs the data analysis and generates a fresh report.
- Deployment and Reporting – index.html generated by the analysis, pushed via the causal-analysis-weekly-powershell.ps1 (or equivalent Bash script) to GitHub Pages.
- Scheduling and Alerting – daily and weekly scheduling is achieved through PowerShell (or Cron) scripts. In addition, there is a mention of a potential send_email.py script that can be used for alerts when the pipeline fails.

##### Relationships and interactions
- The ETL Pipeline should then be shown populating the Dockerized PostgreSQL database.
- Finally, illustrate the flow from the Analysis component to the deployment stage (GitHub pages hosting index.html) using the weekly PowerShell/Bash script.

##### Architectural patterns and technologies
- The ETL pattern is a key design decision.
- You’re using Python-based scripting (extract.py, transform.py, load.py) integrating with the Gmail API.
- Docker is central to hosting the PostgreSQL database.
- GitHub Pages is used for static website hosting of the analysis report.
- Scheduling is implemented externally via system-level scripting (PowerShell/Bash) and Task Scheduler or Cron.
- Including the flow from raw email extraction through transformation and loading as a “pipeline” block to emphasize the separation of concerns.

##### Detail and logical grouping
- Distinguish between daily operations (ETL scheduled scripts) and weekly operations (analysis and index.html deployment).
- Configuration files like requirements.txt that help set up the environment, and mention the role that Docker plays (with the PostgreSQL container) if necessary.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Gmail API (Email Source)"]:::external
    I["Scheduler (causal-jobs-powershell.ps1)"]:::schedule

    subgraph "ETL Pipeline"
        B["Extract Script (extract.py)"]:::etl
        C["Transform Script (transform.py)"]:::etl
        D["Load Script (load.py)"]:::etl
    end

    E["Dockerized PostgreSQL Database"]:::db

    subgraph "Analysis & Reporting"
        F["Jupyter Notebook Analysis (Analysis.ipynb)"]:::analysis
        H["Deployment Script (causal-analysis-weekly-powershell.ps1)"]:::deploy
        G["Report Deployment (GitHub Pages - index.html)"]:::deploy
    end

    A -->|"ExtractsEmails"| B
    I -->|"TriggersETL"| B
    B -->|"TransformsData"| C
    C -->|"LoadsData"| D
    D -->|"StoresData"| E
    E -->|"ServesData"| F
    F -->|"GeneratesReport"| H
    H -->|"DeploysReport"| G

    click A "https://github.com/ggiannarakis/causal-jobs/blob/main/extract.py"
    click B "https://github.com/ggiannarakis/causal-jobs/blob/main/extract.py"
    click C "https://github.com/ggiannarakis/causal-jobs/blob/main/transform.py"
    click D "https://github.com/ggiannarakis/causal-jobs/blob/main/load.py"
    click F "https://github.com/ggiannarakis/causal-jobs/blob/main/Analysis.ipynb"
    click G "https://github.com/ggiannarakis/causal-jobs/blob/main/index.html"
    click H "https://github.com/ggiannarakis/causal-jobs/blob/main/causal-analysis-weekly-powershell.ps1"
    click I "https://github.com/ggiannarakis/causal-jobs/blob/main/causal-jobs-powershell.ps1"

    classDef external fill:#ADD8E6,stroke:#0000FF,stroke-width:2px;
    classDef etl fill:#FFD700,stroke:#DAA520,stroke-width:2px;
    classDef db fill:#90EE90,stroke:#228B22,stroke-width:2px;
    classDef analysis fill:#FFFACD,stroke:#DAA520,stroke-width:2px;
    classDef deploy fill:#E0FFFF,stroke:#20B2AA,stroke-width:2px;
    classDef schedule fill:#FFB6C1,stroke:#FF69B4,stroke-width:2px;
```
