# nourhene-azaiez/bi-payment-tracker architecture

> Architecture diagram of the GitHub repository nourhene-azaiez/bi-payment-tracker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nourhene-azaiez/bi-payment-tracker
- Repository: https://github.com/nourhene-azaiez/bi-payment-tracker

## Overview

##### Project type and purpose
- This is a Talend Data Integration (“bi-payment-tracker”) ETL project. It contains Talend Jobs, Joblets, Routines, and Metadata definitions to extract data from various sources, transform it, and load into target analytic stores (e.g., Snowflake, DeltaLake).

##### File structure
- Top-level directories:
– code/beans, code/routines, code/snippets: custom Java code and reusable logic
– metadata/: connection definitions (Snowflake, Excel, JDBC, file schemas, rules)
– joblets/: small reusable sub-processes (EKHCONTR, EKHCUST, etc.)
– process/: main Talend Jobs (Agent_jobs, STAGING_JOB, TEST_0.1)
– sqlPatterns/: library of SQL templates for multiple engines (Snowflake, Hive, MySQL, etc.)
– context/: environment variables (dev/test/prod)
– components/, routes/: custom components and message routing
- Configuration files: talend.project, .settings, .gitignore.

##### README
- Minimal README; core insight comes from directory names.

##### Diagramming the architecture
a. Main components
– Data Sources: Excel files, CSV logs, JDBC sources, Salesforce, FTP, JSON/XML files
– Talend Studio (Design-time): holds Jobs, Joblets, Routines, Metadata, SQL Patterns
– Talend Runtime/Agent: orchestrates and executes exported Jobs
– Metadata Repository: holds connection specs and schema definitions
– Reusable Libraries: code routines (Java), beans, SQL pattern templates
– Staging Layer: temporary tables in Snowflake or DeltaLake
– Data Warehouse Layer: final tables in Snowflake (or alternate engines)
– Execution Context: environment-specific contexts (connections, credentials)

b. Relationships & interactions
– Talend Runtime triggers Agent_jobs (via scheduler/cron)
– Agent_jobs call individual Jobs in process/
– Each Job reads Metadata definitions to connect to sources, invokes Joblets for standardized transformations, uses Routines/Beans for custom logic
– Jobs generate SQL via sqlPatterns/ tailored per target engine
– Transformed data is written to Staging tables, then to final Data Warehouse tables

c. Architectural patterns & principles
– Metadata-driven ETL: central metadata folder enables consistent connection and schema reuse
– Component reuse: Joblets encapsulate frequently used sub-processes (e.g., payment header parsing)
– Template-based SQL: sqlPatterns/ ensures consistent SQL generation across multiple databases
– Layered pipeline: ingest → staging → final

d. Key technologies
– Talend Open Studio / DI Runtime
– Snowflake / DeltaLake / Hive / MySQL (multi-engine support via sqlPatterns)
– Java (Routines & Beans)
– Context variables for environment configurability

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Sources
    subgraph "Data Sources"
        direction TB
        DS_Excel["Excel File"]:::source
        DS_CSV["CSV Log File"]:::source
        DS_JDBC["JDBC Source"]:::source
        DS_SF["Salesforce API"]:::source
        DS_FTP["FTP Source"]:::source
        DS_JSON["JSON/XML Files"]:::source
    end

    %% Execution and ETL
    subgraph "ETL Layer"
        direction TB
        TalendRuntime["Talend Runtime<br/>(Agent)"]:::runtime
        AgentJobs["Agent_jobs"]:::job
        TalendStudio["Talend Studio<br/>(Design-time)"]:::studio

        subgraph "Talend Project Artifacts"
            direction TB
            Jobs["Talend Jobs"]:::job
            STAGINGJob["STAGING_JOB_0.1"]:::job
            TESTJob["TEST_0.1"]:::job
            Joblets["Joblets"]:::joblet
            Routines["Routines"]:::lib
            Beans["Beans"]:::lib
            Snippets["Snippets"]:::lib
            SQLPatterns["SQL Patterns"]:::lib
            MetadataRepo["Metadata Repository"]:::lib
            ContextVars["Context Variables"]:::lib
            Components["Custom Components"]:::lib
            Routes["Routes"]:::lib
        end
    end

    %% Data Warehouse Targets
    subgraph "Data Warehouse Layer"
        direction TB
        StagingDB["Staging Layer<br/>(Snowflake/DeltaLake)"]:::db
        FinalDB["Final DW<br/>(Snowflake)"]:::db
    end

    %% Flows
    DS_Excel -->|reads| AgentJobs
    DS_CSV -->|reads| AgentJobs
    DS_JDBC -->|reads| AgentJobs
    DS_SF -->|reads| AgentJobs
    DS_FTP -->|reads| AgentJobs
    DS_JSON -->|reads| AgentJobs

    TalendStudio -->|publishes jobs| TalendRuntime
    TalendRuntime -->|triggers| AgentJobs
    AgentJobs -->|calls| Jobs
    AgentJobs -->|calls| STAGINGJob
    AgentJobs -->|calls| TESTJob
    Jobs -->|uses| Joblets
    Jobs -->|invokes| Routines
    Jobs -->|uses| Beans
    Jobs -->|templates via| SQLPatterns
    Jobs -->|fetches metadata| MetadataRepo
    Jobs -->|applies| ContextVars
    Jobs -->|extends| Components
    Jobs -->|routes| Routes

    Jobs -->|writes to| StagingDB
    STAGINGJob -->|loads to| StagingDB
    StagingDB -->|promotes to| FinalDB

    %% Legend
    Legend

    %% Click Events
    click AgentJobs "https://github.com/nourhene-azaiez/bi-payment-tracker/tree/main/BI_PAYMENT_TRACKER/process/Agent_jobs/"
    click Jobs "https://github.com/nourhene-azaiez/bi-payment-tracker/tree/main/BI_PAYMENT_TRACKER/process/"
    click STAGINGJob "https://github.com/nourhene-azaiez/bi-payment-tracker/blob/main/BI_PAYMENT_TRACKER/process/STAGING_JOB_0.1.*"
    click TESTJob "https://github.com/nourhene-azaiez/bi-payment-tracker/blob/main/BI_PAYMENT_TRACKER/process/TEST_0.1.*"
    click Joblets "https://github.com/nourhene-azaiez/bi-payment-tracker/tree/main/BI_PAYMENT_TRACKER/joblets/"
    click Routines "https://github.com/nourhene-azaiez/bi-payment-tracker/tree/main/BI_PAYMENT_TRACKER/code/routines/"
    click Beans "https://github.com/nourhene-azaiez/bi-payment-tracker/tree/main/BI_PAYMENT_TRACKER/code/beans/"
    click Snippets "https://github.com/nourhene-azaiez/bi-payment-tracker/tree/main/BI_PAYMENT_TRACKER/code/snippets/"
    click SQLPatterns "https://github.com/nourhene-azaiez/bi-payment-tracker/tree/main/BI_PAYMENT_TRACKER/sqlPatterns/"
    click MetadataRepo "https://github.com/nourhene-azaiez/bi-payment-tracker/tree/main/BI_PAYMENT_TRACKER/metadata/"
    click ContextVars "https://github.com/nourhene-azaiez/bi-payment-tracker/tree/main/BI_PAYMENT_TRACKER/context/"
    click Components "https://github.com/nourhene-azaiez/bi-payment-tracker/tree/main/BI_PAYMENT_TRACKER/components/"
    click Routes "https://github.com/nourhene-azaiez/bi-payment-tracker/tree/main/BI_PAYMENT_TRACKER/routes/"

    %% Styles
    classDef source fill:#ccffcc,shape:parallelogram,stroke:#333,stroke-width:1px
    classDef runtime fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef studio fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef job fill:#99ccff,stroke:#333,stroke-width:1px
    classDef joblet fill:#99ccff,stroke:#333,stroke-width:1px
    classDef lib fill:#eeeeee,stroke:#666,stroke-width:1px
    classDef db fill:#ffcc99,shape:cylinder,stroke:#333,stroke-width:1px
    classDef legend fill:#ffffff,stroke:none,font-style:italic,font-size:10px
```
