# fcrispcoach/airflow-bank-transactions architecture

> Architecture diagram of the GitHub repository fcrispcoach/airflow-bank-transactions, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fcrispcoach/airflow-bank-transactions
- Repository: https://github.com/fcrispcoach/airflow-bank-transactions

## Overview

##### Project type and purpose
- This is a data-engineering + ML pipeline orchestrated with Apache Airflow.
- Purpose: generate and ingest synthetic time-series and bank-transaction data, validate data quality, detect anomalies/fraud via ML (Isolation Forest & Prophet), store results in PostgreSQL, and serve interactive dashboards via Streamlit.

##### File structure
- Top-level:
- airflow.cfg, airflow.db, webserver_config.py → Airflow setup and metadata
- requirements.txt → dependencies
- check_reports.py → custom monitoring or report checker
- config/: Python modules for path and global configuration
- dags/: Airflow DAG definitions
- anomaly_detection.py
- bank_transactions_etl.py
- bank_transactions_validation.py
- plugins/: custom Airflow operators
- custom_operators/data_quality_operator.py (wraps Great Expectations and Cerberus checks)
- data/: raw JSON/CSV stores and data population scripts (synthetic generation)
- database/: SQL schema and sample-data population scripts for PostgreSQL
- models/: ML training scripts (Isolation Forest model)
- schemas/: Cerberus JSON schema definitions for transaction validation
- dashboard/: Streamlit app for interactive visualization
- tests/: unit tests for validation and ETL logic

##### Extract insights from the README
- Orchestration: Apache Airflow schedules and coordinates tasks
- Data generation: Faker + custom scripts for both time series and transactions
- Validation: Great Expectations (data quality) + Cerberus (schema)
- ML: Scikit-learn (Isolation Forest for fraud/anomaly) & Prophet (time-series forecasting)
- Storage: PostgreSQL for structured data
- Visualization: Streamlit dashboard

a. Main components
1. Data Generation
– Synthetic time-series script
– Faker-based transaction generator
2. Raw Data Storage
– Flat files (JSON/CSV in data/)
3. Orchestrator
– Apache Airflow (Scheduler + Webserver)
– DAGs: ETL, validation, anomaly detection
– Custom Operators (data_quality_operator)
4. Validation Layer
– Great Expectations checks
– Cerberus schema validation
5. Processing & ML
– ETL scripts (pandas)
– Model training (Isolation Forest, Prophet)
6. Database
– PostgreSQL (initialized by database/init_db.sql)
7. Dashboard
– Streamlit app (dashboard/app.py)
8. Supporting Modules
– config/paths.py, schemas/transaction_schema.py, tests/

b. Relationships & interactions
- Data generators → raw data files → Airflow DAG start
- Airflow Scheduler → runs DAG tasks in order
- Validation tasks → fail early or proceed to ETL
- ETL tasks → load cleansed data into PostgreSQL
- ML tasks → read from PostgreSQL, train model, write anomaly flags back
- Dashboard → reads from PostgreSQL or summary tables, renders charts

c. Architectural patterns & principles
– Modular DAGs in Airflow (separate validation, ETL, ML)
– Separation of concerns: generation vs. validation vs. processing vs. serving
– Operator extensibility (custom Airflow operators for quality checks)
– CI via tests/ for critical pipelines

d. Key technologies & frameworks
– Apache Airflow (orchestrator)
– pandas (data processing)
– Great Expectations, Cerberus (validation)
– scikit-learn, Prophet (ML)
– PostgreSQL (storage)
– Streamlit (visualization)
– Faker (synthetic data)

- Use clear, concise labels: “Synthetic Generator,” “Validation Operator,” “Isolation Forest Task,” “PostgreSQL”
- Orchestration (e.g., blue)
- Storage (e.g., green)
- Processing/ML (e.g., orange)
- Visualization (e.g., purple)
- Rectangles for scripts or services
- Hexagons for custom operators

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Generation Layer
    subgraph "Data Generation"
        TG["Synthetic Generator\n(data/populate_db.py)"]:::proc
    end
    click TG "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/data/populate_db.py"

    RawData["Raw Data Storage\n(data/)"]:::storage
    click RawData "https://github.com/fcrispcoach/airflow-bank-transactions/tree/main/data/"

    TG --> RawData

    %% Orchestration Layer
    subgraph "Orchestration"
        AF_CFG["Airflow Config\n(airflow.cfg\nwebserver_config.py)"]:::orches
        AF_DB["Airflow Metadata DB\n(airflow.db)"]:::orches
        DAG_VAL["Validation DAG\n(bank_transactions_validation.py)"]:::orches
        DAG_ETL["ETL DAG\n(bank_transactions_etl.py)"]:::orches
        DAG_ANO["Anomaly Detection DAG\n(anomaly_detection.py)"]:::orches
        OP["Data Quality Operator\n(data_quality_operator.py)"]:::op
    end
    click AF_CFG "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/airflow.cfg"
    click AF_CFG "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/webserver_config.py"
    click AF_DB "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/airflow.db"
    click DAG_VAL "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/dags/bank_transactions_validation.py"
    click DAG_ETL "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/dags/bank_transactions_etl.py"
    click DAG_ANO "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/dags/anomaly_detection.py"
    click OP "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/plugins/custom_operators/data_quality_operator.py"

    RawData --> DAG_VAL
    DAG_VAL -->|"valid"| DAG_ETL
    DAG_ETL --> DAG_ANO

    %% Supporting Modules for Validation
    CFG["Config Module\n(config/paths.py)"]:::support
    SCHEMA["Cerberus Schema\n(transaction_schema.py)"]:::support
    click CFG "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/config/paths.py"
    click SCHEMA "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/schemas/transaction_schema.py"

    CFG --> DAG_VAL
    SCHEMA --> DAG_VAL

    %% Database Initialization & Storage
    DBINIT["DB Init Script\n(database/init_db.sql)"]:::storage
    POPSAMP["Sample Data Population\n(database/populate_sample_data.py)"]:::storage
    click DBINIT "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/database/init_db.sql"
    click POPSAMP "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/database/populate_sample_data.py"

    DB["PostgreSQL\n(database)"]:::db

    DBINIT --> DB
    POPSAMP --> DB

    DAG_ETL --> DB
    DAG_ANO --> DB

    %% Processing & ML Layer
    ML["Model Training\n(models/train_model.py)"]:::ml
    click ML "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/models/train_model.py"

    DB --> ML
    ML --> DB

    %% Visualization Layer
    subgraph "Visualization"
        DASH["Streamlit App\n(dashboard/app.py)"]:::viz
    end
    click DASH "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/dashboard/app.py"
    DB --> DASH

    %% CI & Other Support
    TEST["Unit Tests\n(tests/test_bank_transactions.py)"]:::support
    REQ["Dependencies\n(requirements.txt)"]:::support
    CHKR["Reports Checker\n(check_reports.py)"]:::support

    click TEST "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/tests/test_bank_transactions.py"
    click REQ "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/requirements.txt"
    click CHKR "https://github.com/fcrispcoach/airflow-bank-transactions/blob/main/check_reports.py"

    %% Styles
    classDef orches fill:#cce5ff,stroke:#333,stroke-width:1px;
    classDef storage fill:#d4edda,stroke:#333,stroke-width:1px;
    classDef proc fill:#fff3cd,stroke:#333,stroke-width:1px;
    classDef op fill:#f8d7da,stroke:#333,stroke-width:1px,shape:hexagon;
    classDef support fill:#e2e3e5,stroke:#333,stroke-width:1px;
    classDef db fill:#c3e6cb,stroke:#333,stroke-width:1px,shape:cylinder;
    classDef ml fill:#ffe5b4,stroke:#333,stroke-width:1px;
    classDef viz fill:#d6b3ff,stroke:#333,stroke-width:1px;
```
