# le-oasis/docker-airflow-spark architecture

> Architecture diagram of the GitHub repository le-oasis/docker-airflow-spark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/le-oasis/docker-airflow-spark
- Repository: https://github.com/le-oasis/docker-airflow-spark

## Overview

##### Main Components
- Docker Containers – The project uses Docker Compose (with multiple compose files) to spin up services. Each service should be represented as a container along with its role.
- Data Orchestration – Airflow is used to define and schedule ETL DAGs including tasks that interact with PostgreSQL (via PostgresOperator) and Spark (via Spark Operators).
- Data Processing – Apache Spark is employed in two ways:
- A distributed Spark cluster is set up (with a master and worker nodes) for large-scale data transformations.
- Dedicated Spark jobs (e.g., SparkOperatorDemo, spark-test, and data transformation applications) are provided in the directories.
- Object Storage – MinIO is used as scalable, S3-compatible object storage.
- Database – PostgreSQL is configured for both Airflow’s backend and as a target/source for data loading operations.
- Additional Components – Jupyter notebooks for interactive data exploration, Trino as the SQL query engine for distributed queries, and Ranger for security configuration in the Hadoop ecosystem. Also, Hive is referenced in the postgres/hive directory.
- Auxiliary Services – Redis, which is also included (as per the Docker image table), might be used for caching or message brokering.

##### Component Relationships & Data Flow
- Airflow serves as the scheduler and orchestration layer, triggering DAGs that execute SQL commands on PostgreSQL and submit Spark jobs.
- Spark jobs process data loaded from various sources:
- They may ingest raw data from MinIO.
- They potentially read from and write to PostgreSQL.
- They incorporate transformation logic that can be triggered from Airflow.
- Jupyter notebooks provide an interactive front-end for data exploration and testing spark and SQL workflows.
- Trino and Hive enhance query capabilities across the lakehouse infrastructure, running on a dedicated container with MinIO and Hive metastore integration.
- Docker Compose/Swarm is used to interconnect these services, ensuring proper network resolution with user-defined Docker networks.
- Ranger is present to enforce security policies on data access, particularly in the context of Hadoop/Spark and possibly Hive.

##### Architectural Patterns & Technologies
- Microservices Containerization – Each service (Airflow, Spark (master & worker), MinIO, PostgreSQL, Redis, Jupyter, Trino) operates as an independent Docker container.
- ETL Pipeline Pattern – The flow is divided in Extraction (data ingestion from MinIO or other sources), Loading (into PostgreSQL), and Transform (using Spark).
- Distributed Data Processing – Using Spark’s distributed model and Trino for query federation over large datasets.
- Inter-Service Communication – Leveraging Docker’s user-defined bridge networks that allow containers to communicate via DNS.
- Extensibility – The file tree shows modular directories for DAGs (Airflow) and Spark applications, indicating a design that promotes evolving pipelines.

##### Final Advice
- Validate that every service mentioned in the README and seen in the file tree is represented where it logically fits in the architecture.
- That you denote the orchestration (Airflow) and the execution of data transformation jobs (Spark) distinctly, as well as their interactions with storage (MinIO, PostgreSQL) and interactive tools (Jupyter, Trino).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        DOCKER["Docker Compose (Containers)"]:::infrastructure
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        AIRFLOW["Airflow - DAG Orchestrator"]:::orchestration
    end

    %% Processing Layer
    subgraph "Processing Layer"
        SPARK["Apache Spark Cluster"]:::processing
        SPARK_MASTER["Spark Master"]:::processing
        SPARK_WORKER["Spark Worker"]:::processing
        SPARK -->|"includes"| SPARK_MASTER
        SPARK -->|"includes"| SPARK_WORKER
    end

    %% Storage Layer
    subgraph "Storage Layer"
        MINIO["MinIO - Object Storage"]:::storage
        POSTGRES["PostgreSQL - Database"]:::storage
        HIVE["Hive Metastore"]:::storage
        REDIS["Redis - Cache/Messaging"]:::storage
    end

    %% Query & Analysis Layer
    subgraph "Query & Analysis Layer"
        JUPYTER["Jupyter Notebook"]:::analysis
        TRINO["Trino Coordinator"]:::analysis
    end

    %% Security/Management Layer
    subgraph "Security/Management"
        RANGER["Ranger - Security"]:::security
    end

    %% Data Flow Connections
    DOCKER -->|"deploys"| AIRFLOW
    DOCKER -->|"deploys"| SPARK
    DOCKER -->|"deploys"| MINIO
    DOCKER -->|"deploys"| POSTGRES
    DOCKER -->|"deploys"| JUPYTER
    DOCKER -->|"deploys"| TRINO
    DOCKER -->|"deploys"| RANGER
    DOCKER -->|"deploys"| REDIS

    AIRFLOW -->|"triggers"| SPARK
    AIRFLOW -->|"manages"| POSTGRES

    SPARK -->|"reads/writes"| MINIO
    SPARK -->|"reads/writes"| POSTGRES

    JUPYTER -->|"queries"| POSTGRES
    JUPYTER -->|"submits"| SPARK
    JUPYTER -->|"accesses"| MINIO

    TRINO -->|"queries"| HIVE
    TRINO -->|"accesses"| MINIO

    RANGER -->|"enforces"| POSTGRES
    RANGER -->|"enforces"| HIVE
    RANGER -->|"enforces"| SPARK

    %% Click Events
    click DOCKER "https://github.com/le-oasis/docker-airflow-spark/blob/main/docker-compose.yml"
    click AIRFLOW "https://github.com/le-oasis/docker-airflow-spark/tree/main/docker-airflow/"
    click SPARK "https://github.com/le-oasis/docker-airflow-spark/tree/main/spark/"
    click MINIO "https://github.com/le-oasis/docker-airflow-spark/tree/main/minio/"
    click POSTGRES "https://github.com/le-oasis/docker-airflow-spark/tree/main/postgres/"
    click JUPYTER "https://github.com/le-oasis/docker-airflow-spark/tree/main/jupyter/"
    click TRINO "https://github.com/le-oasis/docker-airflow-spark/tree/main/trino/coordinator"
    click HIVE "https://github.com/le-oasis/docker-airflow-spark/tree/main/postgres/hive/"
    click RANGER "https://github.com/le-oasis/docker-airflow-spark/tree/main/ranger/"

    %% Styles
    classDef infrastructure fill:#f9e79f,stroke:#b7950b,stroke-width:2px;
    classDef orchestration fill:#aed6f1,stroke:#2980b9,stroke-width:2px;
    classDef processing fill:#a9dfbf,stroke:#27ae60,stroke-width:2px;
    classDef storage fill:#fadbd8,stroke:#e74c3c,stroke-width:2px;
    classDef analysis fill:#d2b4de,stroke:#8e44ad,stroke-width:2px;
    classDef security fill:#fcf3cf,stroke:#f1c40f,stroke-width:2px;
```
