# grilled-swampert/crypto-data-pipeline architecture

> Architecture diagram of the GitHub repository grilled-swampert/crypto-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/grilled-swampert/crypto-data-pipeline
- Repository: https://github.com/grilled-swampert/crypto-data-pipeline

## Overview

##### Project type and purpose
- This is an Airflow-based ETL pipeline project that extracts cryptocurrency market data, runs a Spark transformation job against the raw data, and (presumably) loads the results into a relational store (PostgreSQL).
- It uses Docker and Docker Compose to containerize the Airflow components and dependencies (including a PostgreSQL JDBC driver).

##### File structure
- Dockerfile + docker-compose.yaml
– Define containers for Airflow services (scheduler, webserver, worker), a Postgres database, and any supporting infrastructure.
- airflow_jars/postgresql-42.6.0.jar
– JDBC driver for Airflow to connect and write to Postgres.
- dags/
– crypto_market_etl.py: the Airflow DAG that orchestrates the extraction of crypto data and submits a Spark job.
– spark_jobs/transform_market_data.py: a PySpark script that transforms the raw market data.

##### README
- The README is empty (“huh”), so all inferences must come from the file tree.

a. Main components
1. Data Source(s): External crypto market APIs
2. Airflow Platform (Scheduler, Webserver, Worker/Executor)
3. Airflow DAG Tasks
– Extraction Task (Python operator fetching API data)
– Spark‐Submit Task (Python operator launching transform_market_data.py)
4. Spark Engine (cluster or single‐node Spark inside a container)
5. Data Storage
– Staging (e.g., local volume or object store)
– Postgres database (target for transformed data)
6. Docker Network (orchestration via docker-compose)

b. Relationships and interactions
- Extraction Task → writes raw JSON/CSV → Staging storage
- Spark‐Submit Task → pulls raw data from Staging → runs transform_market_data.py on Spark engine
- Spark job → writes transformed data → Postgres (via JDBC jar)
- Airflow Scheduler → queues tasks → Worker executes operators
- All services communicate over the Docker‐Compose network

c. Architectural patterns & principles
- ETL pipeline organized as an Airflow DAG (directed acyclic graph)
- Separation of concerns: ingestion, transformation, loading
- Containerization for reproducibility and isolation
- Modular DAGs and externalized Spark jobs for maintainability

d. Key technologies/frameworks
- Apache Airflow (Python operators, DAGs)
- Apache Spark (PySpark script)
- PostgreSQL (JDBC driver)
- Docker & Docker Compose
- Python

## Diagram (Mermaid)

```mermaid
flowchart TB
    CryptoAPI["Crypto Market API"]:::external

    subgraph "Docker Compose Network"
        subgraph "Airflow (Orchestration)"
            Scheduler["Airflow Scheduler"]:::orchestration
            Worker["Airflow Worker"]:::orchestration
            Webserver["Airflow Webserver"]:::orchestration
            Scheduler --> Worker
            Scheduler -.-> Webserver
            Worker -->|triggers| ExtractionTask["Extract Task"]:::orchestration
            Worker -->|triggers| SparkSubmit["Spark-Submit Task"]:::orchestration
        end
        JAR["postgresql-42.6.0.jar"]:::storage
        Postgres["Postgres Database"]:::storage
    end

    CryptoAPI -->|fetch data| ExtractionTask
    ExtractionTask -->|writes raw data| Staging["Staging Storage"]:::storage
    SparkSubmit -->|reads raw data| Staging
    SparkSubmit -->|submits job| SparkEngine["Spark Engine"]:::compute
    SparkEngine -->|writes transformed data| Postgres

    DockerConfig["Dockerfile"]:::external
    DockerCompose["docker-compose.yaml"]:::external

    click DockerConfig "https://github.com/grilled-swampert/crypto-data-pipeline/tree/master/Dockerfile"
    click DockerCompose "https://github.com/grilled-swampert/crypto-data-pipeline/blob/master/docker-compose.yaml"
    click JAR "https://github.com/grilled-swampert/crypto-data-pipeline/blob/master/airflow_jars/postgresql-42.6.0.jar"
    click ExtractionTask "https://github.com/grilled-swampert/crypto-data-pipeline/blob/master/dags/crypto_market_etl.py"
    click SparkSubmit "https://github.com/grilled-swampert/crypto-data-pipeline/blob/master/dags/spark_jobs/transform_market_data.py"

    classDef orchestration fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef compute fill:#FFA500,stroke:#000,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#000,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#000,stroke-width:1px
```
