How grilled-swampert/crypto-data-pipeline works
Project type and purpose
• This is an Airflow-based ETL pipeline project that extracts cryptocurrency market data, runs a Spark transformation job against the raw data, and (presumably) loads the results into a relational store (PostgreSQL).
• It uses Docker and Docker Compose to containerize the Airflow components and dependencies (including a PostgreSQL JDBC driver).
File structure
• Dockerfile + docker-compose.yaml
– Define containers for Airflow services (scheduler, webserver, worker), a Postgres database, and any supporting infrastructure.
• airflow_jars/postgresql-42.6.0.jar
– JDBC driver for Airflow to connect and write to Postgres.
• dags/
– crypto_market_etl.py: the Airflow DAG that orchestrates the extraction of crypto data and submits a Spark job.
– spark_jobs/transform_market_data.py: a PySpark script that transforms the raw market data.
README
• The README is empty (“huh”), so all inferences must come from the file tree.
a. Main components
1. Data Source(s): External crypto market APIs
2. Airflow Platform (Scheduler, Webserver, Worker/Executor)
3. Airflow DAG Tasks
– Extraction Task (Python operator fetching API data)
– Spark‐Submit Task (Python operator launching transform_market_data.py)
4. Spark Engine (cluster or single‐node Spark inside a container)
5. Data Storage
– Staging (e.g., local volume or object store)
– Postgres database (target for transformed data)
6. Docker Network (orchestration via docker-compose)
b. Relationships and interactions
• Extraction Task → writes raw JSON/CSV → Staging storage
• Spark‐Submit Task → pulls raw data from Staging → runs transform_market_data.py on Spark engine
• Spark job → writes transformed data → Postgres (via JDBC jar)
• Airflow Scheduler → queues tasks → Worker executes operators
• All services communicate over the Docker‐Compose network
c. Architectural patterns & principles
• ETL pipeline organized as an Airflow DAG (directed acyclic graph)
• Separation of concerns: ingestion, transformation, loading
• Containerization for reproducibility and isolation
• Modular DAGs and externalized Spark jobs for maintainability
d. Key technologies/frameworks
• Apache Airflow (Python operators, DAGs)
• Apache Spark (PySpark script)
• PostgreSQL (JDBC driver)
• Docker & Docker Compose
• Python