How le-oasis/docker-airflow-spark works
Main Components
• Docker Containers – The project uses Docker Compose (with multiple compose files) to spin up services. Each service should be represented as a container along with its role.
• Data Orchestration – Airflow is used to define and schedule ETL DAGs including tasks that interact with PostgreSQL (via PostgresOperator) and Spark (via Spark Operators).
• Data Processing – Apache Spark is employed in two ways:
• A distributed Spark cluster is set up (with a master and worker nodes) for large-scale data transformations.
• Dedicated Spark jobs (e.g., SparkOperatorDemo, spark-test, and data transformation applications) are provided in the directories.
• Object Storage – MinIO is used as scalable, S3-compatible object storage.
• Database – PostgreSQL is configured for both Airflow’s backend and as a target/source for data loading operations.
• Additional Components – Jupyter notebooks for interactive data exploration, Trino as the SQL query engine for distributed queries, and Ranger for security configuration in the Hadoop ecosystem. Also, Hive is referenced in the postgres/hive directory.
• Auxiliary Services – Redis, which is also included (as per the Docker image table), might be used for caching or message brokering.
Component Relationships & Data Flow
• Airflow serves as the scheduler and orchestration layer, triggering DAGs that execute SQL commands on PostgreSQL and submit Spark jobs.
• Spark jobs process data loaded from various sources:
• They may ingest raw data from MinIO.
• They potentially read from and write to PostgreSQL.
• They incorporate transformation logic that can be triggered from Airflow.
• Jupyter notebooks provide an interactive front-end for data exploration and testing spark and SQL workflows.
• Trino and Hive enhance query capabilities across the lakehouse infrastructure, running on a dedicated container with MinIO and Hive metastore integration.
• Docker Compose/Swarm is used to interconnect these services, ensuring proper network resolution with user-defined Docker networks.
• Ranger is present to enforce security policies on data access, particularly in the context of Hadoop/Spark and possibly Hive.
Architectural Patterns & Technologies
• Microservices Containerization – Each service (Airflow, Spark (master & worker), MinIO, PostgreSQL, Redis, Jupyter, Trino) operates as an independent Docker container.
• ETL Pipeline Pattern – The flow is divided in Extraction (data ingestion from MinIO or other sources), Loading (into PostgreSQL), and Transform (using Spark).
• Distributed Data Processing – Using Spark’s distributed model and Trino for query federation over large datasets.
• Inter-Service Communication – Leveraging Docker’s user-defined bridge networks that allow containers to communicate via DNS.
• Extensibility – The file tree shows modular directories for DAGs (Airflow) and Spark applications, indicating a design that promotes evolving pipelines.
Final Advice
• Validate that every service mentioned in the README and seen in the file tree is represented where it logically fits in the architecture.
• That you denote the orchestration (Airflow) and the execution of data transformation jobs (Spark) distinctly, as well as their interactions with storage (MinIO, PostgreSQL) and interactive tools (Jupyter, Trino).