How tmph2003/doantotnghiep works
Project Type and Core Purpose
• This is a full-fledged data warehouse solution that uses Airflow to orchestrate ETL pipelines. The solution uses several containerized services (Airflow, Postgres, Redis [as mentioned in the README], Trino, ClickHouse, Superset, Hive Metastore, etc.) to manage workflow orchestration, metadata, ETL processing, analytical querying, and data visualization.
• That there is also a “warehouse” component implemented in custom ETL Python modules (in the plugins/warehouse folder) that encapsulate data extraction, DDL management, and transformation logic.
File Structure for Architectural Insights
• Top-Level Folders:
• containers: Contains the Dockerfiles and configurations for all major services such as Airflow, Postgres, ClickHouse, Superset, Hive-Metastore, and Trino.
• dags: Contains Airflow DAG definitions, including ETL workflows and data orchestration.
• plugins: Offers custom Python modules for data transformation and schema (ETL) logic that Airflow tasks execute.
• Assets (like CSV files) that may be imported into DAGs.
• data: Contains persistent storage/data files for the services, notably the Postgres database and potentially MinIO (indicated by .minio sys subfolders).
• The Docker Compose YAML files (e.g., docker-compose-bi.yaml, docker-compose-compute.yaml) illustrate the orchestration of these services.
Component Identification
a. Container Orchestrator Environment
• Docker and Docker Compose (the foundational layer for running containers).
• Each containerized service:
• Airflow: further broken down into its three core services – webserver, scheduler, and worker – as well as auxiliary services (e.g., airflow-init, triggered, log grooming).
• Postgres: Used primarily for Airflow metadata (and possibly as a component in the warehouse).
• Redis: Acts as the message broker for Airflow scheduler (as mentioned in the README; although not explicitly seen in the file tree, it is an integral element).
• Trino: A distributed SQL query engine for querying data across multiple sources.
• ClickHouse: Likely used as an analytical database.
• Superset: For data visualization and dashboarding.
• Hive-Metastore: Supporting metadata for data processing.
b. Data Processing and ETL:
• Airflow DAGs located in the “dags” directory that orchestrate task execution.
• Custom ETL modules in the “plugins/warehouse/etl” directory, including separate logic for “bronze” (raw/extraction) and “silver” (transformation/cleaning) layers.
c. Data Repositories:
• Persistent “data” folders used for storing Postgres data and other files (e.g., MinIO for a data lake structure).
• External systems (if any) that feed into or extract from the warehouse (for trading data input) are also represented.
Relationships and Interactions
• How Airflow—as the orchestrator—kicks off ETL tasks (invoking modules in “plugins”) which load/extract data from the “include” and “data” storage areas.
• How Airflow interacts with the Postgres container (for metadata) and the Redis broker.
• How transformed data might then be queried by Trino or loaded into ClickHouse for analytics.
• How Superset connects to these databases (e.g., ClickHouse/Trino) for visualization.
• Bidirectional flows if needed (for example, logging data from Airflow flows back to persistent storage).
Architectural Patterns and Design Principles
• A microservices or containerized architecture: each service is self-contained but interacts through well-defined interfaces.
• Separation of concerns: workflow orchestration (Airflow), data storage (Postgres, ClickHouse), query engine (Trino), and presentation (Superset).
• Extensibility points: custom ETL logic in plugins and modular DAG definitions enable flexible future enhancements.