How nchn471/tiki-recommender-etl-pipeline works
Project Type and Purpose
• This is an ETL pipeline for the Tiki Recommender System. It extracts raw data from Tiki’s API, transforms it (using Apache Spark), loads it into a data warehouse (PostgreSQL), and supports a recommendation system that serves data via a Streamlit web app along with dashboards (using Metabase). The README clearly explains the multiple layers (bronze, silver, gold, recommendation), along with the services/technologies used for data orchestration (Dagster), containerization (Docker), and object storage (MinIO).
Analyzing the File Structure
• Notice separate directories for:
• ETL pipeline assets and resources (in the etl_pipeline folder) where raw extraction (TikiCrawler), transformation functions, and layer-specific assets (bronze, silver, gold, recommendation) are located.
• Tests for the ETL pipeline (in etl_pipeline/etl_pipeline_tests).
• Docker images for various services (Dagster, Spark, Spark Notebook, Streamlit), indicating containerization of different system components.
• Dagster configuration (dagster_home) which supports orchestration and pipeline scheduling.
• Streamlit application directories under docker_images/streamlit, highlighting both the main app and a “cloud_app” variant.
• Also, there are configuration files (e.g., docker-compose.yaml, Makefile, psql_schemas.sql) that indicate deployment and database schema setup.
a. Main Components to Include:
• External Data Source: Tiki's API (data extraction point using TikiCrawler)
• Object Storage: MinIO (for storing raw JSON files and transformed Parquet files)
• ETL Orchestrator: Dagster (managing the sequence and dependencies between the ETL assets)
• Data Processing Engine: Apache Spark (used in transforming data layers – silver, gold, recommendation)
• Data Warehouse: PostgreSQL (final storage for analytics)
• Business Intelligence Tool: Metabase (for dashboard/analytics)
• Serving Application: Streamlit application (for real-time recommendations)
• Containerization: Docker (with separate images for Dagster, Spark, and Streamlit environments)
b. Relationships and Data Flows:
• Data flowing from Tiki’s API → extraction via TikiCrawler → saved as JSON in MinIO (bronze layer).
• Transformation stages: bronze → silver (where data is loaded as Spark DataFrames), silver → gold (further cleaning/optimization), and a dedicated recommendation layer (where collaborative and content-based filtering is applied).
• Flow from transformed data to the data warehouse (loading into PostgreSQL).
• Data warehouse to visualization components (Metabase) and to the Streamlit-based recommendation system.
• Orchestration process (Dagster) managing these assets and scheduling including container deployment flows via Docker.
c. Architectural Patterns and Design Principles:
• Layered Architecture: Separate layers for extraction (bronze), transformation (silver, gold, recommendation), and load (data warehouse).
• Containerized Microservices: Each major component (ETL orchestration, Spark processing, Streamlit serving) runs in its containerized environment, as indicated by individual Docker configurations.
• Modular and Extensible Design: Clearly separated modules for assets, resources, and transformation functions enhance single-responsibility and reusability.
• Data Processing Pipelines: Use of Spark for distributed processing and Dagster for orchestrating complex data flows.
d. Key Technologies and Frameworks:
• Dagster for orchestration (configuration files in dagster_home and related Docker images).
• Apache Spark for distributed processing (referenced in docker_images/spark and within ETL transformation code).
• MinIO for object storage.
• PostgreSQL as the warehousing database.
• Streamlit for building interactive user interfaces.
• Docker (with dedicated Dockerfiles) for containerized deployments.
• Additionally, mention libraries like Gensim and TF-IDF for recommendation processing.
Elements
• Use clear labels for each component (e.g., “Tiki API”, “MinIO – Bronze Storage”, “Spark – Silver Transformation”, “PostgreSQL – DWH”, “Streamlit – Recommendation System”, “Dagster Orchestration”).
• Data extraction flows from Tiki API to raw data storage.
• Transformation flows from bronze → silver → gold.
• Infrastructure containerization aspect (Docker) by possibly adding a background or border grouping the containerized services.
Final Advice
• Be as detailed as possible: separate out components into as many logical units as the file tree and README illustrate.