# nchn471/tiki-recommender-etl-pipeline architecture

> Architecture diagram of the GitHub repository nchn471/tiki-recommender-etl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nchn471/tiki-recommender-etl-pipeline
- Repository: https://github.com/nchn471/tiki-recommender-etl-pipeline

## Overview

##### Project Type and Purpose
- This is an ETL pipeline for the Tiki Recommender System. It extracts raw data from Tiki’s API, transforms it (using Apache Spark), loads it into a data warehouse (PostgreSQL), and supports a recommendation system that serves data via a Streamlit web app along with dashboards (using Metabase). The README clearly explains the multiple layers (bronze, silver, gold, recommendation), along with the services/technologies used for data orchestration (Dagster), containerization (Docker), and object storage (MinIO).

##### Analyzing the File Structure
- Notice separate directories for:
- ETL pipeline assets and resources (in the etl_pipeline folder) where raw extraction (TikiCrawler), transformation functions, and layer-specific assets (bronze, silver, gold, recommendation) are located.
- Tests for the ETL pipeline (in etl_pipeline/etl_pipeline_tests).
- Docker images for various services (Dagster, Spark, Spark Notebook, Streamlit), indicating containerization of different system components.
- Dagster configuration (dagster_home) which supports orchestration and pipeline scheduling.
- Streamlit application directories under docker_images/streamlit, highlighting both the main app and a “cloud_app” variant.
- Also, there are configuration files (e.g., docker-compose.yaml, Makefile, psql_schemas.sql) that indicate deployment and database schema setup.

a. Main Components to Include:
- External Data Source: Tiki's API (data extraction point using TikiCrawler)
- Object Storage: MinIO (for storing raw JSON files and transformed Parquet files)
- ETL Orchestrator: Dagster (managing the sequence and dependencies between the ETL assets)
- Data Processing Engine: Apache Spark (used in transforming data layers – silver, gold, recommendation)
- Data Warehouse: PostgreSQL (final storage for analytics)
- Business Intelligence Tool: Metabase (for dashboard/analytics)
- Serving Application: Streamlit application (for real-time recommendations)
- Containerization: Docker (with separate images for Dagster, Spark, and Streamlit environments)

b. Relationships and Data Flows:
- Data flowing from Tiki’s API → extraction via TikiCrawler → saved as JSON in MinIO (bronze layer).
- Transformation stages: bronze → silver (where data is loaded as Spark DataFrames), silver → gold (further cleaning/optimization), and a dedicated recommendation layer (where collaborative and content-based filtering is applied).
- Flow from transformed data to the data warehouse (loading into PostgreSQL).
- Data warehouse to visualization components (Metabase) and to the Streamlit-based recommendation system.
- Orchestration process (Dagster) managing these assets and scheduling including container deployment flows via Docker.

c. Architectural Patterns and Design Principles:
- Layered Architecture: Separate layers for extraction (bronze), transformation (silver, gold, recommendation), and load (data warehouse).
- Containerized Microservices: Each major component (ETL orchestration, Spark processing, Streamlit serving) runs in its containerized environment, as indicated by individual Docker configurations.
- Modular and Extensible Design: Clearly separated modules for assets, resources, and transformation functions enhance single-responsibility and reusability.
- Data Processing Pipelines: Use of Spark for distributed processing and Dagster for orchestrating complex data flows.

d. Key Technologies and Frameworks:
- Dagster for orchestration (configuration files in dagster_home and related Docker images).
- Apache Spark for distributed processing (referenced in docker_images/spark and within ETL transformation code).
- MinIO for object storage.
- PostgreSQL as the warehousing database.
- Streamlit for building interactive user interfaces.
- Docker (with dedicated Dockerfiles) for containerized deployments.
- Additionally, mention libraries like Gensim and TF-IDF for recommendation processing.

##### Elements
- Use clear labels for each component (e.g., “Tiki API”, “MinIO – Bronze Storage”, “Spark – Silver Transformation”, “PostgreSQL – DWH”, “Streamlit – Recommendation System”, “Dagster Orchestration”).
- Data extraction flows from Tiki API to raw data storage.
- Transformation flows from bronze → silver → gold.
- Infrastructure containerization aspect (Docker) by possibly adding a background or border grouping the containerized services.

##### Final Advice
- Be as detailed as possible: separate out components into as many logical units as the file tree and README illustrate.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% ETL Pipeline Subgraph
    subgraph "ETL Pipeline"
        A["Tiki API"]:::external
        B["TikiCrawler Extraction"]:::extraction
        C["MinIO - Bronze Storage"]:::storage
        E["Silver Layer Transformation"]:::processing
        F["Gold Layer Optimization"]:::processing
        G["Recommendation Layer Processing"]:::processing
        H["PostgreSQL - Data Warehouse"]:::database
    end

    I["Metabase - Dashboard/Analytics"]:::frontend
    J["Streamlit - Recommendation App"]:::frontend
    K["Dagster Orchestration"]:::orchestration
    L["Apache Spark Engine"]:::processing
    M["Docker Containerization"]:::container

    %% Data Flow Connections
    A -->|"extract"| B
    B -->|"storeJSON"| C
    C -->|"initTransform"| E
    E -->|"furtherProcess"| F
    E -->|"runRecAlgorithm"| G
    F -->|"loadData"| H
    G -->|"loadData"| H
    H -->|"visualize"| I
    H -->|"serveData"| J

    %% Orchestration and Infrastructure Flows
    K -.-> B
    K -.-> E
    K -.-> F
    K -.-> G
    L -.-> E
    L -.-> F
    M -.-> K
    M -.-> L
    M -.-> J

    %% Click Events for Component Mapping
    click B "https://github.com/nchn471/tiki-recommender-etl-pipeline/blob/main/etl_pipeline/etl_pipeline/resources/tiki_crawler.py"
    click C "https://github.com/nchn471/tiki-recommender-etl-pipeline/blob/main/etl_pipeline/etl_pipeline/resources/minio_io_manager.py"
    click K "https://github.com/nchn471/tiki-recommender-etl-pipeline/tree/main/dagster_home"
    click K "https://github.com/nchn471/tiki-recommender-etl-pipeline/tree/main/docker_images/dagster"
    click E "https://github.com/nchn471/tiki-recommender-etl-pipeline/blob/main/etl_pipeline/etl_pipeline/assets/silver_layer.py"
    click F "https://github.com/nchn471/tiki-recommender-etl-pipeline/blob/main/etl_pipeline/etl_pipeline/assets/gold_layer.py"
    click G "https://github.com/nchn471/tiki-recommender-etl-pipeline/blob/main/etl_pipeline/etl_pipeline/assets/recommendation_layer.py"
    click G "https://github.com/nchn471/tiki-recommender-etl-pipeline/blob/main/docker_images/streamlit/app/sub_pages/recommendation.py"
    click H "https://github.com/nchn471/tiki-recommender-etl-pipeline/blob/main/etl_pipeline/etl_pipeline/resources/psql_io_manager.py"
    click H "https://github.com/nchn471/tiki-recommender-etl-pipeline/blob/main/psql_schemas.sql"
    click L "https://github.com/nchn471/tiki-recommender-etl-pipeline/tree/main/docker_images/spark"
    click J "https://github.com/nchn471/tiki-recommender-etl-pipeline/tree/main/docker_images/streamlit"
    click M "https://github.com/nchn471/tiki-recommender-etl-pipeline/blob/main/docker-compose.yaml"

    %% Style Classes
    classDef external fill:#f9c,stroke:#333,stroke-width:2px;
    classDef extraction fill:#bbf,stroke:#333,stroke-width:2px;
    classDef storage fill:#bfb,stroke:#333,stroke-width:2px;
    classDef processing fill:#ffeb99,stroke:#333,stroke-width:2px;
    classDef database fill:#fcc,stroke:#333,stroke-width:2px;
    classDef frontend fill:#cff,stroke:#333,stroke-width:2px;
    classDef orchestration fill:#fdd,stroke:#333,stroke-width:2px;
    classDef container fill:#dfd,stroke:#333,stroke-width:2px;
```
