# lqj233/end-to-end_recommendation_system_x architecture

> Architecture diagram of the GitHub repository lqj233/end-to-end_recommendation_system_x, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lqj233/end-to-end_recommendation_system_x
- Repository: https://github.com/lqj233/end-to-end_recommendation_system_x

## Overview

This repository is a learning-oriented, end-to-end recommendation system. The principal path runs from the storefront’s home feed to the business API and four-stage online recommender, then returns ranked items for display and behavior tracking. Tracked events feed a documented streaming loop; batch imports and layered Spark processing supply training data, and scheduled full or incremental training produces serving artifacts. The tree omits the backend and streaming implementations, so those documented components have no source path; edges involving them rely on the README rather than sampled code.

Main entry points: [`rec-serving/app/artifacts.py`](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/artifacts.py) (Model artifacts), [`rec-serving/app/pipeline/real_pipeline.py`](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/pipeline/real_pipeline.py) (Pipeline runner), [`web-frontend/src/views/HomeView.vue`](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/web-frontend/src/views/HomeView.vue) (Home feed), [`airflow/dags/daily_warehouse_and_incremental_train.py`](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/airflow/dags/daily_warehouse_and_incremental_train.py) (Airflow schedules)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Storefront"]
  node_home["Home feed<br/>[HomeView.vue]"]
  node_itemdetail["Item detail<br/>[ItemDetailView.vue]"]
  node_api_client["API client<br/>[api.js]"]
  node_tracking["Behavior tracking<br/>[tracking.js]"]
end

subgraph group_service["Application services"]
  node_backend["Business backend"]
  node_outbox["Event outbox"]
end

subgraph group_recommend["Online recommendation"]
  node_recommend_api["Recommendation API<br/>[recommend.py]"]
  node_pipeline["Pipeline runner<br/>[real_pipeline.py]"]
  node_recall["Multiway recall<br/>[real_pipeline.py]"]
  node_coarse["Coarse ranking<br/>[real_pipeline.py]"]
  node_fine["DeepFM ranking<br/>[real_pipeline.py]"]
  node_rerank["MMR reranking<br/>[real_pipeline.py]"]
  node_features["Feature fetcher<br/>[features.py]"]
  node_artifacts["Model artifacts<br/>[artifacts.py]"]
end

subgraph group_offline["Offline data and training"]
  node_taobao["Taobao import<br/>[import_taobao.py]"]
  node_warehouse["Layered warehouse<br/>[build_dwd.py]"]
  node_incremental_wh["Incremental warehouse"]
  node_cache_recall["Cache recall build"]
  node_train["Full training<br/>[train.py]"]
  node_incremental_train["Incremental training"]
  node_registry["Model registry<br/>[registry.py]"]
  node_scheduler["Airflow schedules"]
end

subgraph group_platform["Data platform"]
  node_flink["Realtime processing"]
  node_kafka["Kafka"]
  node_mysql[("MySQL")]
  node_redis[("Redis")]
  node_iceberg[("Iceberg warehouse")]
  node_minio[("MinIO storage")]
  node_milvus[("Milvus Lite")]
end

node_shopper(("Shopper"))

node_shopper -->|"views"| node_home
node_shopper -->|"opens"| node_itemdetail
node_home -->|"fetches feed"| node_api_client
node_itemdetail -->|"fetches item"| node_api_client
node_api_client -->|"calls HTTP API"| node_backend
node_home -->|"records exposure"| node_tracking
node_home -->|"records clicks"| node_tracking
node_tracking -->|"posts events"| node_backend
node_backend -->|"requests recommendations"| node_recommend_api
node_backend -->|"reads and writes"| node_mysql
node_backend -->|"uses cache"| node_redis
node_backend -->|"stores events"| node_outbox
node_outbox -->|"publishes events"| node_kafka
node_recommend_api -->|"runs pipeline"| node_pipeline
node_pipeline -->|"runs recall"| node_recall
node_pipeline -->|"runs coarse rank"| node_coarse
node_pipeline -->|"runs fine rank"| node_fine
node_pipeline -->|"runs rerank"| node_rerank
node_recall -->|"reads history and hot list"| node_features
node_fine -->|"reads user features"| node_features
node_features -->|"reads features"| node_redis
node_recall -->|"uses indexes"| node_artifacts
node_coarse -->|"uses embeddings"| node_artifacts
node_fine -->|"uses model"| node_artifacts
node_rerank -->|"uses item attributes"| node_artifacts
node_artifacts -.->|"searches vectors"| node_milvus
node_recommend_api -->|"returns ranked items"| node_home
node_taobao -->|"loads source data"| node_iceberg
node_warehouse -->|"builds warehouse layers"| node_iceberg
node_incremental_wh -->|"processes new data"| node_iceberg
node_cache_recall -.->|"exports recall cache"| node_redis
node_scheduler -->|"schedules ETL"| node_warehouse
node_scheduler -->|"schedules increment"| node_incremental_wh
node_scheduler -->|"schedules training"| node_incremental_train
node_scheduler -->|"schedules full training"| node_train
node_iceberg -->|"supplies samples"| node_train
node_train -->|"registers versions"| node_registry
node_incremental_train -->|"registers versions"| node_registry
node_train -->|"produces serving bundle"| node_artifacts
node_incremental_train -->|"produces updated bundle"| node_artifacts
node_kafka -->|"streams events"| node_flink
node_flink -->|"writes online features"| node_redis
node_flink -->|"stores raw events"| node_minio
node_minio -->|"supplies event data"| node_iceberg
node_kafka -.->|"carries CDC changes"| node_flink

click node_home "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/web-frontend/src/views/HomeView.vue"
click node_itemdetail "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/web-frontend/src/views/ItemDetailView.vue"
click node_api_client "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/web-frontend/src/services/api.js"
click node_tracking "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/web-frontend/src/services/tracking.js"
click node_recommend_api "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/api/recommend.py"
click node_pipeline "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/pipeline/real_pipeline.py"
click node_recall "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/pipeline/real_pipeline.py"
click node_coarse "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/pipeline/real_pipeline.py"
click node_fine "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/pipeline/real_pipeline.py"
click node_rerank "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/pipeline/real_pipeline.py"
click node_features "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/features.py"
click node_artifacts "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/artifacts.py"
click node_taobao "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/pyspark/import_taobao.py"
click node_warehouse "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/pyspark/build_dwd.py"
click node_incremental_wh "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/pyspark/build_dws_incremental.py"
click node_cache_recall "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/pyspark/build_cache_recall.py"
click node_train "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/training/train.py"
click node_incremental_train "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/training/incremental_train.py"
click node_registry "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/training/registry.py"
click node_scheduler "https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/airflow/dags/daily_warehouse_and_incremental_train.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_home,node_itemdetail,node_api_client,node_tracking toneBlue
class node_backend,node_outbox toneAmber
class node_recommend_api,node_pipeline,node_recall,node_coarse,node_fine,node_rerank,node_features,node_artifacts toneMint
class node_taobao,node_warehouse,node_incremental_wh,node_cache_recall,node_train,node_incremental_train,node_registry,node_scheduler toneRose
class node_flink,node_kafka,node_mysql,node_redis,node_iceberg,node_minio,node_milvus,node_shopper toneIndigo
```

## Components

### Storefront

- [Home feed](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/web-frontend/src/views/HomeView.vue): `web-frontend/src/views/HomeView.vue`
- [Item detail](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/web-frontend/src/views/ItemDetailView.vue): `web-frontend/src/views/ItemDetailView.vue`
- [API client](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/web-frontend/src/services/api.js): `web-frontend/src/services/api.js`
- [Behavior tracking](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/web-frontend/src/services/tracking.js): `web-frontend/src/services/tracking.js`

### Application services

- **Business backend**
- **Event outbox**

### Online recommendation

- [Recommendation API](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/api/recommend.py): `rec-serving/app/api/recommend.py`
- [Pipeline runner](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/pipeline/real_pipeline.py): `rec-serving/app/pipeline/real_pipeline.py`
- [Multiway recall](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/pipeline/real_pipeline.py): `rec-serving/app/pipeline/real_pipeline.py`
- [Coarse ranking](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/pipeline/real_pipeline.py): `rec-serving/app/pipeline/real_pipeline.py`
- [DeepFM ranking](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/pipeline/real_pipeline.py): `rec-serving/app/pipeline/real_pipeline.py`
- [MMR reranking](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/pipeline/real_pipeline.py): `rec-serving/app/pipeline/real_pipeline.py`
- [Feature fetcher](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/features.py): `rec-serving/app/features.py`
- [Model artifacts](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/rec-serving/app/artifacts.py): `rec-serving/app/artifacts.py`

### Offline data and training

- [Taobao import](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/pyspark/import_taobao.py): `offline/pyspark/import_taobao.py`
- [Layered warehouse](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/pyspark/build_dwd.py): `offline/pyspark/build_dwd.py`
- [Incremental warehouse](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/pyspark/build_dws_incremental.py): `offline/pyspark/build_dws_incremental.py`
- [Cache recall build](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/pyspark/build_cache_recall.py): `offline/pyspark/build_cache_recall.py`
- [Full training](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/training/train.py): `offline/training/train.py`
- [Incremental training](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/training/incremental_train.py): `offline/training/incremental_train.py`
- [Model registry](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/offline/training/registry.py): `offline/training/registry.py`
- [Airflow schedules](https://github.com/lqj233/end-to-end_recommendation_system_x/blob/main/airflow/dags/daily_warehouse_and_incremental_train.py): `airflow/dags/daily_warehouse_and_incremental_train.py`

### Data platform

- **Realtime processing**
- **Kafka**
- **MySQL**
- **Redis**
- **Iceberg warehouse**
- **MinIO storage**
- **Milvus Lite**

### Other components

- **Shopper**

## Connections

- Shopper → Home feed: views
- Shopper → Item detail: opens
- Home feed → API client: fetches feed (evidence: `web-frontend/src/views/HomeView.vue`)
- Item detail → API client: fetches item (evidence: `web-frontend/src/views/ItemDetailView.vue`)
- API client → Business backend: calls HTTP API (evidence: `web-frontend/src/services/api.js`)
- Home feed → Behavior tracking: records exposure (evidence: `web-frontend/src/views/HomeView.vue`)
- Home feed → Behavior tracking: records clicks (evidence: `web-frontend/src/views/HomeView.vue`)
- Behavior tracking → Business backend: posts events (evidence: `web-frontend/src/services/tracking.js`)
- Business backend → Recommendation API: requests recommendations
- Business backend → MySQL: reads and writes
- Business backend → Redis: uses cache
- Business backend → Event outbox: stores events
- Event outbox → Kafka: publishes events
- Recommendation API → Pipeline runner: runs pipeline (evidence: `rec-serving/app/api/recommend.py`)
- Pipeline runner → Multiway recall: runs recall (evidence: `rec-serving/app/pipeline/real_pipeline.py`)
- Pipeline runner → Coarse ranking: runs coarse rank (evidence: `rec-serving/app/pipeline/real_pipeline.py`)
- Pipeline runner → DeepFM ranking: runs fine rank (evidence: `rec-serving/app/pipeline/real_pipeline.py`)
- Pipeline runner → MMR reranking: runs rerank (evidence: `rec-serving/app/pipeline/real_pipeline.py`)
- Multiway recall → Feature fetcher: reads history and hot list (evidence: `rec-serving/app/pipeline/real_pipeline.py`)
- DeepFM ranking → Feature fetcher: reads user features (evidence: `rec-serving/app/pipeline/real_pipeline.py`)
- Feature fetcher → Redis: reads features (evidence: `rec-serving/app/features.py`)
- Multiway recall → Model artifacts: uses indexes (evidence: `rec-serving/app/pipeline/real_pipeline.py`)
- Coarse ranking → Model artifacts: uses embeddings (evidence: `rec-serving/app/pipeline/real_pipeline.py`)
- DeepFM ranking → Model artifacts: uses model (evidence: `rec-serving/app/pipeline/real_pipeline.py`)
- MMR reranking → Model artifacts: uses item attributes (evidence: `rec-serving/app/pipeline/real_pipeline.py`)
- Model artifacts → Milvus Lite: searches vectors
- Recommendation API → Home feed: returns ranked items
- Taobao import → Iceberg warehouse: loads source data (evidence: `offline/pyspark/import_taobao.py`)
- Layered warehouse → Iceberg warehouse: builds warehouse layers
- Incremental warehouse → Iceberg warehouse: processes new data (evidence: `offline/pyspark/build_dws_incremental.py`)
- Cache recall build → Redis: exports recall cache
- Airflow schedules → Layered warehouse: schedules ETL
- Airflow schedules → Incremental warehouse: schedules increment
- Airflow schedules → Incremental training: schedules training (evidence: `airflow/dags/daily_warehouse_and_incremental_train.py`)
- Airflow schedules → Full training: schedules full training
- Iceberg warehouse → Full training: supplies samples
- Full training → Model registry: registers versions (evidence: `offline/training/train.py`)
- Incremental training → Model registry: registers versions (evidence: `offline/training/incremental_train.py`)
- Full training → Model artifacts: produces serving bundle
- Incremental training → Model artifacts: produces updated bundle
- Kafka → Realtime processing: streams events
- Realtime processing → Redis: writes online features
- Realtime processing → MinIO storage: stores raw events
- MinIO storage → Iceberg warehouse: supplies event data
- Kafka → Realtime processing: carries CDC changes