How lqj233/end-to-end_recommendation_system_x works
This repository is a learning-oriented, end-to-end recommendation system. The principal path runs from the storefront’s home feed to the business API and four-stage online recommender, then returns ranked items for display and behavior tracking. Tracked events feed a documented streaming loop; batch imports and layered Spark processing supply training data, and scheduled full or incremental training produces serving artifacts. The tree omits the backend and streaming implementations, so those documented components have no source path; edges involving them rely on the README rather than sampled code.
Main entry points: artifacts.py Model artifacts · real_pipeline.py Pipeline runner · HomeView.vue Home feed · daily_warehouse_and_incremental_train.py Airflow schedules
Components 30
Storefront
- Home feed
web-frontend/src/views/HomeView.vue - Item detail
web-frontend/src/views/ItemDetailView.vue - API client
web-frontend/src/services/api.js - Behavior tracking
web-frontend/src/services/tracking.js
Application services
- Business backend
- Event outbox
Online recommendation
- Recommendation API
rec-serving/app/api/recommend.py - Pipeline runner
rec-serving/app/pipeline/real_pipeline.py - Multiway recall
rec-serving/app/pipeline/real_pipeline.py - Coarse ranking
rec-serving/app/pipeline/real_pipeline.py - DeepFM ranking
rec-serving/app/pipeline/real_pipeline.py - MMR reranking
rec-serving/app/pipeline/real_pipeline.py - Feature fetcher
rec-serving/app/features.py - Model artifacts
rec-serving/app/artifacts.py
Offline data and training
- Taobao import
offline/pyspark/import_taobao.py - Layered warehouse
offline/pyspark/build_dwd.py - Incremental warehouse
offline/pyspark/build_dws_incremental.py - Cache recall build
offline/pyspark/build_cache_recall.py - Full training
offline/training/train.py - Incremental training
offline/training/incremental_train.py - Model registry
offline/training/registry.py - Airflow schedules
airflow/dags/daily_warehouse_and_incremental_train.py
Data platform
- Realtime processing
- Kafka
- MySQL
- Redis
- Iceberg warehouse
- MinIO storage
- Milvus Lite
Other components
- Shopper
Connections 45
- Shopper → Home feed: views
- Shopper → Item detail: opens
- Home feed → API client: fetches feed
web-frontend/src/views/HomeView.vue - Item detail → API client: fetches item
web-frontend/src/views/ItemDetailView.vue - API client → Business backend: calls HTTP API
web-frontend/src/services/api.js - Home feed → Behavior tracking: records exposure
web-frontend/src/views/HomeView.vue - Home feed → Behavior tracking: records clicks
web-frontend/src/views/HomeView.vue - Behavior tracking → Business backend: posts events
web-frontend/src/services/tracking.js - Business backend → Recommendation API: requests recommendations
- Business backend → MySQL: reads and writes
- Business backend → Redis: uses cache
- Business backend → Event outbox: stores events
- Event outbox → Kafka: publishes events
- Recommendation API → Pipeline runner: runs pipeline
rec-serving/app/api/recommend.py - Pipeline runner → Multiway recall: runs recall
rec-serving/app/pipeline/real_pipeline.py - Pipeline runner → Coarse ranking: runs coarse rank
rec-serving/app/pipeline/real_pipeline.py - Pipeline runner → DeepFM ranking: runs fine rank
rec-serving/app/pipeline/real_pipeline.py - Pipeline runner → MMR reranking: runs rerank
rec-serving/app/pipeline/real_pipeline.py - Multiway recall → Feature fetcher: reads history and hot list
rec-serving/app/pipeline/real_pipeline.py - DeepFM ranking → Feature fetcher: reads user features
rec-serving/app/pipeline/real_pipeline.py - Feature fetcher → Redis: reads features
rec-serving/app/features.py - Multiway recall → Model artifacts: uses indexes
rec-serving/app/pipeline/real_pipeline.py - Coarse ranking → Model artifacts: uses embeddings
rec-serving/app/pipeline/real_pipeline.py - DeepFM ranking → Model artifacts: uses model
rec-serving/app/pipeline/real_pipeline.py - MMR reranking → Model artifacts: uses item attributes
rec-serving/app/pipeline/real_pipeline.py - Model artifacts → Milvus Lite: searches vectors
- Recommendation API → Home feed: returns ranked items
- Taobao import → Iceberg warehouse: loads source data
offline/pyspark/import_taobao.py - Layered warehouse → Iceberg warehouse: builds warehouse layers
- Incremental warehouse → Iceberg warehouse: processes new data
offline/pyspark/build_dws_incremental.py - Cache recall build → Redis: exports recall cache
- Airflow schedules → Layered warehouse: schedules ETL
- Airflow schedules → Incremental warehouse: schedules increment
- Airflow schedules → Incremental training: schedules training
airflow/dags/daily_warehouse_and_incremental_train.py - Airflow schedules → Full training: schedules full training
- Iceberg warehouse → Full training: supplies samples
- Full training → Model registry: registers versions
offline/training/train.py - Incremental training → Model registry: registers versions
offline/training/incremental_train.py - Full training → Model artifacts: produces serving bundle
- Incremental training → Model artifacts: produces updated bundle
- Kafka → Realtime processing: streams events
- Realtime processing → Redis: writes online features
- Realtime processing → MinIO storage: stores raw events
- MinIO storage → Iceberg warehouse: supplies event data
- Kafka → Realtime processing: carries CDC changes