How annelovepeace/game-recommendation-system works
Main Components
• Data Collection:
• Components: Two web crawling modules (as seen in the notebooks app_detail_crawl.ipynb and user_inventory_crawl.ipynb).
• External Connections: Interaction with the Steam API and Steamspy API to gather user inventories, game details, and game statistics.
• Data Storage:
• Data Files: Raw data stored in the “data” directory (files like app_detail.txt, app_stats.json, steam_user_id.txt, user_inventory.txt).
• Future/Intermediate Storage: The README also mentions writing processed data into a MySQL database, which suggests a dedicated storage component.
• Data Engineering:
• Processes: Feature extraction, transformation of raw txt/JSON files, handling missing values, and selecting relevant features (price, name, score, release date, etc.).
• Data Modeling:
• Models: Includes various recommendation approaches (popularity based, content based with TF-IDF, and collaborative filtering using item-based and user-based (ALS in Pyspark)).
• Code: The recommendation.py file appears to be central in implementing these strategies.
Out the Relationships and Interactions
• Data flows from external services (Steam API / Steamspy API) into the Data Collection component. The crawlers interact with these external APIs to fetch data.
• This raw data is stored in the “data” folder.
• Processed data is then written’s into a persistent store (MySQL database), which serves as the input for the Data Modeling component.
• Finally, the Data Modeling component leverages the prepared dataset to apply multiple machine learning models for generating recommendations.
Architectural Patterns and Key Design Principles
• Modular Pipeline: Clearly separated stages (Data Collection → Data Engineering → Data Modeling).
• Batch Processing: Web crawlers likely run in a batch mode (as suggested by the different notebooks and scripts).
• ETL Principles: Extract (via crawling), Transform (data cleanup and feature selection), and Load (writing into the database before modeling).
• Scalability and Extensibility: The modular design allows easy replacement or addition of models (such as additional collaborative filtering methods).