How rohit11may/lyricist works
Key Components to Include in the System Design:
Data Collection Layer
• Crawler.py as a component that orchestrates data collection
• A data storage component representing the pickle files in the /data directory
Data Processing Pipeline
• NLP processing pipeline (indicated by nltk.txt presence)
• Topic modeling component (LDA with 10 topics)
Model Architecture
• Main model component (model.py)
• Trained model artifacts (model.hdf5 and associated pickle files)
• Dictionary and topic model components
Service Layer
• Server component (server.py)
• API endpoints for recommendation service
• Scheduling component (represented by crontab.txt)
Design Guidelines:
• [Data Sources]
• ↓
• [Data Collection]
• ↓
• [Data Processing]
• ↓
• [Model Training]
• ↓
• [Serving Layer]
Specific data flows
• How lyrics and audio features are collected separately
• Where data is cached (pickle files)
• Recommendation flow from input to output
Technical Details to Highlight
• The use of multiple pickle files for caching different types of data
• The separation between crawling and serving components
• The machine learning pipeline with topic modeling (LDA)
Important Relationships
• How the crawler interacts with both API clients
• Flow of data from raw collection to processed features
Additional Elements
• A small metrics/monitoring section
• Separation between training and inference paths
Keep the following aspects in mind:
• Machine learning pipeline as it's central to the project
• Clear data flow between components
• Where caching happens to improve performance
• Scheduling mechanism for updates