# rohit11may/lyricist architecture

> Architecture diagram of the GitHub repository rohit11may/lyricist, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rohit11may/lyricist
- Repository: https://github.com/rohit11may/lyricist

## Overview

Key Components to Include in the System Design:

##### Data Collection Layer
- Crawler.py as a component that orchestrates data collection
- A data storage component representing the pickle files in the /data directory

##### Data Processing Pipeline
- NLP processing pipeline (indicated by nltk.txt presence)
- Topic modeling component (LDA with 10 topics)

##### Model Architecture
- Main model component (model.py)
- Trained model artifacts (model.hdf5 and associated pickle files)
- Dictionary and topic model components

##### Service Layer
- Server component (server.py)
- API endpoints for recommendation service
- Scheduling component (represented by crontab.txt)

Design Guidelines:

- [Data Sources]
- ↓
- [Data Collection]
- ↓
- [Data Processing]
- ↓
- [Model Training]
- ↓
- [Serving Layer]

##### Specific data flows
- How lyrics and audio features are collected separately
- Where data is cached (pickle files)
- Recommendation flow from input to output

##### Technical Details to Highlight
- The use of multiple pickle files for caching different types of data
- The separation between crawling and serving components
- The machine learning pipeline with topic modeling (LDA)

##### Important Relationships
- How the crawler interacts with both API clients
- Flow of data from raw collection to processed features

##### Additional Elements
- A small metrics/monitoring section
- Separation between training and inference paths

Keep the following aspects in mind:
- Machine learning pipeline as it's central to the project
- Clear data flow between components
- Where caching happens to improve performance
- Scheduling mechanism for updates

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External APIs
    subgraph APIs
        SpotifyAPI[Spotify API]:::external
        GeniusAPI[Genius API]:::external
    end

    %% API Clients
    subgraph Clients
        SpotifyClient[Spotify Client]:::client
        GeniusClient[Genius Client]:::client
    end

    %% Data Collection
    subgraph Collection
        Crawler[Data Crawler]:::collection
        DevNotebooks[Development Notebooks]:::collection
    end

    %% Data Storage
    subgraph Storage
        LyricsCache[Lyrics Cache]:::storage
        AudioCache[Audio Features Cache]:::storage
        TracksCache[Tracks Cache]:::storage
        RecommendHistory[Recommendation History]:::storage
        GenreMap[Genre Mapping]:::storage
    end

    %% Processing Pipeline
    subgraph Processing
        TextUtils[Text Processing Utils]:::processing
        NLPPipeline[NLP Pipeline]:::processing
    end

    %% Model Components
    subgraph ModelLayer
        TopicModel[LDA Topic Model]:::model
        Dictionary[Model Dictionary]:::model
        MainModel[Main Model]:::model
        ModelArtifacts[Model Artifacts]:::model
    end

    %% Service Layer
    subgraph Service
        Server[API Server]:::service
        Scheduler[Update Scheduler]:::service
    end

    %% Relationships
    SpotifyAPI <--> SpotifyClient
    GeniusAPI <--> GeniusClient
    SpotifyClient --> Crawler
    GeniusClient --> Crawler
    Crawler --> LyricsCache
    Crawler --> AudioCache
    Crawler --> TracksCache
    Crawler --> DevNotebooks
    LyricsCache --> TextUtils
    TextUtils --> NLPPipeline
    NLPPipeline --> TopicModel
    TopicModel --> Dictionary
    Dictionary --> MainModel
    MainModel --> ModelArtifacts
    ModelArtifacts --> Server
    Server --> RecommendHistory
    GenreMap --> Server
    Scheduler --> Crawler

    %% Click Events
    click SpotifyClient "https://github.com/rohit11may/lyricist/blob/master/api_clients/SpotifyClient.py"
    click GeniusClient "https://github.com/rohit11may/lyricist/blob/master/api_clients/GeniusClient.py"
    click Crawler "https://github.com/rohit11may/lyricist/blob/master/crawler.py"
    click LyricsCache "https://github.com/rohit11may/lyricist/blob/master/data/crawled_lyrics.pkl"
    click AudioCache "https://github.com/rohit11may/lyricist/blob/master/data/crawled_audio_features.pkl"
    click TracksCache "https://github.com/rohit11may/lyricist/blob/master/data/crawled_tracks.pkl"
    click TextUtils "https://github.com/rohit11may/lyricist/blob/master/utils.py"
    click NLPPipeline "https://github.com/rohit11may/lyricist/blob/master/nltk.txt"
    click TopicModel "https://github.com/rohit11may/lyricist/blob/master/models/lda_10topics.pkl"
    click Dictionary "https://github.com/rohit11may/lyricist/blob/master/models/dictionary.pkl"
    click MainModel "https://github.com/rohit11may/lyricist/blob/master/model.py"
    click ModelArtifacts "https://github.com/rohit11may/lyricist/blob/master/models/model.hdf5"
    click Server "https://github.com/rohit11may/lyricist/blob/master/server.py"
    click Scheduler "https://github.com/rohit11may/lyricist/blob/master/crontab.txt"
    click RecommendHistory "https://github.com/rohit11may/lyricist/blob/master/data/already_recommended.pkl"
    click GenreMap "https://github.com/rohit11may/lyricist/blob/master/data/genre_map.pkl"
    click DevNotebooks "https://github.com/rohit11may/lyricist/tree/master/notebooks"

    %% Styles
    classDef external fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef client fill:#ADD8E6,stroke:#333,stroke-width:2px
    classDef collection fill:#90EE90,stroke:#333,stroke-width:2px
    classDef storage fill:#D3D3D3,stroke:#333,stroke-width:2px
    classDef processing fill:#98FB98,stroke:#333,stroke-width:2px
    classDef model fill:#FFD700,stroke:#333,stroke-width:2px
    classDef service fill:#DDA0DD,stroke:#333,stroke-width:2px
```
