# jakeson99/rugai architecture

> Architecture diagram of the GitHub repository jakeson99/rugai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jakeson99/rugai
- Repository: https://github.com/jakeson99/rugai

## Overview

##### Main Components

A. Data Collection Layer
- Web scraping components
- Both selenium_scraper.py and simple_scraper.py as alternative scrapers
- Configuration management (.env file)

B. Model Layer
- ResNet model architecture
- Weights directory
- Training workflow (notebook)
- Integration with Weights & Biases for experiment tracking

C. Application Layer
- Streamlit UI component
- Inference pipeline
- Preprocessing utilities
- Configuration management

##### Component Relationships

**[Data Collection] → [Model Training/Storage] → [Application Serving]**

##### Specific Implementation Details

- Rectangles: Core components (Scraper, Model, UI)
- Hexagons: External services (Streamlit Cloud, Weights & Biases)

##### Technical Stack Highlights

Technology badges or labels for:
- Python
- Streamlit
- CNN/ResNet
- Selenium
- Poetry (dependency management)
- MPS (M1 GPU acceleration)

##### Key Flows

A. Training Flow:
**Web Sources → Scrapers → Training Data → Model Training → Weights Storage**

B. Inference Flow:
**User Upload → Preprocessing → Model Inference → Classification Result → UI Display**

##### Important Elements

- Configuration Management: Show how config.yaml and environment variables are used
- Model Artifacts: Indicate where weights are stored and loaded
- Deployment: Show the Streamlit Cloud hosting
- Development Environment: Indicate Poetry and devcontainer setup

## Diagram (Mermaid)

```mermaid
graph TB
    %% Define styles
    classDef scraper fill:#3498db,stroke:#2980b9,color:white
    classDef model fill:#2ecc71,stroke:#27ae60,color:white
    classDef app fill:#e67e22,stroke:#d35400,color:white
    classDef config fill:#95a5a6,stroke:#7f8c8d,color:white
    classDef external fill:#9b59b6,stroke:#8e44ad,color:white
    classDef storage fill:#34495e,stroke:#2c3e50,color:white

    %% Data Collection Layer
    subgraph DataCollection
        Scrapers["Web Scrapers"]
        SeleniumScraper["Selenium Scraper"]
        SimpleScraper["Simple Scraper"]
        ScraperConfig["Scraper Config"]
        ScraperRunner["Scraper Runner"]
    end

    %% Model Layer
    subgraph ModelLayer
        ModelArch["ResNet Model"]
        ModelWeights[(Model Weights)]
        Training["Training Workflow"]
        WandB["Weights & Biases"]:::external
        InferencePipeline["Inference Pipeline"]
    end

    %% Application Layer
    subgraph AppLayer
        StreamlitUI["Streamlit UI"]
        Preprocessing["Preprocessing Utils"]
        AppConfig["App Config"]
        StreamlitCloud["Streamlit Cloud"]:::external
    end

    %% Development Environment
    subgraph DevEnv
        DevContainer["Dev Container"]
        Poetry["Poetry Dependencies"]
    end

    %% Relationships
    Scrapers --> SeleniumScraper
    Scrapers --> SimpleScraper
    SeleniumScraper & SimpleScraper --> ScraperRunner
    ScraperConfig --> ScraperRunner
    ScraperRunner --> Training
    Training --> ModelArch
    Training --> WandB
    ModelArch --> ModelWeights
    ModelWeights --> InferencePipeline
    InferencePipeline --> StreamlitUI
    Preprocessing --> InferencePipeline
    AppConfig --> StreamlitUI
    StreamlitUI --> StreamlitCloud

    %% Click events for component mapping
    click SeleniumScraper "https://github.com/jakeson99/rugai/blob/main/src/scraper/selenium_scraper.py"
    click SimpleScraper "https://github.com/jakeson99/rugai/blob/main/src/scraper/simple_scraper.py"
    click ScraperConfig "https://github.com/jakeson99/rugai/tree/main/src/scraper/config"
    click ScraperRunner "https://github.com/jakeson99/rugai/blob/main/src/scraper_runner.py"
    click ModelArch "https://github.com/jakeson99/rugai/blob/main/src/app/ml_framework/model.py"
    click ModelWeights "https://github.com/jakeson99/rugai/blob/main/model/weights/ResNet_20241206_070301_best.h5"
    click Training "https://github.com/jakeson99/rugai/blob/main/notebooks/model_training_workflow.ipynb"
    click InferencePipeline "https://github.com/jakeson99/rugai/blob/main/src/app/ml_framework/inference.py"
    click StreamlitUI "https://github.com/jakeson99/rugai/blob/main/src/app/app.py"
    click Preprocessing "https://github.com/jakeson99/rugai/blob/main/src/app/utils/preprocessing.py"
    click AppConfig "https://github.com/jakeson99/rugai/blob/main/src/app/config/config.yaml"
    click DevContainer "https://github.com/jakeson99/rugai/blob/main/.devcontainer/devcontainer.json"
    click Poetry "https://github.com/jakeson99/rugai/blob/main/pyproject.toml"

    %% Style applications
    class SeleniumScraper,SimpleScraper,ScraperRunner,Scrapers scraper
    class ModelArch,Training,InferencePipeline model
    class StreamlitUI,Preprocessing app
    class ScraperConfig,AppConfig config
    class ModelWeights storage
    class WandB,StreamlitCloud external
```
