# nemo-lab-repo/reproduce_rfeats architecture

> Architecture diagram of the GitHub repository nemo-lab-repo/reproduce_rfeats, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nemo-lab-repo/reproduce_rfeats
- Repository: https://github.com/nemo-lab-repo/reproduce_rfeats

## Overview

This project is a machine learning pipeline focused on training a Variational Autoencoder (VAE) and a classifier, along with generating synthetic “environment” data. The workflow is clearly divided into several steps (as indicated by the code files):

##### Main Components
- Data Storage: Represent the raw data (classification data folders such as “data/1”, “data/2”, … "data/p2", "data/p3", etc.), experiment data (“data/experimentdata1” and “data/experimentdata2”), and generated data (“data/generate”). Also include instructions for adding new data.
- VAE Training Module: Include the script “code/step1_train_vae.py”, with its associated model code in “models/vae.py”. This module reads data from the data layer, trains the VAE, and outputs model artifacts (stored in “logs/train_vae.log” and “logs/model.pt”).
- Classifier Training Module: Represent the classifier training process (“code/step3_train_classifier.py” and “code/step4_classifier.py”). This module depends on the data layer (which can include either raw classification data or data generated by the VAE, based on the provided flag “--use_generator”) and uses the classifier definition from “models/classifier.py”. Outputs such as logs (e.g., “logs/train_classifier.log”, “logs/classify.log”) should be noted.
- Data Augmentation/Conversion Utility: Include the “scripts/generate_p.py” process that converts added “experimentdata” files (environment related) into the new data format in the “data/p{i}” folders.
- Dependencies & Configuration: Represent dependency management using “requirements.txt” and any configurable parameters passed to the scripts (like number of epochs, batch size, etc.).
- Logging & Model Artifacts: Indicate the “logs” and “models” directories where training logs and saved models reside.

##### Relationships and Data Flow
- From there, show that both raw data and generated data (if the flag “--use_generator” is activated) feed into the Classifier Training Module.
- Finally, the outputs (trained models and logs) are stored in the “models” and “logs” directories respectively.

##### Architectural Patterns and Design Principles
- Separation of Concerns: The system is clearly modularized into separate scripts and modules for training the VAE, generating data, and training the classifier.
- Pipeline Flow: The sequential execution—from data ingestion through model training and data generation—hints at a pipeline architecture where artifacts from one stage feed into the next.
- Data Augmentation via Synthetic Generation: Emphasize that the generated data (from the VAE) augments the available data for classifier training.

##### Relevant Technologies and Libraries
- The presence of “models/__init__.py” and Python modules suggests standard Python package management practices.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    A["Raw Data Storage (raw classification, experiment & generated data)"] 
      
    %% VAE Training Module Subgraph
    subgraph "VAE Training Module"
        B1["Script: step1_train_vae.py"]
        B2["Model: vae.py"]
        B1 --> B2
    end

    %% Data Generation Module
    C["Data Generator: step2_generate_env_data.py"]

    %% Classifier Training Module Subgraph
    subgraph "Classifier Training Module"
        D1["Script: step3_train_classifier.py"]
        D2["Script: step4_classifier.py"]
        D3["Model: classifier.py"]
        D1 --> D2
        D2 --> D3
    end

    %% Data Conversion Utility
    E["Data Conversion Utility: generate_p.py"]

    %% Dependencies & Configuration
    F["Dependencies & Config: requirements.txt"]

    %% Artifacts Subgraph
    subgraph "Artifacts"
        G1["Logs"]
        G2["Models"]
    end

    %% Relationships and Data Flow
    F -->|"configures"| B1
    F -->|"configures"| C
    F -->|"configures"| D1
    F -->|"configures"| E

    A -->|"feeds"| B1
    B2 -->|"supplies"| C
    A -->|"supplies"| D1
    C -->|"augments"| D1
    E -->|"converts"| A

    B2 -->|"outputs"| G1
    B2 -->|"outputs"| G2
    D3 -->|"outputs"| G1
    D3 -->|"outputs"| G2

    %% Class Definitions for styling
    classDef data fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef module fill:#A9DFBF,stroke:#117A65,stroke-width:2px;
    classDef utility fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef config fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
    classDef artifact fill:#D2B4DE,stroke:#7D3C98,stroke-width:2px;

    %% Applying classes to nodes
    class A data;
    class B1,B2,C,D1,D2,D3 module;
    class E utility;
    class F config;
    class G1,G2 artifact;

    %% Click Events
    click A "https://github.com/nemo-lab-repo/reproduce_rfeats/tree/main/data/"
    click B1 "https://github.com/nemo-lab-repo/reproduce_rfeats/blob/main/code/step1_train_vae.py"
    click B2 "https://github.com/nemo-lab-repo/reproduce_rfeats/blob/main/models/vae.py"
    click C "https://github.com/nemo-lab-repo/reproduce_rfeats/blob/main/code/step2_generate_env_data.py"
    click D1 "https://github.com/nemo-lab-repo/reproduce_rfeats/blob/main/code/step3_train_classifier.py"
    click D2 "https://github.com/nemo-lab-repo/reproduce_rfeats/blob/main/code/step4_classifier.py"
    click D3 "https://github.com/nemo-lab-repo/reproduce_rfeats/blob/main/models/classifier.py"
    click E "https://github.com/nemo-lab-repo/reproduce_rfeats/blob/main/scripts/generate_p.py"
    click F "https://github.com/nemo-lab-repo/reproduce_rfeats/blob/main/requirements.txt"
    click G1 "https://github.com/nemo-lab-repo/reproduce_rfeats/tree/main/logs/"
    click G2 "https://github.com/nemo-lab-repo/reproduce_rfeats/tree/main/models/"
```
