# rodsiry/nem architecture

> Architecture diagram of the GitHub repository rodsiry/nem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rodsiry/nem
- Repository: https://github.com/rodsiry/nem

## Overview

##### PROJECT OVERVIEW
- Purpose: The project implements the NEM algorithm which evolves neuron update rules (via genetic search) to achieve efficient continual learning on class-incremental tasks (with MNIST as a primary example, and evaluations on tasks like SVHN).
- Type: This is a research/experimental, single-process script-based project designed primarily for algorithm development, training, testing, and visualizing learned neural filters.

##### ANALYZING THE FILE STRUCTURE
- README.md: Offers an overview of the project, instructions to run the main components, and a reference to the original research paper.
- nem.py: Likely the main entry point for launching the genetic search or the meta-training procedure.
- nem_test.py: The test harness for evaluating trained update rules on various meta-test tasks.
- basic_train.py: Presumably provides an alternative or base training loop for running experiments.
- models.py: Contains the neural network model definitions or the specific implementation of NEM’s neuron model.
- loaders.py: Holds data loading routines, ensuring that tasks like MNIST and SVHN are correctly fed into the system.
- plot.py: Provides plotting and visualization capabilities to display the learned filters.
- utils.py: A collection of helper functions probably used across modules (e.g., logging, configuration, metrics).

A. Core Components:
- Genetic Search / Meta-Training Module (nem.py):
- Central executable for launching the genetic optimization process.
- Implements the search for update rules that exhibit continual learning.
- Test/Evaluation Module (nem_test.py):
- Evaluates the learned update rules on meta-test datasets like MNIST and SVHN.
- Triggers filter visualization via the plot module.
- Training Module (basic_train.py):
- Provides baseline or alternative training routines.

B. Supporting Components:
- Data Loading Layer (loaders.py):
- Manages dataset import and preprocessing.
- Ensures proper data flow from raw input to the training/evaluation modules.
- Model Definitions (models.py):
- Contains model architectures and possibly the NEM neuron implementation.
- Visualization (plot.py):
- Handles displaying the learned filters and other experiment results.
- Utility Functions (utils.py):
- Provides shared functionality (e.g., logging, configuration, error handling) across modules.

##### CONNECTIONS AND INTERACTIONS
- Data Flow:
- Data starts at the Data Loading Layer, feeding into both the training (basic_train.py) and testing (nem_test.py) pipelines.
- The output from the training modules and the evolution process in nem.py produces updated models and update rules which will later be tested.
- Module Dependencies:
- “nem.py” orchestrates the meta-training genetic search process and interfaces with “models.py” for neural computations.
- “nem_test.py” depends on the outputs of “nem.py” and “basic_train.py” to evaluate performance and invoke visualizations via “plot.py.”
- “utils.py” is integrated across the system to support logging, experiments configuration, and utility operations.

##### ARCHITECTURAL PRINCIPLES AND TECHNOLOGIES
- Design Pattern: The structure can conceptually follow a modular design where each file represents a self-contained component:
- Model Layer (models.py)
- Data Layer (loaders.py)
- Training/Evaluation Layer (nem.py, basic_train.py, nem_test.py)
- Visualization and Utility Support (plot.py, utils.py)
- Experimental Workflow: Emphasize the separation of experimentation phases – genetic search/meta-training vs. evaluation and visualization.
- Technologies: Although not explicitly detailed in the README, expect Python’s scientific ecosystem (e.g., NumPy, potentially PyTorch or TensorFlow if used in models) to be underlying, along with libraries for plotting.

##### FINAL REMARKS FOR THE PRINCIPAL SOFTWARE ENGINEER
- Being as detailed as possible. Do not shy away from separating the components into fine-grained modules even if that results in many parts. This ensures that all architectural elements, especially those critical for understanding the experimental workflow (meta-training, evaluation, and visualization), are captured.
- Confirm that every module mentioned in the file tree is accounted for and their relationships are clearly illustrated.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Layer"
        A["Data Loader (loaders.py)"]:::data
    end

    subgraph "Core Modules"
        B["Genetic Search (nem.py)"]:::core
        C["Training Module (basic_train.py)"]:::core
        D["Model Definitions (models.py)"]:::support
    end

    subgraph "Evaluation and Visualization"
        E["Test/Evaluation (nem_test.py)"]:::core
        F["Visualization (plot.py)"]:::support
    end

    U["Utility Functions (utils.py)"]:::utility

    A -->|"feeds"| B
    A -->|"feeds"| C
    B -->|"updates"| D
    B -->|"triggers"| E
    C -->|"providesData"| E
    E -->|"invokes"| F

    B ---|"uses"| U
    C ---|"uses"| U
    D ---|"uses"| U
    A ---|"uses"| U
    E ---|"uses"| U
    F ---|"uses"| U

    click A "https://github.com/rodsiry/nem/blob/main/loaders.py"
    click B "https://github.com/rodsiry/nem/blob/main/nem.py"
    click C "https://github.com/rodsiry/nem/blob/main/basic_train.py"
    click D "https://github.com/rodsiry/nem/blob/main/models.py"
    click E "https://github.com/rodsiry/nem/blob/main/nem_test.py"
    click F "https://github.com/rodsiry/nem/blob/main/plot.py"
    click U "https://github.com/rodsiry/nem/blob/main/utils.py"

    classDef data fill:#AEDFF7,stroke:#333,stroke-width:2px;
    classDef core fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef support fill:#FDFD96,stroke:#333,stroke-width:2px;
    classDef utility fill:#F0E68C,stroke:#333,stroke-width:2px;
```
