# lemniscate-world/neural architecture

> Architecture diagram of the GitHub repository lemniscate-world/neural, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lemniscate-world/neural
- Repository: https://github.com/lemniscate-world/neural

## Overview

##### Project Type & Purpose
- Neural is not a standard full-stack web app but rather a specialized DSL and toolchain for deep learning. Its purpose is to let users declaratively define neural networks (via YAML-like syntax), generate code for multiple backends (TensorFlow, PyTorch, ONNX), orchestrate training, and provide real-time visualization and debugging (NeuralDbg). This is evident from the README’s overview and the DSL examples.

##### Analyzing the File Tree & Key Architectural Indicators
- Top-Level Directories and Files:
– The “neural” folder serves as the main codebase, containing modules such as:
▪ cli.py: Command-line interface entry point.
▪ code_generation/: Contains code generation logic (translating DSL into target-framework code).
▪ parser/: Implements DSL parsing.
▪ dashboard/ and visualization/: Provide the interactive debugging and visualization interfaces (NeuralDbg).
▪ execution_optimization/ and hpo/: Offer training optimization and hyperparameter tuning features.
▪ config/: Contains YAML configuration for the system.
– “docs/” and “examples/” supply usage guides and model definitions.
– Workflow configuration in “.github/workflows” and scripts aid in CI/CD/testing.

##### Main Components to be Represented
a. DSL Frontend:
– DSL Interface (Model Definition Files, e.g., *.neural, YAML-like syntax)
– Parser Module (neural/parser): Converts DSL scripts into an intermediate representation.

b. Code Generation & Backend Selection:
– Code Generation Module (neural/code_generation): Transforms the intermediate DSL representation into executable code for various backends, i.e., TensorFlow, PyTorch, or ONNX.
– Backend Adapters: Components that customize output per framework (possibly represented as separate “Code Generator” blocks for TensorFlow, PyTorch, etc.).

c. Execution & Training Orchestrator:
– Training Module (neural/training): Manages training routines, optimizer configurations, and device handling.
– Execution Optimization (neural/execution_optimization): Responsible for performance tuning.
– HPO Module (neural/hpo): Manages hyperparameter optimization across backends.

d. Debugging & Visualization:
– Visualization Tools: Both static (PNG, HTML outputs) and dynamic visualizations (d3-based components in neural/src/d3.v7.js and dynamic_visualizer folder).

e. No-Code Interface & CLI:
– No-Code Interface: A separate module invoked with a “--no_code” flag that provides a graphical user interface.

f. Plugins & Extensibility:
– Plugins (neural/plugins): Allow custom layers and extensions.
– Neural Chat and Research Generation modules: Additional functionalities for interactive help and research paper generation.

##### Relationships & Data Flow
- The user writes a model definition using the DSL format (stored in files like “mnist.neural” in examples/tests).
- The CLI initiates the process, invoking the Parser which converts the DSL into an intermediate representation.
- This representation is handed off to the Code Generation module which, based on command-line flags (e.g., “--backend tensorflow”), creates code targetting a chosen framework.
- Once generated, the Training module takes over to execute (or simulate) training routines.
- The system also provides an optional no-code interface for visually specifying models.
- Plugin extensions are applied where custom behavior is needed.

##### Architectural Patterns & Design Principles
- Modular Design: Each major feature (e.g., HPO optimization, NeuralDbg, and backend code generation) is implemented as independent, but interacting, modules.
- Extensibility: Indicate open points for plugins and integrations.
- Interoperability: Showcase the multi-backend design allowing smooth switching between TensorFlow, PyTorch, and ONNX.

- All main components with clear labels.
– DSL file input to Parser.
– Parser output dispatched to Code Generation and Training modules.
– Code Generation feeding into framework-specific adapters.
– Training engine interacting with NeuralDbg and visualization dashboards.
– Plugin integrations as add-on modules enhancing core functionality.
– Rectangles for processing modules.

- The progression from user input (DSL scripts) through parsing, code generation, and training.
- The branch for debugging and visualization including real-time monitoring using NeuralDbg.
- The ability to generate outputs for multiple frameworks and the modular approach allowing for plugins and customization.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% DSL & CLI Subgraph
    subgraph "DSL & CLI"
        DSL["DSL Model File"]:::input
        CLI["CLI Module"]:::processing
    end
    DSL -->|"uses"| CLI
    CLI -->|"triggers"| Parser["Parser Module"]:::processing
    click Parser "https://github.com/lemniscate-world/neural/blob/main/neural/parser/parser.py"

    %% Code Generation & Backend Selection Subgraph
    subgraph "Code Generation & Backend Selection"
        CodeGen["Code Generation Module"]:::processing
        click CodeGen "https://github.com/lemniscate-world/neural/blob/main/neural/code_generation/code_generator.py"
        TF["TensorFlow Adapter"]:::processing
        PT["PyTorch Adapter"]:::processing
        ONNX["ONNX Adapter"]:::processing
    end
    Parser -->|"producesIR"| CodeGen
    CodeGen -->|"generatesCodeFor"| TF
    CodeGen -->|"generatesCodeFor"| PT
    CodeGen -->|"generatesCodeFor"| ONNX

    %% Execution & Training Orchestrator Subgraph
    subgraph "Execution & Training Orchestrator"
        Training["Training Module"]:::processing
        click Training "https://github.com/lemniscate-world/neural/blob/main/neural/training/training.py"
        ExecOpt["Execution Optimization Module"]:::processing
        click ExecOpt "https://github.com/lemniscate-world/neural/blob/main/neural/execution_optimization/execution.py"
        HPO["HPO Module"]:::processing
        click HPO "https://github.com/lemniscate-world/neural/blob/main/neural/hpo/hpo.py"
    end
    Parser -->|"producesIR"| Training
    Training -->|"optimizes"| ExecOpt
    Training -->|"tunesHyperparams"| HPO

    %% Debugging & Visualization Subgraph
    subgraph "Debugging & Visualization"
        Dashboard["NeuralDbg Dashboard"]:::ui
        click Dashboard "https://github.com/lemniscate-world/neural/blob/main/neural/dashboard/dashboard.py"
        Viz["Visualization Tools"]:::ui
        click Viz "https://github.com/lemniscate-world/neural/blob/main/neural/visualization/visualizer.py"
    end
    Training -->|"monitors"| Dashboard
    Dashboard -->|"displays"| Viz

    %% No-Code Interface Subgraph
    subgraph "No-Code Interface"
        NoCode["No-Code Interface"]:::ui
        click NoCode "https://github.com/lemniscate-world/neural/blob/main/neural/no_code/no_code.py"
    end
    CLI -->|"alternativeUI"| NoCode

    %% Plugins & Extensibility Subgraph
    subgraph "Plugins & Extensibility"
        Plugins["Plugins Module"]:::ext
        click Plugins "https://github.com/lemniscate-world/neural/blob/main/neural/plugins/plugins.py"
        Custom["Custom Plugin: Layer"]:::ext
        click Custom "https://github.com/lemniscate-world/neural/blob/main/neural/plugins/my_layer.py"
        Chat["Neural Chat Module"]:::ext
        click Chat "https://github.com/lemniscate-world/neural/blob/main/neural/neural_chat/neural_chat.py"
        Research["Research Generation Module"]:::ext
        click Research "https://github.com/lemniscate-world/neural/blob/main/neural/research_generation/research_paper.py"
    end
    CodeGen -->|"canUse"| Plugins
    Training -->|"canUse"| Plugins
    Plugins -->|"extends"| Chat
    Plugins -->|"extends"| Research

    %% Styles
    classDef input fill:#E6E6FA,stroke:#333,stroke-width:2px;
    classDef processing fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef ui fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef ext fill:#FFA500,stroke:#333,stroke-width:2px;
```
