# verssae/neurojit architecture

> Architecture diagram of the GitHub repository verssae/neurojit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/verssae/neurojit
- Repository: https://github.com/verssae/neurojit

## Overview

##### Main Components
- Data Layer:
- Repository of Datasets: The “data” directory holds raw datasets, intermediate outputs (in subdirectories like dataset, combined, cuf), experiment outputs (data/output), and generated plots (data/output/plots).
- Archive of Pre-trained Models: The “archive” directory contains pickled models used in experiments.
- Experiment & Processing Pipeline:
- Preprocessing Scripts: Located in the “scripts” directory (e.g., data_utils.py, calculate.py, pre_analysis.py) for data cleaning, feature extraction, and experiments.
- Analysis & Visualization Scripts: Scripts such as analysis.py, correlation.py, visualization.py provide analysis functionality and output plots.
- Reproduction Script: The “reproduce.sh” script automates the replication of experimental results.
- Core Module (NeuroJIT):
- Source Code in “src/neurojit”: Contains modules like commit.py for commit filtering, cuf (with files such as cfg.py, halstead.py, metrics.py, rii.py) for calculating commit understandability features, and tools for implementations like sliding window methods.
- Distribution & Packaging:
- The “dist” directory shows the built neurojit package distributable (.whl file) and its availability on PyPI.
- Deployment & Containerization:
- Docker Configuration: Dockerfile and docker-compose.yml (and .dockerignore) indicate containerization, facilitating a consistent execution environment.
- Environment Setup: Files like pyproject.toml, indentation_config.xml, and requirements files ensure reproducibility via dependency management.

##### Relationships and Data Flows
- Data Input: Datasets from “data/dataset” are fed into preprocessing scripts using commands defined in scripts/data_utils.py and calculate.py.
- Preprocessing Pipeline: These scripts process raw data, combine features (baseline, commit understandability features, etc.), and output intermediary results to “data/output.”
- Core Processing: The NeuroJIT module in “src/neurojit” receives the preprocessed data to compute additional features (in cuf and commit modules) using domain-specific algorithms (e.g., sliding window method).
- Model Training & Evaluation: Scripts (e.g., jit_sdp.py) invoke machine learning models and perform evaluations, storing outcomes/modeled data as JSON files in “data/output.”
- Analysis/Visualization: Additional scripts perform further statistical analysis and generate plots saved to “data/output/plots.”
- Container Orchestration: Docker-compose ties it all together—building the environment using Dockerfile, running scripts, and automating experiments through commands like “docker-compose run.”

##### Architectural Patterns and Design Principles
- Modularity: The project separates concerns among data ingestion, preprocessing, core logic (NeuroJIT), and analysis.
- Reproducibility: Docker is used for environment standardization; the replication script automates the experimental pipeline.
- Extensibility: Customizable modules such as those for computing commit features (cuf modules) promote further experimentation.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer Subgraph
    subgraph "Data Layer"
        Data["Raw Datasets (data/dataset)"]:::dataLayer
        Outputs["Processed Outputs (data/output)"]:::dataLayer
        Plots["Generated Plots (data/output/plots)"]:::dataLayer
        Archive["Pre-trained Models (archive)"]:::dataLayer
    end

    %% Experiment & Processing Pipeline Subgraph
    subgraph "Experiment & Processing Pipeline"
        subgraph "Preprocessing Scripts"
            DATA_UTILS["data_utils.py"]:::processing
            CALCULATE["calculate.py"]:::processing
            PRE_ANALYSIS["pre_analysis.py"]:::processing
        end
        REPRODUCE["Reproduction Script (reproduce.sh)"]:::processing
        subgraph "Analysis/Visualization Scripts"
            ANALYSIS["analysis.py"]:::analysis
            CORRELATION["correlation.py"]:::analysis
            VISUALIZATION["visualization.py"]:::analysis
            JIT_SDP["jit_sdp.py"]:::analysis
        end
    end

    %% Core Module Subgraph
    subgraph "Core Module (NeuroJIT)"
        NeuroJIT["NeuroJIT Core Module (src/neurojit)"]:::core
        COMMIT["commit.py"]:::core
        subgraph "CUF Modules"
            CFG["cfg.py"]:::core
            HALSTEAD["halstead.py"]:::core
            METRICS["metrics.py"]:::core
            RII["rii.py"]:::core
        end
        TOOLS["Tools (src/neurojit/tools)"]:::core
    end

    %% Distribution & Packaging Subgraph
    subgraph "Distribution & Packaging"
        Distribution["Package Distribution (dist)"]:::distribution
    end

    %% Deployment & Containerization Subgraph
    subgraph "Deployment & Containerization"
        Docker["Docker Configurations (Dockerfile, docker-compose.yml, .dockerignore)"]:::deployment
        Env["Environment Setup (pyproject.toml, indentation_config.xml, requirements.lock, requirements-dev.lock)"]:::deployment
    end

    %% Relationships / Data Flow
    %% Flow from Data Layer to Preprocessing
    Data -->|"raw_data"| DATA_UTILS
    %% Reproduction script triggers preprocessing
    REPRODUCE -->|"triggers"| DATA_UTILS
    %% Preprocessing internal flow (within scripts)
    DATA_UTILS --> CALCULATE
    CALCULATE --> PRE_ANALYSIS
    %% Preprocessing Scripts feed into Core Module
    PRE_ANALYSIS -->|"preprocessed_data"| NeuroJIT
    %% Archive of models also used by NeuroJIT
    Archive -->|"model_data"| NeuroJIT
    %% Core Module processing details
    NeuroJIT --> COMMIT
    NeuroJIT -->|"uses"| CFG
    NeuroJIT --> HALSTEAD
    NeuroJIT --> METRICS
    NeuroJIT --> RII
    NeuroJIT --> TOOLS
    %% Core Module outputs feed into Analysis
    NeuroJIT -->|"feature_data"| ANALYSIS
    %% Analysis scripts internal flow
    ANALYSIS --> CORRELATION
    CORRELATION --> VISUALIZATION
    VISUALIZATION --> JIT_SDP
    %% Analysis outputs to Processed Outputs and Plots
    JIT_SDP -->|"evaluation"| Outputs
    Outputs -->|"plots_generation"| Plots
    %% Distribution built from Core Module
    NeuroJIT -->|"packaged_as"| Distribution
    %% Docker integration touches the entire pipeline
    Docker --- Data
    Docker --- DATA_UTILS
    Docker --- PRE_ANALYSIS
    Docker --- NeuroJIT
    Docker --- ANALYSIS
    Docker --- Distribution
    Docker --- REPRODUCE
    Env --> Docker

    %% Styles
    classDef dataLayer fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef processing fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef core fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef analysis fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef deployment fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef distribution fill:#fde2e2,stroke:#c92a2a,stroke-width:2px;

    %% Click Events
    click Data "https://github.com/verssae/neurojit/tree/main/data"
    click Archive "https://github.com/verssae/neurojit/tree/main/archive"
    click DATA_UTILS "https://github.com/verssae/neurojit/blob/main/scripts/data_utils.py"
    click CALCULATE "https://github.com/verssae/neurojit/blob/main/scripts/calculate.py"
    click PRE_ANALYSIS "https://github.com/verssae/neurojit/blob/main/scripts/pre_analysis.py"
    click REPRODUCE "https://github.com/verssae/neurojit/blob/main/scripts/reproduce.sh"
    click ANALYSIS "https://github.com/verssae/neurojit/blob/main/scripts/analysis.py"
    click CORRELATION "https://github.com/verssae/neurojit/blob/main/scripts/correlation.py"
    click VISUALIZATION "https://github.com/verssae/neurojit/blob/main/scripts/visualization.py"
    click JIT_SDP "https://github.com/verssae/neurojit/blob/main/scripts/jit_sdp.py"
    click NeuroJIT "https://github.com/verssae/neurojit/tree/main/src/neurojit"
    click COMMIT "https://github.com/verssae/neurojit/blob/main/src/neurojit/commit.py"
    click CFG "https://github.com/verssae/neurojit/blob/main/src/neurojit/cuf/cfg.py"
    click HALSTEAD "https://github.com/verssae/neurojit/blob/main/src/neurojit/cuf/halstead.py"
    click METRICS "https://github.com/verssae/neurojit/blob/main/src/neurojit/cuf/metrics.py"
    click RII "https://github.com/verssae/neurojit/blob/main/src/neurojit/cuf/rii.py"
    click TOOLS "https://github.com/verssae/neurojit/tree/main/src/neurojit/tools"
    click Distribution "https://github.com/verssae/neurojit/tree/main/dist"
    click Docker "https://github.com/verssae/neurojit/tree/main/Dockerfile"
    click Docker "https://github.com/verssae/neurojit/blob/main/docker-compose.yml"
    click Docker "https://github.com/verssae/neurojit/blob/main/ .dockerignore"
    click Env "https://github.com/verssae/neurojit/blob/main/pyproject.toml"
    click Env "https://github.com/verssae/neurojit/blob/main/indentation_config.xml"
    click Env "https://github.com/verssae/neurojit/blob/main/requirements.lock"
    click Env "https://github.com/verssae/neurojit/blob/main/requirements-dev.lock"
```
