How verssae/neurojit works
Main Components
• Data Layer:
• Repository of Datasets: The “data” directory holds raw datasets, intermediate outputs (in subdirectories like dataset, combined, cuf), experiment outputs (data/output), and generated plots (data/output/plots).
• Archive of Pre-trained Models: The “archive” directory contains pickled models used in experiments.
• Experiment & Processing Pipeline:
• Preprocessing Scripts: Located in the “scripts” directory (e.g., data_utils.py, calculate.py, pre_analysis.py) for data cleaning, feature extraction, and experiments.
• Analysis & Visualization Scripts: Scripts such as analysis.py, correlation.py, visualization.py provide analysis functionality and output plots.
• Reproduction Script: The “reproduce.sh” script automates the replication of experimental results.
• Core Module (NeuroJIT):
• Source Code in “src/neurojit”: Contains modules like commit.py for commit filtering, cuf (with files such as cfg.py, halstead.py, metrics.py, rii.py) for calculating commit understandability features, and tools for implementations like sliding window methods.
• Distribution & Packaging:
• The “dist” directory shows the built neurojit package distributable (.whl file) and its availability on PyPI.
• Deployment & Containerization:
• Docker Configuration: Dockerfile and docker-compose.yml (and .dockerignore) indicate containerization, facilitating a consistent execution environment.
• Environment Setup: Files like pyproject.toml, indentation_config.xml, and requirements files ensure reproducibility via dependency management.
Relationships and Data Flows
• Data Input: Datasets from “data/dataset” are fed into preprocessing scripts using commands defined in scripts/data_utils.py and calculate.py.
• Preprocessing Pipeline: These scripts process raw data, combine features (baseline, commit understandability features, etc.), and output intermediary results to “data/output.”
• Core Processing: The NeuroJIT module in “src/neurojit” receives the preprocessed data to compute additional features (in cuf and commit modules) using domain-specific algorithms (e.g., sliding window method).
• Model Training & Evaluation: Scripts (e.g., jit_sdp.py) invoke machine learning models and perform evaluations, storing outcomes/modeled data as JSON files in “data/output.”
• Analysis/Visualization: Additional scripts perform further statistical analysis and generate plots saved to “data/output/plots.”
• Container Orchestration: Docker-compose ties it all together—building the environment using Dockerfile, running scripts, and automating experiments through commands like “docker-compose run.”
Architectural Patterns and Design Principles
• Modularity: The project separates concerns among data ingestion, preprocessing, core logic (NeuroJIT), and analysis.
• Reproducibility: Docker is used for environment standardization; the replication script automates the experimental pipeline.
• Extensibility: Customizable modules such as those for computing commit features (cuf modules) promote further experimentation.