# ameyaranadee/automatic-short-answer-grading architecture

> Architecture diagram of the GitHub repository ameyaranadee/automatic-short-answer-grading, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ameyaranadee/automatic-short-answer-grading
- Repository: https://github.com/ameyaranadee/automatic-short-answer-grading

## Overview

##### Project type and purpose
- From the file tree and README, this is a Python‑based machine‑learning pipeline (“Automatic Short Answer Grading”) for scoring student free‑text answers using feature extraction and regression models.
- It is not a web or microservices app but an ML experimentation/training tool with clear pipeline stages.

##### File structure
- Top‑level modules:
- args.py – CLI argument parsing
- preprocessing.py – text normalization and spelling correction
- embeddings.py – loading Baroni embeddings into memory
- feature_extraction.py – computing similarity‑based features
- run.py & main.sh – orchestration scripts to execute the full pipeline
- Data directories:
- data/ – raw CSVs (Data.csv, QA1.csv) and downloaded embeddings file
- downloads/ – intermediate processed files (combined_final_features.csv, etc.)
- requirements.txt – lists Python libraries (e.g., scikit‑learn, NLTK, fuzzywuzzy, python‑Levenshtein)
- No separate “models/” folder, but training outputs and logs likely live in “downloads/”

##### README for architecture hints
- Pipeline stages: Preprocessing → Feature Extraction → Train/Test split → Model Training/Evaluation
- External dependencies: Baroni embeddings, NLTK’s WordNet, a spellchecker library

a. Main components
- Data Sources
– Raw Student Answers & Reference Answers (data/Data.csv, QA1.csv)
– Pre‑trained Word Embeddings (data/EN‑wform…txt)
- Preprocessing Module (preprocessing.py)
– Text cleaning, tokenization, stop‑word removal, spell‑correction via WordNet & spellchecker
- Embedding Loader (embeddings.py)
– Loads CBOW vectors into in‑memory lookup structure
- Feature Extraction Module (feature_extraction.py)
– Computes cosine similarity, alignment score, length ratio, Euclidean distance, fuzzy matches
- Model Training & Evaluation Module (run.py & main.sh)
– Data split (75/25), trains Random Forest, Ridge, Neural Net regressors, outputs predictions
- Outputs
– Processed feature CSVs (downloads/feature_df.csv, processed_df.csv)
– Final model performance metrics and trained model artifacts

b. Relationships & data flow
1. Raw CSVs + Embeddings → Preprocessing Module
2. Preprocessed text → Embedding Loader → Feature Extraction
3. Features → Train/Test Split → Model Training
4. Trained Models → Predictions → Evaluation reports

c. Architectural patterns/design principles
- Modular pipeline: each stage is a separate Python module
- Data separation: raw data, intermediate feature files, final outputs
- Reproducibility: orchestrated via main.sh and run.py

d. Relevant technologies/frameworks
- Python 3.x, pandas, scikit‑learn, NLTK, fuzzywuzzy, python‑Levenshtein

##### Key points to ensure detail
- Intermediate artifact files in “downloads/” as outputs of specific modules

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Sources
    subgraph "Data Sources"
        Data1["Student Answers\n(data/Data.csv)"]:::data
        Data2["Reference Answers\n(data/QA1.csv)"]:::data
        EmbeddingsFile["Pre-trained Word Embeddings\n(data/EN-wform...txt)"]:::data
    end

    %% External Services
    WordNetSpell["WordNet & Spellchecker"]:::ext

    %% Processing Modules
    subgraph "Processing Modules"
        Args["CLI Argument Parser\n(args.py)"]:::proc
        Preproc["Preprocessing Module\n(preprocessing.py)"]:::proc
        EmbedLoader["Embedding Loader Module\n(embeddings.py)"]:::proc
        FeatureExt["Feature Extraction Module\n(feature_extraction.py)"]:::proc
    end

    %% Orchestration & Model Training/Evaluation
    subgraph "Orchestration & Training"
        RunScript["Pipeline Orchestration & Model Training/Evaluation\n(run.py, main.sh)"]:::model
    end

    %% Intermediate Artifacts
    subgraph "Intermediate Artifacts"
        FeatureCSV["Feature CSV\n(downloads/feature_df.csv)"]:::data
        ProcessedCSV["Processed Features\n(downloads/processed_df.csv)"]:::data
        CombinedCSV["Combined Final Features\n(downloads/combined_final_features.csv)"]:::data
    end

    %% Outputs
    Metrics["Model Metrics & Artifacts"]:::model

    %% Data Flow
    Data1 -->|raw CSV| Preproc
    Data2 -->|raw CSV| Preproc
    EmbeddingsFile -->|embeddings file| EmbedLoader
    Preproc -->|tokenized text| FeatureExt
    Preproc -->|uses| WordNetSpell
    EmbedLoader -->|embedding vectors| FeatureExt
    FeatureExt -->|feature CSV| FeatureCSV
    FeatureExt -->|processed df| ProcessedCSV
    FeatureExt -->|combined features| CombinedCSV
    FeatureCSV -->|features| RunScript
    ProcessedCSV -->|features| RunScript
    CombinedCSV -->|features| RunScript
    Args -->|CLI args| RunScript
    RunScript -->|trained models & predictions| Metrics

    %% Click Events
    click Data1 "https://github.com/ameyaranadee/automatic-short-answer-grading/blob/main/data/Data.csv"
    click Data2 "https://github.com/ameyaranadee/automatic-short-answer-grading/blob/main/data/QA1.csv"
    click Preproc "https://github.com/ameyaranadee/automatic-short-answer-grading/blob/main/preprocessing.py"
    click EmbedLoader "https://github.com/ameyaranadee/automatic-short-answer-grading/blob/main/embeddings.py"
    click FeatureExt "https://github.com/ameyaranadee/automatic-short-answer-grading/blob/main/feature_extraction.py"
    click Args "https://github.com/ameyaranadee/automatic-short-answer-grading/blob/main/args.py"
    click RunScript "https://github.com/ameyaranadee/automatic-short-answer-grading/blob/main/run.py"
    click RunScript "https://github.com/ameyaranadee/automatic-short-answer-grading/blob/main/main.sh"
    click FeatureCSV "https://github.com/ameyaranadee/automatic-short-answer-grading/blob/main/downloads/feature_df.csv"
    click ProcessedCSV "https://github.com/ameyaranadee/automatic-short-answer-grading/blob/main/downloads/processed_df.csv"
    click CombinedCSV "https://github.com/ameyaranadee/automatic-short-answer-grading/blob/main/downloads/combined_final_features.csv"

    %% Styles
    classDef data fill:#cce5ff,stroke:#004085,color:#004085;
    classDef proc fill:#d4edda,stroke:#155724,color:#155724;
    classDef model fill:#fff3cd,stroke:#856404,color:#856404;
    classDef ext fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
```
