# xannadoo/researchproject architecture

> Architecture diagram of the GitHub repository xannadoo/researchproject, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xannadoo/researchproject
- Repository: https://github.com/xannadoo/researchproject

## Overview

##### Project Type and Purpose
- This is a research‐oriented data pipeline for generating and analyzing model interpretability artifacts (counterfactual examples and integrated gradients) over a corpus of questions.
- Key indicators:
- “scripts/” directory containing generation code (counterfactuals.py, intgrads.py)
- “data/” directory with raw and processed inputs plus JSON outputs
- “analysis/” notebooks performing post‐hoc analyses

##### High‐Level File Structure Analysis
- Top‐level:
- scripts/ – core Python modules and a notebook for data preparation & testing
- data/ – input data (questions, responses) and output artifacts (JSONs of counterfactuals/intgrads)
- analysis/ – Jupyter notebooks for detailed analyses of generated artifacts
- logs/ – job logs for long‐running compute steps
- images/ – figures (likely for reports or paper)
- .job files (counterfactuals.job, intgrads.job) – job scheduler configurations
- Patterns: single‐stage data processing pipeline → artifact generation → analysis
- No web frontend or microservices; it is a batch processing/research workflow

##### README Insights
- Minimal description; project name “ResearchProject.”
- No explicit architecture section, so derive architecture from directory layout and file naming

##### Components
a. Data Layer
- Raw inputs: data/extracted_questions.zip, data/responses.csv
- Preprocessed inputs: data/prepped_questions.json
b. Compute & Job Scheduling
- Job scheduler (e.g., SLURM, .job files) invoking scripts
- Compute environment (e.g., HPC cluster or cloud instances)
c. Generation Engines
- Counterfactual generator: scripts/counterfactuals.py
- Integrated gradients calculator: scripts/intgrads.py
- Shared utilities & data classes: scripts/classes.py
d. Outputs & Logs
- Generated artifacts: data/outputs/*.json (large_*, small_*)
- Job logs: logs/intgrads_*.out, logs/counterfactuals_*.out
e. Analysis Layer
- Jupyter notebooks in analysis/ that ingest JSON outputs and produce figures
f. Dependencies
- Python environment with libraries listed in requirements.txt
- Notebook runtime for exploratory data analysis

##### Relationships & Data Flow
- Scheduler (.job) triggers generation scripts, which consume prepped data and produce JSON outputs + log files
- Outputs flow into analysis notebooks, which generate visualizations stored under images/
- Feedback loop: Analysts update scripts or parameters and re-run jobs

##### Architectural Patterns & Design Principles
- Pipeline pattern: clear staged separation (PREP → GENERATION → ANALYSIS)
- Modular design: separate scripts for each artifact type, shared data classes for consistency
- Reproducibility: job configs and logs tracked in version control

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    raw1["extracted_questions.zip"]:::data
    raw2["responses.csv"]:::data
    prep_nb["Dataset Prep & Testing Notebook"]:::notebook
    prepped["prepped_questions.json"]:::data

    %% Scheduler & Compute
    scheduler_cf["counterfactuals.job"]:::scheduler
    scheduler_ig["intgrads.job"]:::scheduler
    compute["Compute Environment\n(HPC/Cloud)"]:::compute

    %% Generation Engines
    classes["Shared Utilities & Data Classes"]:::util
    cf_script["Counterfactual Generator"]:::process
    ig_script["Integrated Gradients Calculator"]:::process

    %% Outputs & Logs
    outputs["Generated Artifacts\n(JSON outputs)"]:::data
    logs_cf["Counterfactuals Logs"]:::log
    logs_ig["Intgrads Logs"]:::log

    %% Analysis Layer
    analysis_cf["Counterfactual Analysis Notebook"]:::notebook
    analysis_ig["Intgrad Analysis Notebook"]:::notebook
    images["Visualization Outputs"]:::image

    %% Dependencies
    requirements["requirements.txt"]:::util

    %% Data Flow
    raw1 --> prep_nb
    raw2 --> prep_nb
    prep_nb --> prepped

    prepped --> cf_script
    prepped --> ig_script

    scheduler_cf --> compute
    scheduler_ig --> compute
    compute --> cf_script
    compute --> ig_script

    classes --> cf_script
    classes --> ig_script
    requirements --> cf_script
    requirements --> ig_script

    cf_script --> outputs
    cf_script --> logs_cf
    ig_script --> outputs
    ig_script --> logs_ig

    outputs --> analysis_cf
    outputs --> analysis_ig
    analysis_cf --> images
    analysis_ig --> images

    %% Click Events
    click raw1 "https://github.com/xannadoo/researchproject/blob/main/data/extracted_questions.zip"
    click raw2 "https://github.com/xannadoo/researchproject/blob/main/data/responses.csv"
    click prepped "https://github.com/xannadoo/researchproject/blob/main/data/prepped_questions.json"
    click scheduler_cf "https://github.com/xannadoo/researchproject/blob/main/counterfactuals.job"
    click scheduler_ig "https://github.com/xannadoo/researchproject/blob/main/intgrads.job"
    click cf_script "https://github.com/xannadoo/researchproject/blob/main/scripts/counterfactuals.py"
    click ig_script "https://github.com/xannadoo/researchproject/blob/main/scripts/intgrads.py"
    click classes "https://github.com/xannadoo/researchproject/blob/main/scripts/classes.py"
    click prep_nb "https://github.com/xannadoo/researchproject/blob/main/scripts/dataset_and_testing.ipynb"
    click outputs "https://github.com/xannadoo/researchproject/blob/main/data/outputs/*.json"
    click logs_cf "https://github.com/xannadoo/researchproject/blob/main/logs/counterfactuals_*.out"
    click logs_ig "https://github.com/xannadoo/researchproject/blob/main/logs/intgrads_*.out"
    click analysis_cf "https://github.com/xannadoo/researchproject/blob/main/analysis/counterfactual_analysis.ipynb"
    click analysis_ig "https://github.com/xannadoo/researchproject/blob/main/analysis/intgrad_analysis.ipynb"
    click images "https://github.com/xannadoo/researchproject/tree/main/images/"
    click requirements "https://github.com/xannadoo/researchproject/blob/main/requirements.txt"

    %% Styles
    classDef data fill:#cfe2f3,stroke:#034694,stroke-width:2px
    classDef process fill:#d9ead3,stroke:#38761d,stroke-width:2px
    classDef scheduler fill:#fff2cc,stroke:#bf9000,stroke-width:2px
    classDef compute fill:#f4cccc,stroke:#a61c00,stroke-width:2px
    classDef notebook fill:#ead1dc,stroke:#980000,stroke-width:2px
    classDef log fill:#fce5cd,stroke:#bf854e,stroke-width:2px
    classDef image fill:#d0e0e3,stroke:#0b5394,stroke-width:2px
    classDef util fill:#ecd9f2,stroke:#674ea7,stroke-width:2px
```
