# zoenguyenramirez/arc-prize-2025 architecture

> Architecture diagram of the GitHub repository zoenguyenramirez/arc-prize-2025, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zoenguyenramirez/arc-prize-2025
- Repository: https://github.com/zoenguyenramirez/arc-prize-2025

## Overview

##### Project type and purpose
- This is a research‐grade, full-stack training pipeline for a “Pseudo-Reinforcement Learning” framework applied to ARC tasks.
- It orchestrates multi‐iteration model training, self-supervised trajectory generation, data management, and monitoring to improve abstract reasoning models.

##### File structure
- Top-level folders:
- src/                  → Core Python modules (data loaders, model, training orchestrator, inference, utils)
- scripts/              → Bash and Python scripts driving end-to-end pipelines (training, evaluation, data prep, monitoring)
- cloud/                → Helpers for local ↔ remote environment setup, SCP/SSH, TensorBoard on a server
- input_data/           → Raw JSON datasets (ARC-2024, ARC-2025, B-ARC, Re-ARC)
- intermediate_data/    → Prepared .pth datasets for fast loading
- runs/                 → Generated checkpoints, TensorBoard logs, trajectories per iteration
- notebooks/            → Exploratory analysis, debugging, visualization
- tests/                → Unit and integration tests validating data prep, training, inference, scripts
- Key files for architecture: orchestrate_training.py, train.py, generate_rollout.py, checkpoint_handler.py, submission.py, hyperparameter_tuning.py.

##### README for additional insights
- Confirms a 24‐iteration RL-style loop: each iteration trains on combined “original + previously generated trajectories,” then emits new trajectories.
- Emphasizes TensorBoard for monitoring, a “scratch space” of ATTEMPT_START/END tokens, and a detailed run directory layout under runs/.

1. Data Sources
- input_data (ARC-2024/5, B-ARC, Re-ARC JSON)
2. Data Preparation
- prepare_data.py → intermediate_data/*.pth
3. Training Orchestrator
- orchestrate_training.py
- train.py, rotational_train.py, meta_adaptive_train.py
- hyperparameter_tuning.py
4. Model & Checkpoint Management
- model.py, checkpoint_handler.py
- runs/iterN/…/*.pt (checkpoints, optimizer & scheduler states)
5. Trajectory Generator
- generate_rollout.py → runs/trajectories/trajectories_iterN.pt
6. Self-Supervised Loop
7. Monitoring & Logging
- TensorBoard (scripts/start_tensorboard.sh) reading runs/
8. Deployment & Evaluation
- submission.py, sample module for inference
- scripts/evaluation_test.sh, compare_inference.sh
9. Cloud/Remote Infrastructure
- cloud/local_scripts & cloud/remote_scripts enabling SSH, SCP, environment setup
10. Testing & CI
- tests/unit & tests/integration
- Raw JSON → Data Preparation → Prepared .pth
- Prepared .pth + trajectories_iterN-1.pt → Training Orchestrator → New Checkpoint (.pt)
- New Checkpoint → Trajectory Generator → trajectories_iterN.pt
- trajectories_iterN.pt → appended to training data input for iteration N+1
- Training & Generation produce logs → runs/ → TensorBoard
- scripts/ wrap calls to orchestrator, training, evaluation
- cloud scripts interact with runs/ remotely (SSH/SCP)
- tests/ validate scripts & src components
c. Architectural patterns & principles:
- Pipeline + feedback loop (iterative refinement)
- Clear separation of concerns: Data Prep / Training / Inference / Logging / Deployment / Remote Ops
- Modular code in src/, orchestration in scripts/
- Self-supervised “scratch space” via special tokens
d. Technologies & frameworks:
- Python, PyTorch (.pt model files)
- JSON for datasets, .pth for serialized datasets
- Bash shell scripts
- TensorBoard for visualization
- SSH/SCP for cloud orchestration

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Storage Layer
    subgraph "Storage" 
        DS[(Data Sources\ninput_data/)]:::storage
        IMD[(Prepared Data\nintermediate_data/)]:::storage
        RUNS[(Runs Outputs\nruns/)]:::storage
        TRAJ[(Trajectories\nruns/trajectories/)]:::storage
    end

    %% Data Preparation
    DP["Data Preparation Module\nprepare_data.py"]:::code

    %% Training Pipeline
    subgraph "Training Orchestrator & Core Training"
        ORCH["Training Orchestrator\norchestrate_training.py"]:::orches
        subgraph "Core Training Scripts"
            TS1["train.py"]:::code
            TS2["rotational_train.py"]:::code
            TS3["meta_adaptive_train.py"]:::code
            TS4["hyperparameter_tuning.py"]:::code
        end
    end

    %% Model & Checkpoint Management
    subgraph "Model & Checkpoint Management"
        ModelFile["model.py"]:::code
        CheckHandler["checkpoint_handler.py"]:::code
    end

    %% Trajectory Generation
    subgraph "Trajectory Generator"
        TG1["generate_rollout.py"]:::code
        TG2["rl_trajectory_generator.py"]:::code
    end

    %% Monitoring & Logging
    subgraph "Monitoring & Logging"
        TB1["start_tensorboard.sh"]:::scripts
        TB2["monitor_training.sh"]:::scripts
        TB3["continuous_monitor.py"]:::scripts
        TB4["logger_helper.py"]:::code
    end

    %% Deployment & Evaluation
    subgraph "Deployment & Evaluation"
        DP1["submission.py"]:::code
        DP2["adaptive_submission.py"]:::code
        DP3["evaluation_test.sh"]:::scripts
        DP4["compare_inference.sh"]:::scripts
    end

    %% Cloud / Remote Infrastructure
    subgraph "Cloud/Remote Infrastructure"
        CL1["local_scripts/"]:::cloud
        CL2["remote_scripts/"]:::cloud
    end

    %% Testing & CI
    subgraph "Testing & CI"
        TU["tests/unit/"]:::cloud
        TI["tests/integration/"]:::cloud
        RT["run_all_tests.sh"]:::scripts
    end

    %% Data Flow
    DS --> DP
    DP --> IMD
    IMD --> ORCH
    ORCH --> TS1
    ORCH --> TS2
    ORCH --> TS3
    ORCH --> TS4
    TS1 --> ModelFile
    TS2 --> ModelFile
    TS3 --> ModelFile
    TS4 --> ModelFile
    ModelFile --> CheckHandler
    CheckHandler --> RUNS
    RUNS --> TG1
    RUNS --> TG2
    TG1 --> TRAJ
    TG2 --> TRAJ
    TRAJ --> ORCH

    %% Monitoring & Control Flow
    RUNS --> TB1
    RUNS --> TB2
    RUNS --> TB3
    TB4 -.-> RUNS

    %% Deployment & Evaluation Flow
    ORCH --> DP1
    ORCH --> DP2
    DP1 --> DP3
    DP2 --> DP4

    %% Cloud Interactions
    CL1 -.-> RUNS
    CL2 -.-> RUNS
    CL1 -.-> TB1
    CL2 -.-> TB2

    %% Testing & CI
    DS -.-> TU
    DP -.-> TU
    ORCH -.-> TI
    TS1 -.-> RT

    %% Click Events
    click DS "https://github.com/zoenguyenramirez/arc-prize-2025/tree/to_publish/input_data/"
    click DP "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/src/prepare_data.py"
    click ORCH "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/src/orchestrate_training.py"
    click TS1 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/src/train.py"
    click TS2 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/src/rotational_train.py"
    click TS3 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/src/meta_adaptive_train.py"
    click TS4 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/src/hyperparameter_tuning.py"
    click ModelFile "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/src/model.py"
    click CheckHandler "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/src/checkpoint_handler.py"
    click TG1 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/src/generate_rollout.py"
    click TG2 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/src/rl_trajectory_generator.py"
    click TB1 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/scripts/start_tensorboard.sh"
    click TB2 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/scripts/monitor_training.sh"
    click TB3 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/scripts/continuous_monitor.py"
    click TB4 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/src/utils/logger_helper.py"
    click DP1 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/src/submission.py"
    click DP2 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/src/adaptive_submission.py"
    click DP3 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/scripts/evaluation_test.sh"
    click DP4 "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/scripts/compare_inference.sh"
    click CL1 "https://github.com/zoenguyenramirez/arc-prize-2025/tree/to_publish/cloud/local_scripts/"
    click CL2 "https://github.com/zoenguyenramirez/arc-prize-2025/tree/to_publish/cloud/remote_scripts/"
    click TU "https://github.com/zoenguyenramirez/arc-prize-2025/tree/to_publish/tests/unit/"
    click TI "https://github.com/zoenguyenramirez/arc-prize-2025/tree/to_publish/tests/integration/"
    click RT "https://github.com/zoenguyenramirez/arc-prize-2025/blob/to_publish/scripts/run_all_tests.sh"

    %% Styles
    classDef code fill:#D0E6FF,stroke:#0366d6,color:#0366d6
    classDef storage fill:#C8E6C9,stroke:#2E7D32,color:#2E7D32
    classDef orches fill:#FFE0B2,stroke:#EF6C00,color:#E65100
    classDef scripts fill:#FFECB3,stroke:#FF8F00,color:#BF360C
    classDef cloud fill:#E1BEE7,stroke:#8E24AA,color:#6A1B9A
```
