# aman-22byte/prism_iiitdmk_24vi20iiitk architecture

> Architecture diagram of the GitHub repository aman-22byte/prism_iiitdmk_24vi20iiitk, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aman-22byte/prism_iiitdmk_24vi20iiitk
- Repository: https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk

## Overview

##### Project type + purpose (what you’re diagramming)
This is an **offline ML data-generation pipeline** (not a web service): it **generates synthetic videos** by **adding learned motion to static image content** to create a **video denoising dataset**.

Core idea from the README:
- Based on **MAGE+** (derived from MAGE): **VQ-VAE** for tokenization/latent codebook + **autoregressive Transformer** for motion modeling.
- You run an **inference script** that takes **either AVI or YUV input**, extracts frames / a reference image, runs the pretrained model, and outputs a synthesized video into a local folder.

##### File tree-driven architecture (map folders/files to components)
From the tree, the likely major components are:

##### Entry points / orchestration
- `load_image.py`
- `dataload.py`
Likely a **dataset/input loader** abstraction used by the run script and/or transforms.

##### Model assets (pretrained + config)
- `models/MAGE+/catergenv2_diverse/config.yaml`
- README references `models/MAGE+/catergenv2_diverse/model_best.pth`

- **Config** (model hyperparams, tokenizer params, etc.)
- **Weights checkpoint**

##### Shared utilities / processing blocks
- `utils/videotransforms.py` → **Video transforms** (resize/crop/normalize/frame sampling)
- `utils/util.py` → generic helpers (I/O, tensor ops, logging, etc.)
- `utils/timer.py` → profiling/latency measurement

##### What the README implies about runtime flow
The README describes a concrete execution flow:
##### Output saved to `Video/` folder

This implies:
- There is a **human-in-the-loop CLI decision** (input format)
- There is a **decoder path** depending on AVI vs YUV
- Output is **file-based** (no DB, no service)

##### Component / Container view (high level)
Shows major blocks: user, CLI script, preprocess, model, postprocess, filesystem artifacts.

##### Inference pipeline view (sequence/dataflow)
Shows step-by-step: decode → frames → transforms → tokenize (VQ-VAE encoder) → motion generation (AR transformer) → decode frames → encode video → write outputs.

##### 5a) External actors
- **User / Researcher** (invokes script, selects mode, provides inputs)

##### 5b) Runtime components (code)
- **CLI Orchestrator**: `load_image.py`
- **Input Loader**: `dataload.py` (and any loader logic in `load_image.py`)
- **Decode module**:
- AVI reader (e.g., OpenCV/ffmpeg-backed)
- YUV reader (custom parsing likely driven by `yuv_json.json`)
- **Transforms**: `utils/videotransforms.py`
- **Model Inference (MAGE+)**:
- **VQ-VAE** encoder/decoder (tokenization ↔ frames)
- **Autoregressive Transformer** (motion token generation)
- **Config loader**: `config.yaml`
- **Weights loader**: `model_best.pth`
- **Postprocess / Encode**:
- convert frames to AVI/MP4 (or similar)
- **Output writer**:
- writes to `Video/` directory

##### 5c) Data stores (local filesystem)
- **Input Files**: `.avi`, `.yuv`, plus JSON metadata
- **Model Artifacts**: checkpoint + config under `models/`
- **Output Folder**: `Video/` (generated)
- (Optional) **Logs/Timing**: from `utils/timer.py`

##### User → load_image.py
(“run + choose mode”)
##### load_image.py → Input Decoder
(“AVI frames” or “YUV frames”)
##### Decoder → Transforms
(“frame tensors, normalized”)
##### Transforms → MAGE+ inference
(“preprocessed frames / conditioning image”)
##### MAGE+ inference ↔ Model Artifacts
Reads `config.yaml` + `model_best.pth`
##### MAGE+ inference → Postprocess/Encoder
(“generated frames”)
##### Encoder → Output Folder (Video/)
(“synthesized video file”)

##### Important branching to show explicitly
- A **decision diamond** (or two separate paths) for **AVI vs YUV input**:
- AVI path uses `input_avi/...` examples
- YUV path uses `yuv_json.json` + `.yuv` (frame size/format metadata)

##### Architectural patterns worth highlighting
- **Pipeline / Batch processing architecture**: deterministic stages, file-in/file-out.
- **Model-as-a-component** with **artifact-driven configuration**: `config.yaml` + `.pth`.
- **Pluggable input format adapters**: AVI reader vs YUV reader (same downstream pipeline).

These are the “design principles” that explain the structure you see in the tree.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Actors
    U["User / Researcher"]:::actor

    %% Entry / Orchestration
    subgraph "Run & Orchestration"
        CLI["CLI Orchestrator\n(load_image.py)"]:::script
        DL["Input Loader / Dataset Abstraction\n(dataload.py)"]:::script
        DECIDE{"SelectInputMode\n0=AVI | 1=YUV"}:::decision
    end

    %% Inputs (Local Filesystem)
    subgraph "Local Filesystem Inputs"
        AVI_ROOT["Sample AVI (.avi)\nCATER_GEN_v1_00010.avi"]:::datafile
        YUV_ROOT["Sample YUV (.yuv)\nCATER_GEN_v1_00007.yuv"]:::datafile
        YUV_META["YUV Metadata / Format Spec\n(yuv_json.json)"]:::datafile

        subgraph "Example AVI Inputs"
            AVI_DIR["input_avi/"]:::datadir
            AVI_EX["Example AVI\ninput_avi/videos/CATER_GEN_v1_00007.avi"]:::datafile
            TAMB["Test Spec\ninput_avi/test_ambiguous.json"]:::datafile
            TAMB1["Test Spec\ninput_avi/test_ambiguous1.json"]:::datafile
        end

        subgraph "Example YUV Inputs"
            YUV_DIR["yuv_vedios/"]:::datadir
            YUV_EX["Example YUV\nyuv_vedios/CATER_GEN_v1_00007.yuv"]:::datafile
        end
    end

    %% Decode / Preprocess
    subgraph "Decode & Preprocess"
        AVI_DEC["AVI Decoder\n(Video I/O: OpenCV/ffmpeg)"]:::proc
        YUV_DEC["YUV Decoder\n(Custom parsing using metadata)"]:::proc
        VTX["Video Transforms\n(utils/videotransforms.py)\nresize/crop/normalize/sample"]:::util
        UTIL["Generic Utilities\n(utils/util.py)"]:::util
        TIMER["Timing / Profiling\n(utils/timer.py)"]:::util
    end

    %% Model Inference (MAGE+)
    subgraph "Model Inference (MAGE+)"
        INFER["MAGE+ Inference Runtime\n(DL runtime: PyTorch)"]:::model
        VQVAE["VQ-VAE\nEncode/Decode (tokenize ↔ frames)"]:::model
        AR["Autoregressive Transformer\nMotion Token Generation"]:::model
    end

    %% Model Artifacts
    subgraph "Local Model Artifacts"
        ART_DIR["Model Artifact Store\nmodels/MAGE+/catergenv2_diverse/"]:::datadir
        CFG["Model Config\nconfig.yaml"]:::datafile
        WTS["Weights Checkpoint\nmodel_best.pth (referenced)"]:::datafileMissing
    end

    %% Postprocess / Output
    subgraph "Postprocess & Output"
        ENC["Video Encoder / Writer\n(Video I/O: OpenCV/ffmpeg)"]:::proc
        OUT["Output Folder\nVideo/ (generated)"]:::datadir
    end

    %% Dependency Spec
    REQ["Dependency Spec\nour_requirements.txt"]:::meta

    %% Relationships (vertical flow)
    U -->|"run+choose_mode"| CLI
    CLI -->|"uses"| DL
    CLI -->|"prompt_selection"| DECIDE

    %% Input selection fan-in
    DECIDE -->|"mode=AVI"| AVI_DEC
    DECIDE -->|"mode=YUV"| YUV_DEC

    %% Inputs into decoders
    AVI_ROOT -->|"raw_video_bytes"| AVI_DEC
    AVI_DIR -->|"dataset_path"| DL
    AVI_EX -->|"raw_video_bytes"| AVI_DEC
    TAMB -->|"eval_spec"| DL
    TAMB1 -->|"eval_spec"| DL

    YUV_ROOT -->|"raw_yuv_stream"| YUV_DEC
    YUV_DIR -->|"dataset_path"| DL
    YUV_EX -->|"raw_yuv_stream"| YUV_DEC
    YUV_META -->|"format_metadata"| YUV_DEC

    %% Decode -> transforms
    AVI_DEC -->|"decoded_frames"| VTX
    YUV_DEC -->|"decoded_frames"| VTX

    %% Utils around pipeline
    CLI -->|"helpers/io"| UTIL
    CLI -->|"measure_latency"| TIMER
    VTX -->|"helpers"| UTIL
    INFER -->|"helpers"| UTIL
    INFER -->|"timing_hooks"| TIMER

    %% Transforms -> model inference
    VTX -->|"normalized_tensors\n(ref_image+frames)"| INFER

    %% Model internal composition
    INFER -->|"encode_frames"| VQVAE
    VQVAE -->|"latent_tokens"| AR
    AR -->|"generated_motion_tokens"| VQVAE
    VQVAE -->|"decoded_generated_frames"| INFER

    %% Model artifacts
    INFER -->|"load_config"| CFG
    INFER -->|"load_weights"| WTS
    CFG -->|"located_in"| ART_DIR
    WTS -->|"located_in"| ART_DIR

    %% Model output -> encode -> filesystem
    INFER -->|"generated_frames"| ENC
    ENC -->|"write_synth_video"| OUT

    %% Requirements reference
    CLI -->|"installed_deps"| REQ
    INFER -->|"installed_deps"| REQ
    ENC -->|"installed_deps"| REQ

    %% Click events (mapped components)
    click CLI "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/load_image.py"
    click DL "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/dataload.py"
    click AVI_ROOT "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/CATER_GEN_v1_00010.avi"
    click YUV_ROOT "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/CATER_GEN_v1_00007.yuv"
    click YUV_META "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/yuv_json.json"
    click AVI_DIR "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/tree/main/input_avi/"
    click AVI_EX "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/input_avi/videos/CATER_GEN_v1_00007.avi"
    click TAMB "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/input_avi/test_ambiguous.json"
    click TAMB1 "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/input_avi/test_ambiguous1.json"
    click YUV_DIR "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/tree/main/yuv_vedios/"
    click YUV_EX "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/yuv_vedios/CATER_GEN_v1_00007.yuv"
    click ART_DIR "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/tree/main/models/MAGE+/catergenv2_diverse/"
    click CFG "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/models/MAGE+/catergenv2_diverse/config.yaml"
    click VTX "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/utils/videotransforms.py"
    click UTIL "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/utils/util.py"
    click TIMER "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/utils/timer.py"
    click REQ "https://github.com/aman-22byte/prism_iiitdmk_24vi20iiitk/blob/main/our_requirements.txt"

    %% Styles
    classDef actor fill:#e6e6e6,stroke:#666,color:#111,stroke-width:1px
    classDef script fill:#cfe8ff,stroke:#2b6cb0,color:#0b1f33,stroke-width:1px
    classDef decision fill:#fff3bf,stroke:#b08900,color:#3b2f00,stroke-width:1px
    classDef util fill:#c7f0ea,stroke:#0f766e,color:#073b3a,stroke-width:1px
    classDef proc fill:#d9f99d,stroke:#3f6212,color:#1a2e05,stroke-width:1px
    classDef model fill:#e9d5ff,stroke:#6b21a8,color:#2e1065,stroke-width:1px
    classDef datafile fill:#ffd8a8,stroke:#c2410c,color:#3b1a07,stroke-width:1px
    classDef datadir fill:#ffedd5,stroke:#9a3412,color:#3b1a07,stroke-width:1px
    classDef meta fill:#fee2e2,stroke:#991b1b,color:#450a0a,stroke-width:1px
    classDef datafileMissing fill:#ffe0b2,stroke:#9a3412,color:#3b1a07,stroke-width:2px,stroke-dasharray: 6 4
```
