How aman-22byte/prism_iiitdmk_24vi20iiitk works
Project type + purpose (what you’re diagramming)
This is an offline ML data-generation pipeline (not a web service): it generates synthetic videos by adding learned motion to static image content to create a video denoising dataset.
Core idea from the README:
• Based on MAGE+ (derived from MAGE): VQ-VAE for tokenization/latent codebook + autoregressive Transformer for motion modeling.
• You run an inference script that takes either AVI or YUV input, extracts frames / a reference image, runs the pretrained model, and outputs a synthesized video into a local folder.
File tree-driven architecture (map folders/files to components)
From the tree, the likely major components are:
Entry points / orchestration
• load_image.py
• dataload.py
Likely a dataset/input loader abstraction used by the run script and/or transforms.
Model assets (pretrained + config)
• models/MAGE+/catergenv2_diverse/config.yaml
• README references models/MAGE+/catergenv2_diverse/model_best.pth
• Config (model hyperparams, tokenizer params, etc.)
• Weights checkpoint
Shared utilities / processing blocks
• utils/videotransforms.py → Video transforms (resize/crop/normalize/frame sampling)
• utils/util.py → generic helpers (I/O, tensor ops, logging, etc.)
• utils/timer.py → profiling/latency measurement
What the README implies about runtime flow
The README describes a concrete execution flow:
Output saved to Video/ folder
This implies:
• There is a human-in-the-loop CLI decision (input format)
• There is a decoder path depending on AVI vs YUV
• Output is file-based (no DB, no service)
Component / Container view (high level)
Shows major blocks: user, CLI script, preprocess, model, postprocess, filesystem artifacts.
Inference pipeline view (sequence/dataflow)
Shows step-by-step: decode → frames → transforms → tokenize (VQ-VAE encoder) → motion generation (AR transformer) → decode frames → encode video → write outputs.
5a) External actors
• User / Researcher (invokes script, selects mode, provides inputs)
5b) Runtime components (code)
• CLI Orchestrator: load_image.py
• Input Loader: dataload.py (and any loader logic in load_image.py)
• Decode module:
• AVI reader (e.g., OpenCV/ffmpeg-backed)
• YUV reader (custom parsing likely driven by yuv_json.json)
• Transforms: utils/videotransforms.py
• Model Inference (MAGE+):
• VQ-VAE encoder/decoder (tokenization ↔ frames)
• Autoregressive Transformer (motion token generation)
• Config loader: config.yaml
• Weights loader: model_best.pth
• Postprocess / Encode:
• convert frames to AVI/MP4 (or similar)
• Output writer:
• writes to Video/ directory
5c) Data stores (local filesystem)
• Input Files: .avi, .yuv, plus JSON metadata
• Model Artifacts: checkpoint + config under models/
• Output Folder: Video/ (generated)
• (Optional) Logs/Timing: from utils/timer.py
User → load_image.py
(“run + choose mode”)
load_image.py → Input Decoder
(“AVI frames” or “YUV frames”)
Decoder → Transforms
(“frame tensors, normalized”)
Transforms → MAGE+ inference
(“preprocessed frames / conditioning image”)
MAGE+ inference ↔ Model Artifacts
Reads config.yaml + model_best.pth
MAGE+ inference → Postprocess/Encoder
(“generated frames”)
Encoder → Output Folder (Video/)
(“synthesized video file”)
Important branching to show explicitly
• A decision diamond (or two separate paths) for AVI vs YUV input:
• AVI path uses input_avi/... examples
• YUV path uses yuv_json.json + .yuv (frame size/format metadata)
Architectural patterns worth highlighting
• Pipeline / Batch processing architecture: deterministic stages, file-in/file-out.
• Model-as-a-component with artifact-driven configuration: config.yaml + .pth.
• Pluggable input format adapters: AVI reader vs YUV reader (same downstream pipeline).
These are the “design principles” that explain the structure you see in the tree.