How josh00-lu/csbnet works
Project Type and Purpose
• This is a research‐oriented deep learning project—specifically, a PyTorch implementation of an arbitrary video style‐transfer network (CSBNet).
• It’s not a web or full‐stack service but a data‐processing/model pipeline: you feed it content images or videos plus style images, it extracts features, runs them through three custom network modules (Crystallization, Separation, Blending), and outputs stylized images/videos.
File‐Structure Analysis
Top‐level scripts:
• train.py ← orchestrates the training loop (data loading, forward/backward pass, loss computation, checkpointing)
• test_image.py ← runs inference on images
• test_video.py ← runs inference on videos
Core modules:
• net.py ← defines the CSBNet architecture (Cr, Sp, Bd modules, encoder/decoder)
• sampler.py ← handles video‐frame sampling or batch creation
• function.py ← helper routines: I/O, pre/post‐processing, saving outputs
Models directory:
• models/vgg_normalised.pth ← pretrained VGG encoder weights
• (optional) csbnet pretrained .pth for quick testing
Key Architectural Patterns
• Pipeline pattern: sequential data flow from input → encoder → Cr → Sp → Bd → decoder → output
• Modular NN design: three specialized “blocks” for crystallization, separation, and blending wrapped in one top‐level CSBNet class
• Training vs. inference modes: shared model graph but different entry points and I/O scripts
Components
A. Data Ingestion
• Content Loader: reads images or videos (via OpenCV, imageio‐ffmpeg)
• Frame Sampler: splits videos into frames or batches of images
• Style Loader: reads one or multiple style images
B. Feature Extraction
• VGG Encoder (pretrained weights)
C. CSBNet Core Modules (inside net.py)
• Crystallization Module (Cr)
• Separation Module (Sp)
• Blending Module (Bd)
D. Decoder / Reconstruction
• Upsampling blocks to form final stylized frames
E. Loss Computation (during training)
• Content loss, style loss, component enhancement losses
F. Script Orchestrators
• train.py
• test_image.py / test_video.py
G. Persistence & Configuration
• Model checkpoint saving/loading (models/)
• Command‐line args (KC, KS, batch_size, GPU IDs, etc.)
H. External Dependencies
• PyTorch, torchvision, NumPy, SciPy, OpenCV, imageio‐ffmpeg
Relationships & Data Flow
1. Start at train.py or test_*.py →
2. Invoke Loader → Sampler
3. Feed batch into VGG Encoder
4. Pass features to Cr → output crystal nuclei
5. Feed nuclei to Sp → separate content/style features
6. Feed separated features into Bd → blended features
7. Decoder reconstructs stylized images/frames
8. (Training only) compute losses → backprop → update CSBNet parameters
9. Write outputs to disk and (if training) save checkpoints
Capturing Essential Elements
• That both image and video inference share the same backbone, differing only at the input loader stage.
• Multi‐GPU or batch_size config flowing into train.py.
• Every module from net.py is represented with its role named exactly as in code (Cr, Sp, Bd).