How ganchaofan0000/dac works
Project type and purpose
• DAC is a research‐oriented deep learning framework (PyTorch) implementing a 2D–3D cross‐modal retrieval method with noisy label correction.
• It is not a client‐server or web application, but a modular training and evaluation pipeline for experiments from the ACM MM 2024 paper.
Top‐level component analysis
File tree:
• data/
– train_200.json: noisy labels for Objaverse‐N200
• losses/
– CCL_loss.py, MAE.py, cross_modal_loss.py
• models/
– image_encoder.py, pt_encoder.py, fused_layer.py, resnet.py
• tools/
– dataset.py (data loading & preprocessing), utils.py
• train.sh, train_dual.py (entry points)
Key architectural patterns and dependencies
• Modular separation:
a. Data ingestion & preprocessing (tools/)
b. Model definitions (models/)
c. Loss computations (losses/)
d. Training orchestration (train_dual.py / train.sh)
• Pluggable components: easy to swap encoders or loss functions.
• Single‐process, GPU‐driven training loop (no microservices or distributed components here).
A. Data Store (data/)
– JSON label file, point-cloud features, 2D images (external storage)
B. Data Loader & Preprocessor (tools/dataset.py)
– Reads data, applies transforms, yields batches
C. Model Encoders
1. Image Encoder (models/image_encoder.py + resnet backbone)
2. Point-cloud Encoder (models/pt_encoder.py)
D. Fusion Module (models/fused_layer.py)
– Combines 2D and 3D embeddings
E. Loss Modules (losses/)
– Cross‐modal loss, CCL, MAE
– Adaptive alignment & correction logic
F. Training Orchestrator (train_dual.py + train.sh)
– Instantiate DataLoader, Models, Losses, Optimizer
– Runs forward/backward passes, logging, checkpointing
G. External Pretrained Models
– Openshape, ResNet weights, Uni3D for label generation
H. Outputs
– Checkpoints, metrics, tensorboard logs, evaluation results
Relationships and data flows
1. The Training Orchestrator triggers the Data Loader.
2. Data Loader fetches raw images & point features, outputs paired batches.
3. Batches flow into the two encoders in parallel.
4. Embeddings converge into the fused_layer module.
5. Fused embeddings + true/noisy labels are passed to Loss Modules.
6. Loss Modules compute multiple sub‐losses (cross‐modal, CCL, MAE), sum them.
7. Backprop through the encoders and fusion module.
8. Optimizer updates model weights.
9. Loop until convergence; logs and checkpoints are persisted.
Architectural patterns & design principles
• Single Responsibility: Each folder handles one aspect (data, model, loss, utils).
• Dependency Injection: train_dual.py wires together components at runtime.
• Modular Pluggability: New losses or encoders can be added by dropping files into losses/ or models/.
• Clear separation between data pipeline, model definition, and training logic.
– Clear textual labels on every component
– Version identifiers for key dependencies (e.g., PyTorch 1.7.1, Python 3.8)