How rfarell/guided-dropout-experiments works
Main Components
A. Data Pipeline
• CIFAR-10 Dataset Input (3x32x32 images)
• Data Preprocessing/Flattening (3072 features)
B. Model Architecture (Three Variants)
1. Guided Dropout Model
2. Regular Dropout Model
3. No Dropout Model
C. Training System
• Loss Calculation
• Optimization
• Metrics Collection
D. Evaluation System
• Test Accuracy Measurement
• Signal Strength Analysis
• Visualization Generation
[Data Layer]
[Model Layer]
[Training Layer]
[Evaluation Layer]
Specific Component Details
For each model variant, show:
• Input Layer (3072 nodes)
• Hidden Layer 1 (8192 nodes)
• Hidden Layer 2 (8192 nodes)
• Output Layer (10 nodes)
• Regular connections
• Dropout connections (for Regular Dropout)
• Guided Dropout connections (with strength parameter t_j)
Key Flows
A. Training Flow:
Input → Flatten → Hidden Layers → Output → Loss → Backprop
B. Strength Calculation Flow:
Activations → Mean/Variance/SNR Calculations → Visualization
Implementation Details
• ReLU Activation: f(z) = max(0, z)
• Guided Dropout: a_j^(l) = t_j^(l) * f(z_j^(l))
• Signal Strength Metrics:
• Mean Activation
• Variance
• SNR
Additional Elements
• Strength parameter distribution
• Dropout mechanism comparison
• Metrics visualization pipeline
The relationship between components
Keep the following files in mind when showing component relationships:
• models/guided_dropout_model.py
• models/regular_dropout_model.py
• models/no_dropout_model.py
• train.py
• visualize_activation_strengths.py