How trivikram-yeluri/gan-project works
Project type and purpose
• This is a full-stack machine learning system focused on anti-malware detection.
• It combines data pipelines (datasets/datalakes), deep learning training (GANs, CNNs), evaluation/analysis notebooks, and eventual deployment (API, real-time detection).
File structure
Top-level:
• README.md – project overview
• ml-antimalware.ipynb – primary Jupyter notebook for training/testing
• dataset_malwares.csv, dataset_test.csv – raw data files
• yaml/ – configuration files (e.g., model or pipeline specs)
As described in README (even if not present in the file tree):
• datasets/ – organizes raw and preprocessed data
• models/ – stores trained GAN, CNN, Hybrid models
• notebooks/ – exploratory, analysis, reporting
• src/ – core pipeline scripts (data ingestion, preprocessing, training loops, evaluation, utilities)
README insights
• Core ML components: GAN generator/discriminator, CNN classifier, hybrid deep-learning model.
• Workflow phases: data ingestion → synthetic sample generation → classifier training → performance analysis → future deployment.
• Dependencies: TensorFlow/PyTorch (GAN, CNN), Pandas/NumPy (data), Matplotlib/Seaborn (plots), Jupyter.
a. Main components:
1. Data Layer
• Raw Dataset (malware & benign CSVs)
• Preprocessor (feature extraction, normalization)
2. Training Pipelines
• GAN Module: Generator ↔ Discriminator loop
• CNN Classifier Module
• Hybrid Model Module (ensemble or stacked)
3. Storage
• Model Store (models/)
• Logs & Metrics (accuracy, FPR, training history)
4. Analysis & Monitoring
• Jupyter Notebooks (notebooks/)
• Visualization dashboards
5. Deployment/API (future)
• RESTful service wrapping classifier
• Real-time feature extractor → classifier → decision
b. Relationships & Data Flow
• Raw Data → Preprocessor → Feature Batches →
→ GAN Training (generates synthetic malware samples) → Augmented Dataset →
→ CNN/Hybrid Training → Trained Models → Model Store
• Trained Models + Held-out Test Data → Evaluation Module → Metrics →
→ Reports/Notebooks
• Future: Incoming files → Feature Extractor → Classifier API → Predictions
c. Architectural Patterns & Design Principles
• Modular pipelines: separate data, training, inference, evaluation.
• MVC-style separation:
• Model (GAN, CNN, Hybrid in models/)
• Controller (src/scripts orchestrating training/eval)
• View (Jupyter notebooks, plots)
• Reproducibility via yaml configs.
• Extensibility: plug in new model architectures or datasets by adding to src/ and updating configs.
d. Key Technologies & Frameworks
• TensorFlow/PyTorch for deep learning
• Pandas/NumPy for data manipulation
• Jupyter for interactive analysis
• YAML for pipeline/config management
• (Future) Flask/FastAPI for deployment