How haddybhaiya/sem-i-con works
This repository is an end-to-end computer vision pipeline for semiconductor defect classification from grayscale SEM images, with a clear separation between model development, edge deployment, and evaluation.
At a high level, the architecture has five major subsystems:
1. Data preparation and test-set construction
2. Model training and validation
3. Compression and export for edge deployment
4. ONNX-based edge inference and auto-edge logic
5. Evaluation, benchmarking, and reporting
The project is centered on a two-model strategy:
• a larger teacher model for higher-accuracy learning
• a lightweight student model for edge inference
Main runtime and technology boundaries:
• Training and evaluation are implemented in Python with PyTorch
• Edge deployment uses ONNX Runtime on CPU
• The models operate on grayscale SEM images resized to 224x224
• Quantization and export utilities exist to reduce model size and improve edge efficiency
Repository structure by subsystem
1. Training subsystem
Located under training/
Files:
• training/model.py
• training/train.py
• training/trainer.py
• training/validate.py
Role:
• Defines the model architectures and training workflow.
• Supports the two core model families described in the README:
- ConvNeXt-like teacher model
- MobileNetV3-based student model
• Handles optimization, validation, and likely experiment iteration.
Architecture boundary:
• Inputs are labeled grayscale SEM images from a dataset directory.
• Outputs are trained PyTorch models saved as .pth checkpoints.
• This subsystem is the source of the model artifacts later exported to ONNX.
2. Edge export and inference subsystem
Located under edge/
Files:
• edge/export_onnx.py
• edge/export_onnx_int8.py
• edge/torch_quantize.py
• edge/edge_infer.py
• edge/auto_edge.py
• edge/auto_quantize.py
• edge/batch_test.py
• edge/check.py
Role:
• Converts trained PyTorch models into deployable ONNX models.
• Supports quantization paths, including integer quantization.
• Runs single-image inference and batch testing in an edge-compatible way.
• Contains the “auto edge” logic mentioned in the README, which appears to adapt the inference/deployment path depending on the model or constraints.
Key architectural flow:
• PyTorch checkpoint in models/ or training outputs
-> export scripts
-> ONNX model artifacts in models/
-> ONNX Runtime inference in edge_infer.py or auto_edge.py
Important boundary:
• This subsystem is the deployment boundary between research/training code and runtime inference code.
• It isolates PyTorch-only training logic from ONNX-based execution.
Inference pipeline described in the README:
• grayscale loading with PIL
• resize to 224x224
• normalize
• reshape to [1, 1, 224, 224]
• ONNX Runtime inference
• softmax and confidence thresholding
• semantic guard for ambiguous outputs
3. Evaluation subsystem
Located under evaluation/
Files:
• evaluation/create_test_dataset.py
• evaluation/create_test_set.py
• evaluation/edge_benchmark.py
• evaluation/metrics.py
• evaluation/phase1_eval.py
• evaluation/pytorch_acc.py
• evaluation/report.json
Role:
• Builds or curates test sets.
• Measures classification accuracy and other metrics.
• Benchmarks edge inference performance.
• Produces report artifacts for phase-1 analysis.
Data flow:
• Test images and labels are assembled into evaluation datasets.
• Models are run through evaluation scripts either in PyTorch or ONNX form.
• Metrics are written into JSON report files, such as report.json.
• The README also references evaluation outputs like confusion matrices and class-wise accuracy.
Architectural boundary:
• This subsystem sits between training/export and deployment verification.
• It validates both accuracy and edge performance, so it is the key checkpoint before using a model in the field.
4. Learning and self-learning subsystem
Located under learning/
Files:
• learning/self_learn.py
• learning/buffer/logs.csv
Role:
• Implements the repository’s adaptive or iterative learning loop.
• The README mentions a “self learn pipeline with integrated auto edge system,” and this folder likely contains the logic for collecting inference outputs or feedback for future retraining.
• The buffer/logs.csv file suggests a log-based memory or replay mechanism.
Architectural role:
• This is the feedback loop boundary.
• It connects deployed inference back to future training or adaptation, rather than being part of the static training pipeline.
5. Artifacts and model storage
Located under models/
Files:
• models/edge_model.onnx.data
• models/edge_model_int8.onnx.data
• models/mobilenetv3_sem.onnx.data
Role:
• Stores model data files associated with ONNX exports.
• The repository README also references .pth and .onnx files that may be external or not checked in.
Architectural meaning:
• This is the artifact boundary between code and deployable model binaries.
• The ONNX files are the deployment target for edge runtime.
• The .onnx.data files indicate externalized weights or large model data bundled with ONNX export.
Supporting / experiment files
• phs1/ and phs2/ contain phase-specific results, logs, and datasets/prediction scripts.
• ipynb checks/ and root notebooks are experiment and inspection tools, not production flow.
• semIcon.pptx.pdf is documentation rather than runtime architecture.
• requirements.txt defines the Python environment.
End-to-end data flow
A realistic system flow in this repository is:
1. Input data acquisition
• Raw grayscale SEM defect images are collected, labeled, and possibly synthesized for balance and robustness.
2. Dataset preparation
• Training and evaluation scripts assemble train/validation/test splits.
• Synthetic augmentation helps compensate for limited real data.
3. Model training
• The teacher model learns higher-capacity features.
• The student model is trained, likely using distillation from the teacher.
4. Model export and compression
• Trained models are exported from PyTorch to ONNX.
• Quantization scripts produce smaller or faster variants, including INT8 models.
5. Edge deployment
• ONNX Runtime executes the model on CPU.
• Preprocessing is grayscale-safe and fixed to the expected input geometry.
6. Post-processing / reliability logic
• Softmax confidence is computed.
• A semantic guard handles ambiguous outputs, reducing unsafe or uncertain classifications.
7. Evaluation and benchmarking
• Accuracy and edge performance are measured.
• Confusion matrices and JSON reports summarize performance.
8. Optional feedback loop
• Inference or evaluation results can be logged into the learning subsystem for future iteration.
What to emphasize in an architecture diagram
If you are drawing this repo, the most important boxes and boundaries are:
• Data source / synthetic data generation
• Training pipeline with teacher and student models
• Export/quantization pipeline
• ONNX Runtime edge inference service
• Evaluation and benchmarking pipeline
• Self-learning feedback loop
• Model artifact store
The central design pattern is a train-export-deploy loop:
• PyTorch for model development
• ONNX for deployment
• Evaluation as a gate between them
• Optional self-learning to feed operational data back into model improvement
In short, this is not just a model-training repo. It is an edge-AI product pipeline for semiconductor defect detection, with explicit support for model compression, deployment, evaluation, and iterative improvement.
Main entry points: train.py Train loop · edge_infer.py Edge infer · models Model store · create_test_dataset.py Test set build
Components 23
Data prep
Dataset construction and split preparation for training and evaluation.
- SEM images · Grayscale semiconductor defect images used as the core input domain.
- Test set build · Creates curated test sets and evaluation splits from image data.
evaluation/create_test_dataset.py
Training
PyTorch model development, optimization, and validation.
- Model defs · Defines the teacher and student architectures used in training.
training/model.py - Train loop · Coordinates optimization and experiment runs.
training/train.py - Trainer · Encapsulates the training workflow, iteration, and checkpoint handling.
training/trainer.py - Validate · Runs model validation and reports performance during training.
training/validate.py - Teacher · ConvNeXt-like model used for higher-accuracy learning.
- Student · MobileNetV3-based lightweight model intended for deployment.
Edge deployment
ONNX export, quantization, and CPU inference flow.
- Export ONNX · Converts trained PyTorch checkpoints into ONNX artifacts.
edge/export_onnx.py - INT8 export · Produces integer-quantized ONNX variants for smaller edge models.
edge/export_onnx_int8.py - Torch quantize · Applies PyTorch-side quantization utilities before export.
edge/torch_quantize.py - Edge infer · Runs single-image ONNX inference on CPU with preprocessing and softmax post-processing.
edge/edge_infer.py - Auto edge · Selects or adapts the edge inference path based on model or constraints.
edge/auto_edge.py - Auto quantize · Automates choosing and applying quantization settings.
edge/auto_quantize.py - Batch test · Runs bulk edge inference checks over a dataset.
edge/batch_test.py - Check · Performs consistency or readiness checks on exported edge models.
edge/check.py
Evaluation
Accuracy, benchmarking, and reporting for phase analysis.
- Metrics · Computes classification metrics and summary statistics.
evaluation/metrics.py - Phase eval · Produces phase-level evaluation outputs and analysis artifacts.
evaluation/phase1_eval.py - Edge bench · Measures edge runtime performance and throughput.
evaluation/edge_benchmark.py - PyTorch acc · Evaluates model accuracy in the native PyTorch runtime.
evaluation/pytorch_acc.py
Feedback loop
Logs and self-learning path that can feed future retraining.
- Self learn · Collects operational feedback for iterative adaptation or retraining.
learning/self_learn.py - Logs · CSV log buffer used to retain learning or inference history.
learning/buffer/logs.csv
Other components
- Model store · Holds deployable ONNX weight data for edge models.
models
Connections 26
- SEM images → Test set build: split · Raw SEM images are organized into curated test and evaluation sets.
- SEM images → Train loop: train on · Training consumes labeled grayscale SEM images.
- Model defs → Train loop: define · Model definitions are instantiated by the training entrypoint.
- Train loop → Trainer: orchestrate · The training entrypoint delegates execution to the trainer.
- Trainer → Validate: validate · Training periodically invokes validation.
- Teacher → Student: distill · The teacher guides the smaller student model.
- Train loop → Teacher: fit · Teacher model training produces high-capacity checkpoints.
- Train loop → Student: fit · Student model training produces compact checkpoints.
- Teacher → Export ONNX: checkpoint · Trained checkpoints are exported for deployment.
- Student → Export ONNX: checkpoint · Student checkpoints are exported for edge use.
- Torch quantize → INT8 export: prepare · Torch quantization feeds the INT8 export path.
- Export ONNX → Model store: write · Exported ONNX artifacts are stored in the model directory.
- INT8 export → Model store: write · Quantized ONNX artifacts are stored in the model directory.
- Model store → Edge infer: load · Edge inference loads the deployable ONNX model artifacts.
- Auto quantize → Torch quantize: choose · Automated quantization selects a compression strategy.
- Auto edge → Edge infer: route · Auto-edge logic routes requests to the runtime path.
- Edge infer → Batch test: batch · The inference path can be exercised in batch mode.
- Edge infer → Metrics: score · Inference outputs are summarized into metrics.
- Batch test → Edge bench: benchmark · Batch tests feed edge benchmarking.
- PyTorch acc → Phase eval: report · Native-runtime accuracy contributes to phase evaluation outputs.
- Edge bench → Phase eval: report · Edge performance results contribute to phase evaluation outputs.
- Test set build → PyTorch acc: evaluate · Curated test sets are used for PyTorch accuracy runs.
- Test set build → Batch test: evaluate · Curated test sets are used for batch edge testing.
- Self learn → Logs: record · Operational feedback is persisted to the learning buffer.
- Logs → Train loop: feedback · Logged feedback can inform future training iterations.
- Check → Model store: inspect · Checks verify exported model artifacts before deployment.