How vinaybr03/face-liveness-detection works
1. Repository purpose
- This is a face-liveness detection system that classifies a short clip as real or fake/spoofed.
- The architecture is organized around a single model family that is trained in PyTorch, exported to ONNX, then deployed as TFLite or ONNX for inference.
2. Core ML boundary
- The central model lives in model.py and is trained/tested as a multi-modal classifier.
- Inputs are described as:
- a short image clip of face frames
- an 8-dimensional sensor vector stream per timestep
- The important architectural boundary is that the model consumes a fixed-length clip, not a single frame, so inference and training both operate on sequence windows.
3. Data layer
- data/train/{real,fake} and data/test/{real,fake} provide a binary-labeled dataset split.
- dataset.py is the ingestion layer that turns the folder structure into model-ready samples, applies preprocessing, and likely assembles clip-level examples.
- The repo’s data contract is: RGB images resized to 224×224 plus sensor features normalized/packed into the clip.
4. Training subsystem
- train.py is the training entry point and produces the canonical PyTorch artifact liveness_model.pth.
- Training output also includes training_history.json, which feeds the analysis pipeline.
- analyze_results.py is a separate post-training subsystem for metrics, plots, and confusion matrix generation, keeping evaluation out of the training loop.
5. Model export pipeline
- The repository has a clear conversion chain:
- convert_torch_to_onnx.py converts PyTorch weights to liveness_model.onnx
- convert_onnx_to_tflite.py produces a FP32 TFLite model
- convert_onnx_to_tflite_int8.py produces an INT8 quantized model
- This export boundary is important: training happens in PyTorch, but deployment targets are lighter runtimes optimized for mobile or edge use.
6. Deployed inference formats
- Three model artifacts are kept in the repo root:
- liveness_model.pth for training/experimentation
- liveness_model.onnx for portable runtime support
- liveness_model_fp32.tflite and liveness_model_int8.tflite for mobile/embedded inference
- The INT8 model is the preferred low-latency deployment path.
7. API/server layer
- app.py, app_tflite32.py, and app_onnx.py are three alternative serving frontends over the same detection logic.
- They represent a deployment abstraction boundary: same app behavior, different inference backends.
- The README indicates Flask is used to expose the service on localhost:5000.
8. External interaction flow
- The server exposes:
- POST /detect for face localization
- POST /predict for liveness classification
- GET / for the web UI
- The /predict payload combines the two modalities: a 10-frame image_clip plus 10 sensor readings.
- This makes the request contract clip-based rather than image-only.
9. Browser/UI assets
- templates/index.html is the UI shell for the Flask app.
- static/models/tiny_face_detector_model-* suggests the frontend also performs client-side or browser-side face detection using a prepackaged detector model.
- That creates a split architecture: browser handles capture/face localization UI, while the Python backend handles liveness inference.
10. Video inference path
- predict_video.py and predict_video_onnx.py are offline/CLI inference paths for processing video files.
- These scripts are separate from the Flask app, so the repo supports both interactive service deployment and batch evaluation on recorded video.
11. Artifacts and supporting outputs
- images/ stores generated plots/figures, likely from analysis or documentation.
- training_history.json acts as the bridge between training and evaluation reporting.
- The repo keeps both code and trained artifacts together, so architecture diagrams should show the model lifecycle end-to-end.
12. Technology and runtime boundaries
- The stack is primarily PyTorch + TensorFlow/TFLite + ONNX + Flask, with opencv-python, numpy, Pillow, scikit-learn, matplotlib, and seaborn supporting preprocessing, inference, and evaluation.
- The material architectural choice is portability: train in PyTorch, export through ONNX, deploy with TFLite or ONNX depending on performance/runtime constraints.
13. What to draw as the main systems
- A useful diagram would show four main zones:
- Dataset and preprocessing
- Training and evaluation
- Model conversion/export
- Deployment/inference
- The key flows are:
- data folders → dataset loader → PyTorch training
- trained weights → ONNX/TFLite conversions
- deployed model → Flask/video inference
- training logs → analysis/metrics
14. Overall architectural takeaway
- This is not a monolithic app; it is a model development and deployment pipeline with multiple inference entry points.
- The most important design decisions are the clip-based multimodal input, the binary real/fake output, and the runtime split between high-level training and edge-optimized serving.
Main entry points: liveness_model.onnx ONNX model · index.html Web UI · model.py Liveness model · app.py Flask app
Components 20
Data layer
Dataset folders and ingestion into clip-level samples.
- Train/Test data · Real/fake folders for train and test splits.
data - Dataset · Builds fixed-length multimodal samples from folder structure and preprocessing.
dataset.py
Training & eval
PyTorch training and post-training analysis.
Model export
Conversion path from PyTorch to ONNX and TFLite.
- To ONNX · Converts PyTorch weights into portable ONNX format.
convert_torch_to_onnx.py - To TFLite · Produces FP32 TFLite model from ONNX.
convert_onnx_to_tflite.py - To INT8 · Produces INT8 quantized TFLite model for low-latency deployment.
convert_onnx_to_tflite_int8.py - PTH model · Canonical PyTorch artifact for experimentation and training reuse.
liveness_model.pth - ONNX model · Runtime-neutral model used by ONNX inference and conversion paths.
liveness_model.onnx - FP32 TFLite · Floating-point TFLite deployment artifact.
liveness_model_fp32.tflite - INT8 TFLite · Preferred edge deployment artifact.
liveness_model_int8.tflite
Serving & inference
Flask app, CLI video inference, and deployment backends.
- Flask app · Main HTTP service exposing detection and prediction endpoints.
app.py - ONNX app · Serving frontend using ONNX inference.
app_onnx.py - TFLite app · Serving frontend using TFLite inference.
app_tflite32.py - Video infer · Offline video processing path using the Python model runtime.
predict_video.py - Video ONNX · Offline video inference path backed by ONNX.
predict_video_onnx.py
Browser UI
Web template and bundled face detector assets.
- Web UI · Browser interface shell for capture and prediction.
templates/index.html - Face detector · Bundled frontend detector assets for client-side face localization.
static/models/tiny_face_detector_model-weights_manifest.json
Other components
- Liveness model · Consumes image clips plus 8D sensor streams and predicts real vs fake.
model.py
Connections 22
- Train/Test data → Dataset: samples · Dataset folders are ingested into clip-level training examples.
- Dataset → Liveness model: clips · Preprocessed image clips and sensor vectors feed the model.
- Liveness model → Train: train · Training optimizes the multimodal classifier.
- Train → PTH model: save · Produces the canonical PyTorch checkpoint.
- Train → History: log · Stores training metrics and history.
- History → Analyze: report · Post-training analysis reads the history file.
- PTH model → To ONNX: export · PyTorch weights are converted to ONNX.
- To ONNX → ONNX model: write · Creates the portable ONNX model.
- ONNX model → To TFLite: convert · ONNX is converted to FP32 TFLite.
- ONNX model → To INT8: quantize · ONNX is converted to INT8 TFLite.
- To TFLite → FP32 TFLite: write · Writes the floating-point TFLite artifact.
- To INT8 → INT8 TFLite: write · Writes the quantized TFLite artifact.
- Liveness model → Flask app: predict · Core liveness logic is served through Flask.
- ONNX model → ONNX app: serve · ONNX backend powers one deployment frontend.
- FP32 TFLite → TFLite app: serve · TFLite backend powers another deployment frontend.
- INT8 TFLite → TFLite app: serve · Quantized TFLite is the preferred low-latency backend.
- Liveness model → Video infer: batch · Offline video inference uses the model runtime.
- ONNX model → Video ONNX: batch · Offline video inference can also run through ONNX.
- Flask app → Web UI: ui · The web service renders the browser interface.
- Web UI → Face detector: detect · Frontend uses bundled face detector assets for localization.
- Web UI → Flask app: POST /predict · Browser submits clip-based multimodal inference requests.
- Web UI → Flask app: POST /detect · Browser requests face localization.