How abekek/cse398_project works
Project Type & Purpose
• This project is a specialized assistive technology application that uses computer vision and deep learning techniques to facilitate eye-gaze based control for a keyboard interface.
• It involves real-time camera input processing, facial/eye landmark detection, neural network inference, and a user interface for keyboard interactions.
• The README clearly shows a training/inference pipeline as well as a demo interface, which means you need to capture both the perception (image processing/inference) and output (keyboard interaction) sides of the system.
File Structure for Architectural Clues
• Top-level Organization:
• The src Directory:
• Contains the main operational scripts (e.g., main.py) that likely tie together the system’s initialization, video capture, and inference operation.
• Contains multiple modules categorized by functionality:
▪ Detection Methods: Files such as CustomLandmarkDetection.py, FaceMeshMethod.py, and NNMethod.py hint at different algorithms or approaches for detecting eye landmarks.
▪ UI/Interaction: SelectionWindow.py suggests a GUI component enabling user interactions.
▪ Model Definitions: The models directory with eyenet.py, layers.py, and losses.py contains code for the neural network architecture involved in eye detection/prediction.
▪ Utility Functions: The util subdirectory (eye_prediction.py, gaze.py, softargmax.py, etc.) supports various computations essential for processing and interpreting eye movements.
▪ Notebooks: The notebooks folder, which contains Jupyter notebooks, indicates that part of the project’s development or exploratory data analysis also revolves around eye gaze and facial landmark detection.
▪ Model Conversion & Saved Models: Files like onnx_convert.py and the saved_models directory hold pre-trained models and conversion scripts, signifying that deployment also includes optimizing model representation.
Main Components
a. Input and Capture:
• External Camera: Capture of real-time video feed.
• Pre-Processing Module: Prepares the input (resizing, normalization) for detection.
b. Detection & Inference:
• Detection Modules: Include the various methodologies – CustomLandmarkDetection, FaceMeshMethod, and NNMethod – which compute facial/eye landmarks.
• Neural Network Models: Represent the deep learning component (depicted in models/eyenet.py, layers.py, and losses.py) that performs gaze estimation.
• Model Conversion & Deployment: ONNX conversion (onnx_convert.py) and saved models form a subcomponent highlighting production readiness.
c. User Interface & Output:
• Selection Window / Keyboard Interface: The module (SelectionWindow.py) responsible for mapping detected eye movements to keyboard actions.
• Feedback Loop: Visualization components (from public directory images) that show either real-time feedback on the screen or an example interface.
d. Utilities & Dependencies:
• Utility Scripts: Functions supporting gaze calculation and further signal processing found in src/util.
• Training/Inference Pipeline: As outlined in the README, include a component that addresses the training phase as well as the actual inference process.
e. External Dependencies & Environment:
• Environment Setup: The environment.yml signals dependency management (Conda environment).
Relationships and Interactions
• Data flow: Video feed from the camera flows into the pre-processing module then through one or more detection methods.
• Dependency: The main.py orchestrates the integration of all detection, model inference, and UI components.
• Feedback/Control: From the gaze detection output to the selection window that emulates a keyboard for user command.
• Conversion Flow: Show how models saved in various formats (.pth, .onnx, etc.) interact with the inference modules.
Architectural Patterns & Design Principles
• Modularity: Each detection method (custom, mesh, NN-based) stands as a separate module making it easier to swap or update techniques.
• Separation of Concerns: Clear distinction between input processing, model inference, and user interface layers.
• Extensibility: The design allows for the simple addition of new processing modules or model types due to the clearly structured directories.
• Pipeline Pattern: The README’s “Training/Inference Pipeline” image should be reflected by having sequential processing steps from data capture to keyboard output.