How keyan2017/anphsp works
Main Components
• Mobile Application (app):
– This is an Android application. It has multiple layers:
• UI & Activity Layer: Contains activities (e.g., CameraActivity, Mainactivity, NavigationActivity, DetailActivity) and UI fragments (home, gallery, detail, etc.).
• Native & C++ Layer: Manages image processing and integration with native libraries. Look at the “app/src/main/cpp” directory, which includes native-lib.cpp, CMakeLists.txt, and the onnxruntime headers.
• Dependencies: OpenCV libraries are integrated (see the “app/opencv” directories and gradle configuration). These are used for image pre-processing tasks.
• Prediction Module (prediction):
– This module holds the code for hemoglobin concentration prediction.
– It is written in Python and includes onnx model files, various network definitions (e.g., C3AE.py, DAANet.py, DeltaAdaINNet.py, ResNets.py), configuration scripts (config.py), and supporting datasets.
– It typically handles the inference process based on pre-trained models and may be used locally or as a backend service interfacing with the mobile app.
• Segmentation Module (segment):
– This part is dedicated to eye image segmentation.
– Code elements indicate a UNet-based segmentation approach (EGE-UNet-seg). It has Python scripts for training, testing, and inference (e.g., train.py, test.py, utils.py) along with configuration files and dataset management (organized into training, validation, and test subdirectories).
– The segmentation output likely serves as pre-processing for the prediction module by isolating the region of interest in the eye images.
Component Interactions and Data Flow
• Once an image is captured, the app may invoke native C++ code (via JNI) for pre-processing. This might include:
– Using OpenCV tools (as evidenced by the opencv directory) for basic image filtering and manipulation.
– Loading and possibly running onnx models locally (if on-device inference is supported) or sending the pre-processed images to the prediction module.
• The segmentation module processes the image to extract the eyelid or other critical regions required for accurate hemoglobin prediction.
• The output (segmented image) is passed to the prediction module, which applies its trained deep learning networks to estimate hemoglobin levels.
• The result flows back to the mobile app for display to the user (or further diagnostic use).
Architectural Patterns and Design Choices
• Separation of Concerns: Each major function (UI, segmentation, prediction) is isolated into its own module or folder.
• Multi-language Integration: The Android app is developed in Java with native C++ integration for performance-critical tasks (via JNI). The prediction and segmentation components are implemented in Python.
• Reuse of Models & Assets: ONNX model files are shared between various modules (located in assets and prediction directories), demonstrating a consistent model inference approach.
• Use of External Libraries: The integration of OpenCV is critical for image processing tasks in the app, and the usage of onnxruntime and native C++ libraries is central to performing efficient inference.