How zhec/realtime_multi-person_pose_estimation works
Main Components
• Model Repository: The “model” folder holds pre-trained models (separated into COCO and MPI formats) with associated prototxt files. This is a key piece used for inference.
• Testing/Inference Components: The “testing” folder is organized into multiple subparts:
– Matlab-based testing (e.g., demo.m, various .m files, and configuration files in config.m) used for COCO evaluation.
– Python-based testing (with a Jupyter demo notebook in testing/python) where interactive exploration and demos can be executed.
– Supporting utilities, including JSON parser libraries (jsonlab) and visualization utilities (like export_fig).
• Training Pipeline: The “training” folder includes scripts and configuration files for preparing training data such as annotation processing (getANNO.m), mask generation (genCOCOMask.m), LMDB generation (genLMDB.py), and scripts that help set up prototxt configurations for training (setLayers.py). It also shows an example training workflow and loss curve generation through example_loss.
• Data Acquisition and Preprocessing:
– Start with the data pipeline (triggered by getData.sh) that downloads and extracts COCO data (images, annotations, and official toolbox data).
– Next, show the annotation conversion (getANNO.m), mask generation (genCOCOMask.m), and JSON creation (genJSON) as preprocessing steps that prepare the input for training.
• LMDB Generation and Network Configuration:
– Illustrate the use of genLMDB.py to produce a database that is then used by modified Caffe for training.
– The setLayers.py script creates necessary configuration (prototxt) files that define the network architecture.
• Training Component:
– Highlight that training is executed on a multi-GPU setup using custom shell scripts (e.g., train_pose.sh). The training initiates with a pre-trained VGG-19 model for the first 10 layers.
• Inference/Testing Component:
– For real-time testing, the system branches into various modalities: Matlab-based testing (via demo.m and other .m files), Python-based demos (through the Jupyter notebook), and a C++ variant (pointing towards the OpenPose project).
• External Dependencies:
– Note dependencies such as Caffe (for training and evaluation), Matlab, Python (and IPython/Jupyter), and external repositories like OpenPose for the C++ implementation.
Architectural Patterns and Design Principles
• Modular Design: The separation of concerns among training, testing, and model storage makes the system modular. This means that each component is responsible for a distinct part of the pipeline.
• External Service/Tool Integration: Include external tools such as the reference to OpenPose for real-time C++ testing.