How gissemari/peruviansignlanguage works
Project type and purpose
• This is a data‐processing and machine‐learning pipeline for building a Peruvian Sign Language interpretation dataset and running experiments.
• It is not a monolithic web app but rather a sequence of stages: preprocessing raw videos and transcripts, segmenting and aligning them, extracting keypoints, generating datasets, then running analysis and classification experiments.
File structure
• Top‐level “Data” directory holds raw and intermediate datasets (videos, SRTs, keypoints, etc.).
• Directories “0.Analysis”, “1.SRTs”, “2.Video”, “3.Metadata”, “4.Keypoints”, “5.splitter” correspond to successive pipeline stages.
• “utils” contains shared helper modules; “shFiles” contains shell wrappers to launch dataset generation.
• Environment definitions under “environmentYMLfiles” specify conda setups for video and SRT processing.
• Scripts like “openPoseLooper.py”, “joinJsonData.py”, and “wandb_rnn_classifier.py” orchestrate OpenPose runs and model training/experiments.
• “experimentsOutput” captures final metrics/reports.
Extract insights from README
• The README outlines a multi‐stage preprocessing pipeline:
a. Crop raw YouTube videos to a fixed square around the signer.
b. Convert transcripts into one‐sentence‐per‐line.
c. Generate properly timed SRTs aligned to audio.
d. Segment the cropped video per sign or sign‐sentence using the SRT.
e. Optionally resize/crop per subject size.
f. Extract keypoints using MediaPipe/OpenPose.
g. Use the processed data for classification (ChaLearn, RNN).
• There are three separate conda environments: one for SRT, one for video, one for Chalearn.
• Raw Data Storage:
• Videos (raw .mp4)
• Transcripts (raw text, unsegmented)
• Subtitle files (.srt, ELAN annotations)
• Preprocessing Module:
• Video cropping scripts (cropInterpreter.py, resizeVideoByFrame.py)
• Transcript‐to‐line conversion (splitTranscriptToLine.py)
• SRT generation (convertToAudioSRT.py, convert_subtitle_sentence.py)
• Segmentation Module:
• Segment video by SRT to produce per‐sign and per‐sentence clips
• Keypoint Extraction Module:
• MediaPipe and OpenPose wrappers (generateMediapipeKeypoints.py, openPoseLooper.py)
• Dataset Assembly Module:
• Data selection, splitting, H5 splitters (split_H5.py)
• Analysis & Modeling Module:
• Analysis scripts (distanceByClassKPL.py, f1FirstExperiments.py)
• Classification/Training scripts (wandb_rnn_classifier.py)
• Supporting Utilities:
• Shared Python utilities (utils/*.py)
• Environment configs (environmentYMLfiles/*.yml)
• Outputs:
• Pickle/JSON keypoint files
• HDF5 or .data datasets
• Experiment summaries (experimentsOutput/*)
• Raw Data → Preprocessing → Segmentation → Keypoint Extraction → Dataset Assembly → Modeling → Experiment Outputs
• Shell scripts (shFiles/*.sh) kick off the full pipeline or individual stages.
c. Architectural patterns and design principles
• Linear pipeline / workflow (DAG) of data transformations.
• Clear stage separation: each top‐level directory is a self‐contained processing stage.
• Reusable utilities centralized under “utils”.
• Environment isolation via separate conda YMLs.
d. Key technologies and frameworks
• Python 3.x
• MediaPipe (holistic keypoints)
• OpenPose (via openPoseLooper.py)
• HDF5 and CSV for data storage
• WandB for experiment tracking
• Conda for environment management