# gissemari/peruviansignlanguage architecture

> Architecture diagram of the GitHub repository gissemari/peruviansignlanguage, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gissemari/peruviansignlanguage
- Repository: https://github.com/gissemari/peruviansignlanguage

## Overview

##### Project type and purpose
- This is a data‐processing and machine‐learning pipeline for building a Peruvian Sign Language interpretation dataset and running experiments.
- It is not a monolithic web app but rather a sequence of stages: preprocessing raw videos and transcripts, segmenting and aligning them, extracting keypoints, generating datasets, then running analysis and classification experiments.

##### File structure
- Top‐level “Data” directory holds raw and intermediate datasets (videos, SRTs, keypoints, etc.).
- Directories “0.Analysis”, “1.SRTs”, “2.Video”, “3.Metadata”, “4.Keypoints”, “5.splitter” correspond to successive pipeline stages.
- “utils” contains shared helper modules; “shFiles” contains shell wrappers to launch dataset generation.
- Environment definitions under “environmentYMLfiles” specify conda setups for video and SRT processing.
- Scripts like “openPoseLooper.py”, “joinJsonData.py”, and “wandb_rnn_classifier.py” orchestrate OpenPose runs and model training/experiments.
- “experimentsOutput” captures final metrics/reports.

##### Extract insights from README
- The README outlines a multi‐stage preprocessing pipeline:
a. Crop raw YouTube videos to a fixed square around the signer.
b. Convert transcripts into one‐sentence‐per‐line.
c. Generate properly timed SRTs aligned to audio.
d. Segment the cropped video per sign or sign‐sentence using the SRT.
e. Optionally resize/crop per subject size.
f. Extract keypoints using MediaPipe/OpenPose.
g. Use the processed data for classification (ChaLearn, RNN).
- There are three separate conda environments: one for SRT, one for video, one for Chalearn.

- Raw Data Storage:
- Videos (raw .mp4)
- Transcripts (raw text, unsegmented)
- Subtitle files (.srt, ELAN annotations)
- Preprocessing Module:
- Video cropping scripts (cropInterpreter.py, resizeVideoByFrame.py)
- Transcript‐to‐line conversion (splitTranscriptToLine.py)
- SRT generation (convertToAudioSRT.py, convert_subtitle_sentence.py)
- Segmentation Module:
- Segment video by SRT to produce per‐sign and per‐sentence clips
- Keypoint Extraction Module:
- MediaPipe and OpenPose wrappers (generateMediapipeKeypoints.py, openPoseLooper.py)
- Dataset Assembly Module:
- Data selection, splitting, H5 splitters (split_H5.py)
- Analysis & Modeling Module:
- Analysis scripts (distanceByClassKPL.py, f1FirstExperiments.py)
- Classification/Training scripts (wandb_rnn_classifier.py)
- Supporting Utilities:
- Shared Python utilities (utils/*.py)
- Environment configs (environmentYMLfiles/*.yml)
- Outputs:
- Pickle/JSON keypoint files
- HDF5 or .data datasets
- Experiment summaries (experimentsOutput/*)

- Raw Data → Preprocessing → Segmentation → Keypoint Extraction → Dataset Assembly → Modeling → Experiment Outputs
- Shell scripts (shFiles/*.sh) kick off the full pipeline or individual stages.

c. Architectural patterns and design principles
- Linear pipeline / workflow (DAG) of data transformations.
- Clear stage separation: each top‐level directory is a self‐contained processing stage.
- Reusable utilities centralized under “utils”.
- Environment isolation via separate conda YMLs.

d. Key technologies and frameworks
- Python 3.x
- MediaPipe (holistic keypoints)
- OpenPose (via openPoseLooper.py)
- HDF5 and CSV for data storage
- WandB for experiment tracking
- Conda for environment management

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Stores
    RawVideos(("Raw Videos .mp4")):::data
    RawTranscripts(("Raw Transcripts .txt")):::data
    RawSRTs(("Raw Subtitles .srt")):::data

    %% Preprocessing Stage
    subgraph "Stage 1: Preprocessing" 
        direction TB
        CropInterpreter["cropInterpreter.py"]:::process
        CropBySRT["cropInterpreterBySRT.py"]:::process
        ResizeVideo["resizeVideoByFrame.py"]:::process
        VideoToFrames["videoToFrames.py"]:::process
        SplitTranscript["splitTranscriptToLine.py"]:::process
        ConvertAudioSRT["convertToAudioSRT.py"]:::process
        ExtractEAF["extractingSRTeaf.py"]:::process
    end

    %% Segmentation Stage
    subgraph "Stage 2: Segmentation" 
        direction TB
        VideoSegmenter["split_H5.py"]:::process
        OldVideoSegmenter["old_split_H5.py"]:::process
        ReadH5["read_H5.py"]:::process
    end

    %% Keypoint Extraction Stage
    subgraph "Stage 3: Keypoint Extraction"
        direction TB
        Mediapipe["generateMediapipeKeypoints.py"]:::process
        ConvertVideoKP["ConvertVideoToKeypoint.py"]:::process
        OpenPoseLooper["openPoseLooper.py"]:::process
        JoinJson["joinJsonData.py"]:::process
    end

    %% Dataset Assembly
    subgraph "Stage 4: Dataset Assembly"
        direction TB
        DatasetSplitter["split_H5.py"]:::process
    end

    %% Analysis & Modeling Stage
    subgraph "Stage 5: Analysis & Modeling"
        direction TB
        DistanceByClass["distanceByClassKPL.py"]:::model
        F1Experiments["f1FirstExperiments.py"]:::model
        AnalysisMain["main.py"]:::model
        RNNClassifier["wandb_rnn_classifier.py"]:::model
    end

    %% Utilities and Configs
    Utils["utils/*.py"]:::util
    EnvConfigs["environmentYMLfiles/*.yml"]:::config
    ShellScripts["shFiles/*.sh"]:::process

    %% Outputs
    ExperimentOutputs(("experimentsOutput/*")):::data

    %% Main Data Flow
    RawVideos --> CropInterpreter
    RawVideos --> CropBySRT
    CropInterpreter --> ResizeVideo --> VideoToFrames
    RawTranscripts --> SplitTranscript
    RawSRTs --> ConvertAudioSRT --> ExtractEAF

    VideoToFrames --> VideoSegmenter
    ExtractEAF --> VideoSegmenter
    VideoSegmenter --> OldVideoSegmenter
    OldVideoSegmenter --> ReadH5

    ReadH5 --> Mediapipe
    ReadH5 --> OpenPoseLooper
    OpenPoseLooper --> JoinJson
    Mediapipe --> ConvertVideoKP
    JoinJson --> ConvertVideoKP
    ConvertVideoKP --> DatasetSplitter

    DatasetSplitter --> DistanceByClass
    DatasetSplitter --> F1Experiments
    DatasetSplitter --> AnalysisMain
    AnalysisMain --> RNNClassifier
    RNNClassifier --> ExperimentOutputs

    %% Auxiliary Dependencies
    Utils -.-> CropInterpreter
    Utils -.-> SplitTranscript
    Utils -.-> VideoSegmenter
    Utils -.-> Mediapipe
    Utils -.-> RNNClassifier
    EnvConfigs -.-> CropInterpreter
    EnvConfigs -.-> ConvertAudioSRT
    EnvConfigs -.-> Mediapipe
    EnvConfigs -.-> RNNClassifier
    ShellScripts -.-> CropInterpreter
    ShellScripts -.-> VideoSegmenter
    ShellScripts -.-> RNNClassifier

    %% Click Events
    click CropInterpreter "https://github.com/gissemari/peruviansignlanguage/blob/master/2.Video/cropInterpreter.py"
    click CropBySRT "https://github.com/gissemari/peruviansignlanguage/blob/master/2.Video/cropInterpreterBySRT.py"
    click ResizeVideo "https://github.com/gissemari/peruviansignlanguage/blob/master/2.Video/resizeVideoByFrame.py"
    click VideoToFrames "https://github.com/gissemari/peruviansignlanguage/blob/master/2.Video/videoToFrames.py"
    click SplitTranscript "https://github.com/gissemari/peruviansignlanguage/blob/master/PERLINE/splitTranscriptToLine.py"
    click ConvertAudioSRT "https://github.com/gissemari/peruviansignlanguage/blob/master/1.SRTs/convertToAudioSRT.py"
    click ExtractEAF "https://github.com/gissemari/peruviansignlanguage/blob/master/1.SRTs/extractingSRTeaf.py"
    click VideoSegmenter "https://github.com/gissemari/peruviansignlanguage/blob/master/5.splitter/split_H5.py"
    click OldVideoSegmenter "https://github.com/gissemari/peruviansignlanguage/blob/master/5.splitter/old_split_H5.py"
    click ReadH5 "https://github.com/gissemari/peruviansignlanguage/blob/master/5.splitter/read_H5.py"
    click Mediapipe "https://github.com/gissemari/peruviansignlanguage/blob/master/4.Keypoints/generateMediapipeKeypoints.py"
    click ConvertVideoKP "https://github.com/gissemari/peruviansignlanguage/blob/master/4.Keypoints/ConvertVideoToKeypoint.py"
    click OpenPoseLooper "https://github.com/gissemari/peruviansignlanguage/blob/master/openPoseLooper.py"
    click JoinJson "https://github.com/gissemari/peruviansignlanguage/blob/master/joinJsonData.py"
    click DistanceByClass "https://github.com/gissemari/peruviansignlanguage/blob/master/0.Analysis/distanceByClassKPL.py"
    click F1Experiments "https://github.com/gissemari/peruviansignlanguage/blob/master/0.Analysis/f1FirstExperiments.py"
    click AnalysisMain "https://github.com/gissemari/peruviansignlanguage/blob/master/0.Analysis/main.py"
    click RNNClassifier "https://github.com/gissemari/peruviansignlanguage/blob/master/wandb_rnn_classifier.py"
    click Utils "https://github.com/gissemari/peruviansignlanguage/tree/master/utils"
    click EnvConfigs "https://github.com/gissemari/peruviansignlanguage/tree/master/environmentYMLfiles"
    click ShellScripts "https://github.com/gissemari/peruviansignlanguage/tree/master/shFiles"
    click ExperimentOutputs "https://github.com/gissemari/peruviansignlanguage/tree/master/experimentsOutput"

    %% Styles
    classDef data fill:#cfe2f3,stroke:#036,stroke-width:1px
    classDef process fill:#d9ead3,stroke:#060,stroke-width:1px
    classDef model fill:#ead1dc,stroke:#a00,stroke-width:1px
    classDef util fill:#f2f2f2,stroke:#999,stroke-width:1px
    classDef config fill:#ffe599,stroke:#a65f00,stroke-width:1px
```
