# siddhesh-04/robotic-arm-with-gesture-control architecture
> Architecture diagram of the GitHub repository siddhesh-04/robotic-arm-with-gesture-control, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-26). It is an AI-made overview: check details against the source.
- Interactive diagram: https://gitdiagram.com/siddhesh-04/robotic-arm-with-gesture-control
- Repository: https://github.com/siddhesh-04/robotic-arm-with-gesture-control
## Overview
1. **Project type and top-level split**
This repository is a **Webots simulation project** for a **UR5e robotic arm** controlled by a **separate Python vision app**. The architecture has two runtime halves:
- a **gesture-recognition client** running outside Webots
- a **Webots robot controller** running inside the simulator
2. **Primary control loop**
The main data flow is:
**webcam video → hand tracking / gesture inference → socket messages → Webots controller → robot motion**
This is the core boundary in the system: perception is kept outside the simulator, while actuation stays inside Webots.
3. **Gesture subsystem**
`controllers/ur5e_gesture_controller/gesture_ur5e.py` is the standalone gesture application.
It uses:
- **OpenCV** for webcam capture and image processing
- **MediaPipe** for hand landmark detection / gesture recognition
- **NumPy** for numeric handling
Its job is to translate live hand gestures into control commands, then send them over the network socket to the robot controller.
4. **Robot control subsystem**
`controllers/ur5e_gesture_controller/ur5e_gesture_controller.py` is the Webots controller attached to the UR5e robot.
It runs in the simulator’s controller process and is responsible for:
- receiving gesture commands from the external app
- converting those commands into robot motion
- applying **smooth velocity control** rather than abrupt position jumps
This keeps the simulated arm motion stable and responsive.
5. **Inter-process communication boundary**
The two Python programs communicate via **sockets**. That makes the gesture app and the robot controller independent processes, which is important architecturally:
- the vision side can fail or restart without directly crashing Webots
- the controller side can remain focused on robot timing and actuation
- the protocol between them is the key integration contract
6. **Simulation/world layer**
The `worlds/Armcontrol.wbt` file is the Webots world definition for the simulation environment.
It contains the robot scene, controllers wiring, and likely the scene setup used for testing the arm.
`worlds/.Armcontrol.wbproj` is the Webots project metadata file that ties the world into the IDE/project structure.
7. **Robot and model definitions**
The `protos/UR5e.proto` file defines the UR5e robot model used in Webots.
`protos/MyProto.proto` is an additional custom prototype, likely for auxiliary scene elements or reusable components.
These proto files are the structural model layer: they define the robot and any reusable simulated objects used by the world.
8. **Assets and presentation**
`assets/demo.gif` is documentation/supporting media rather than runtime architecture.
It confirms the intended interaction pattern: live gesture control of the arm in simulation.
9. **Operational dependency ordering**
The README makes the startup dependency explicit:
1. launch Webots and start the world/controller
2. then start the external gesture script
That ordering matters because the controller is the socket endpoint the gesture app expects to connect to.
10. **Technology stack boundaries**
The architecture is intentionally split across:
- **Python** for both the gesture app and the Webots controller
- **Webots** for physics, robot simulation, and controller hosting
- **MediaPipe/OpenCV** for perception
- **socket communication** for command transport
No database, cloud service, or web framework is involved; this is a local real-time robotics control system.
11. **What to draw in an architecture diagram**
A clear diagram should show:
- a **camera/webcam source**
- the **gesture recognition Python process**
- a **socket link**
- the **Webots controller process inside the UR5e robot**
- the **UR5e robot in the Webots world**
- the **proto/world files** as simulation model inputs to Webots
The main boundary is between perception and actuation, with Webots serving as the simulation/runtime host for the robot side.
Main entry points: [`controllers/ur5e_gesture_controller/gesture_ur5e.py`](https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/controllers/ur5e_gesture_controller/gesture_ur5e.py) (Gesture app), [`worlds/Armcontrol.wbt`](https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/worlds/Armcontrol.wbt) (Arm world), [`controllers/ur5e_gesture_controller/ur5e_gesture_controller.py`](https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/controllers/ur5e_gesture_controller/ur5e_gesture_controller.py) (Robot controller), [`protos/UR5e.proto`](https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/protos/UR5e.proto) (UR5e proto)
## Diagram (Mermaid)
```mermaid
flowchart TD
subgraph group_external["Gesture app"]
node_camera(("Webcam
input source"))
node_gesture_app["Gesture app
python vision client
[gesture_ur5e.py]"]
node_opencv["OpenCV
vision lib"]
node_mediapipe["MediaPipe
hand tracking"]
node_numpy["NumPy
numeric lib"]
end
subgraph group_webots["Webots simulation"]
node_robot_ctrl["Robot controller
webots controller"]
node_world["Arm world
webots world
[Armcontrol.wbt]"]
node_wbproj["Project file
webots metadata
[.Armcontrol.wbproj]"]
node_ur5e_proto["UR5e proto
robot model
[UR5e.proto]"]
node_custom_proto["Custom proto
scene proto
[MyProto.proto]"]
node_webots_runtime[("Webots runtime
simulation host")]
node_ur5e_robot{{"UR5e arm
sim robot"}}
end
node_socket_link["Socket link
ipc channel"]
node_camera -->|"video frames"| node_gesture_app
node_gesture_app -->|"uses"| node_opencv
node_gesture_app -->|"uses"| node_mediapipe
node_gesture_app -->|"uses"| node_numpy
node_gesture_app -->|"sends commands"| node_socket_link
node_socket_link -->|"receives commands"| node_robot_ctrl
node_webots_runtime -->|"loads"| node_world
node_world -.->|"project metadata"| node_wbproj
node_world -->|"instantiates"| node_ur5e_proto
node_world -.->|"reuses"| node_custom_proto
node_webots_runtime -->|"hosts"| node_robot_ctrl
node_robot_ctrl -->|"drives motion"| node_ur5e_robot
node_ur5e_proto -->|"defines model"| node_ur5e_robot
click node_gesture_app "https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/controllers/ur5e_gesture_controller/gesture_ur5e.py"
click node_robot_ctrl "https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/controllers/ur5e_gesture_controller/ur5e_gesture_controller.py"
click node_world "https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/worlds/Armcontrol.wbt"
click node_wbproj "https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/worlds/.Armcontrol.wbproj"
click node_ur5e_proto "https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/protos/UR5e.proto"
click node_custom_proto "https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/protos/MyProto.proto"
classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_camera,node_gesture_app,node_opencv,node_mediapipe,node_numpy toneBlue
class node_robot_ctrl,node_world,node_wbproj,node_ur5e_proto,node_custom_proto,node_webots_runtime,node_ur5e_robot toneAmber
class node_socket_link toneNeutral
```
## Components
### Gesture app
Standalone vision process that captures webcam input and sends gesture commands.
- **Webcam**: input source · Live video feed used for hand gesture recognition.
- [Gesture app](https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/controllers/ur5e_gesture_controller/gesture_ur5e.py): python vision client · Standalone Python process that performs hand tracking and converts gestures to control commands. · `controllers/ur5e_gesture_controller/gesture_ur5e.py`
- **OpenCV**: vision lib · Captures and processes camera frames for the gesture pipeline.
- **MediaPipe**: hand tracking · Detects hand landmarks and supports gesture inference.
- **NumPy**: numeric lib · Numeric helper library used by the gesture pipeline.
### Webots simulation
Simulator-side runtime and model definitions for the UR5e arm.
- [Robot controller](https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/controllers/ur5e_gesture_controller/ur5e_gesture_controller.py): webots controller · Webots controller process that receives commands and drives the UR5e arm with smooth velocity control. · `controllers/ur5e_gesture_controller/ur5e_gesture_controller.py`
- [Arm world](https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/worlds/Armcontrol.wbt): webots world · Simulation scene containing the robot and controller wiring. · `worlds/Armcontrol.wbt`
- [Project file](https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/worlds/.Armcontrol.wbproj): webots metadata · Webots project metadata that binds the world into the IDE/project structure. · `worlds/.Armcontrol.wbproj`
- [UR5e proto](https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/protos/UR5e.proto): robot model · Prototype definition for the UR5e robot used in the simulation. · `protos/UR5e.proto`
- [Custom proto](https://github.com/siddhesh-04/robotic-arm-with-gesture-control/blob/main/protos/MyProto.proto): scene proto · Additional reusable prototype for auxiliary simulated elements. · `protos/MyProto.proto`
- **Webots runtime**: simulation host · Simulator environment that hosts the world, robot model, and controller execution.
- **UR5e arm**: sim robot · Simulated robotic arm receiving motion commands from the controller.
### Other components
- **Socket link**: ipc channel · Network socket used to transport gesture commands between processes.
## Connections
- Webcam → Gesture app: video frames · Live webcam input feeds the gesture recognition app.
- Gesture app → OpenCV: uses · OpenCV handles frame capture and image processing.
- Gesture app → MediaPipe: uses · MediaPipe provides hand landmark detection and gesture inference.
- Gesture app → NumPy: uses · NumPy supports numeric handling in the gesture pipeline.
- Gesture app → Socket link: sends commands · Gesture app transmits control messages over a socket.
- Socket link → Robot controller: receives commands · The Webots controller consumes gesture commands from the socket.
- Webots runtime → Arm world: loads · Webots opens the world as the simulation scene.
- Arm world → Project file: project metadata · The Webots project file binds the world into the project structure.
- Arm world → UR5e proto: instantiates · The world uses the UR5e prototype to define the simulated robot.
- Arm world → Custom proto: reuses · The world can reference additional custom prototype definitions.
- Webots runtime → Robot controller: hosts · The controller runs inside the Webots simulation runtime.
- Robot controller → UR5e arm: drives motion · Controller converts commands into smooth arm movement.
- UR5e proto → UR5e arm: defines model · The prototype provides the robot structure used in simulation.