siddhesh-04/robotic-arm-with-gesture-control

1. Project type and top-level split

This repository is a Webots simulation project for a UR5e robotic arm controlled by a separate Python vision app. The architecture has two runtime halves:

- a gesture-recognition client running outside Webots

- a Webots robot controller running inside the simulator

2. Primary control loop

The main data flow is:

webcam video → hand tracking / gesture inference → socket messages → Webots controller → robot motion

This is the core boundary in the system: perception is kept outside the simulator, while actuation stays inside Webots.

3. Gesture subsystem

controllers/ur5e_gesture_controller/gesture_ur5e.py is the standalone gesture application.

It uses:

- OpenCV for webcam capture and image processing

- MediaPipe for hand landmark detection / gesture recognition

- NumPy for numeric handling

Its job is to translate live hand gestures into control commands, then send them over the network socket to the robot controller.

4. Robot control subsystem

controllers/ur5e_gesture_controller/ur5e_gesture_controller.py is the Webots controller attached to the UR5e robot.

It runs in the simulator’s controller process and is responsible for:

- receiving gesture commands from the external app

- converting those commands into robot motion

- applying smooth velocity control rather than abrupt position jumps

This keeps the simulated arm motion stable and responsive.

5. Inter-process communication boundary

The two Python programs communicate via sockets. That makes the gesture app and the robot controller independent processes, which is important architecturally:

- the vision side can fail or restart without directly crashing Webots

- the controller side can remain focused on robot timing and actuation

- the protocol between them is the key integration contract

6. Simulation/world layer

The worlds/Armcontrol.wbt file is the Webots world definition for the simulation environment.

It contains the robot scene, controllers wiring, and likely the scene setup used for testing the arm.

worlds/.Armcontrol.wbproj is the Webots project metadata file that ties the world into the IDE/project structure.

7. Robot and model definitions

The protos/UR5e.proto file defines the UR5e robot model used in Webots.

protos/MyProto.proto is an additional custom prototype, likely for auxiliary scene elements or reusable components.

These proto files are the structural model layer: they define the robot and any reusable simulated objects used by the world.

8. Assets and presentation

assets/demo.gif is documentation/supporting media rather than runtime architecture.

It confirms the intended interaction pattern: live gesture control of the arm in simulation.

9. Operational dependency ordering

The README makes the startup dependency explicit:

1. launch Webots and start the world/controller

2. then start the external gesture script

That ordering matters because the controller is the socket endpoint the gesture app expects to connect to.

10. Technology stack boundaries

The architecture is intentionally split across:

- Python for both the gesture app and the Webots controller

- Webots for physics, robot simulation, and controller hosting

- MediaPipe/OpenCV for perception

- socket communication for command transport

No database, cloud service, or web framework is involved; this is a local real-time robotics control system.

11. What to draw in an architecture diagram

A clear diagram should show:

- a camera/webcam source

- the gesture recognition Python process

- a socket link

- the Webots controller process inside the UR5e robot

- the UR5e robot in the Webots world

- the proto/world files as simulation model inputs to Webots

The main boundary is between perception and actuation, with Webots serving as the simulation/runtime host for the robot side.