How thomwolf/magic-sand works
1. What this repository is
- A desktop openFrameworks application for an augmented-reality sandbox.
- It sits between a Kinect depth sensor and a projector, turning measured sand height into projected colors and, optionally, game logic.
2. Top-level application shell
- src/main.cpp and src/ofApp.* are the entry point and app coordinator.
- The app is structured around the openFrameworks update/draw loop, with a normal UI window and a separate projector window.
3. Two-window rendering model
- The projector is treated as a distinct render target, not just another panel in the UI.
- main.cpp wires a projector-window draw callback; the projector-side rendering is isolated so calibration and projection can be controlled independently from the GUI window.
4. Sensor + calibration subsystem
- src/KinectProjector/* is the core infrastructure layer.
- KinectProjector owns Kinect acquisition, calibration, ROI selection, base-plane estimation, coordinate transforms, and state tracking.
- Supporting pieces include KinectGrabber, TemporalFrameFilter, and KinectProjectorCalibration.
5. Coordinate-system boundary
- This subsystem is the main architectural boundary in the repo: it converts between
- Kinect image coordinates,
- 3D world coordinates,
- projector coordinates.
- The rest of the app consumes these conversions instead of talking to Kinect calibration directly.
6. Surface visualization pipeline
- src/SandSurfaceRenderer/* turns depth/elevation data into projected imagery.
- ColorMap and the GLSL shaders under bin/data/shaders/* implement the height-to-color mapping and projector-space rendering.
- There are separate shader sets for GL2 and GL3, so rendering is adapted to the available graphics path.
7. Game and interaction layer
- src/Games/* contains gameplay built on top of the sandbox surface.
- BoidGameController, MapGameController, SandboxScoreTracker, ReferenceMapHandler, and vehicle implement different sandbox modes and scoring.
- These controllers depend on the calibrated sand surface and projected output, but they do not own sensor calibration.
8. Game-specific data
- The map game uses external reference assets in bin/data/mapGame/ReferenceData and stores results in bin/data/mapGame/Scores.
- Color lookup tables live in bin/data/colorMaps, and calibration/runtime settings are persisted in bin/data/settings/*.xml.
9. Runtime data flow
- Kinect frames enter the KinectProjector layer first.
- They are filtered, calibrated, and transformed into sand height/elevation data.
- SandSurfaceRenderer visualizes the surface in projector space.
- Game controllers read the same calibrated terrain data to decide object movement, scoring, and state transitions.
10. User-interface and setup boundary
- The app starts in a setup/calibration-oriented mode, then switches into live sandbox mode.
- UI controls are provided by openFrameworks addons such as ofxDatGui and ofxModal, while the actual projection output remains separated in the projector window.
11. Platform and build model
- This is a cross-platform native C++ project with build files for Visual Studio, Xcode, Qbs, and Make.
- The repo is meant to be built against openFrameworks 0.9.3 and its addons, rather than as a standalone library or service.
12. Embedded third-party code
- src/KinectProjector/libs/dlib vendors a substantial subset of dlib, indicating the project relies on local numeric/geometry utilities without an external package manager.
- Most other dependencies are openFrameworks addons rather than separate services or networked components.
Main entry points: KinectProjector.h KinectProjector · SandSurfaceRenderer.h SandSurfaceRenderer · ofApp.h ofApp · BoidGameController.h Boid Game
Components 20
App Shell
openFrameworks entry point, window orchestration, and runtime mode switching.
Sensor Boundary
Kinect acquisition, calibration, filtering, and coordinate transforms.
- KinectProjector · Owns acquisition, calibration, transforms, ROI, and state tracking.
src/KinectProjector/KinectProjector.h - Grabber · Wraps Kinect frame acquisition.
src/KinectProjector/KinectGrabber.h - Frame Filter · Smooths incoming depth frames before downstream processing.
src/KinectProjector/TemporalFrameFilter.h - Calibration · Stores calibration data and coordinate transforms between Kinect, world, and projector spaces.
src/KinectProjector/KinectProjectorCalibration.h - dlib vendored · Local numeric and geometry utilities bundled under KinectProjector.
src/KinectProjector/libs/dlib
Surface Rendering
Height-to-color mapping and projector-space rendering pipeline.
- SandSurfaceRenderer · Builds the projected surface visualization from elevation data.
src/SandSurfaceRenderer/SandSurfaceRenderer.h - ColorMap · Maps height/elevation values to display colors.
src/SandSurfaceRenderer/ColorMap.h - GL2 shaders · Projector-space rendering path for the GL2 pipeline.
bin/data/shaders/shadersGL2 - GL3 shaders · Projector-space rendering path for the GL3 pipeline.
bin/data/shaders/shadersGL3
Games
Sandbox gameplay and scoring built on calibrated surface data.
- Boid Game · Sandbox gameplay controller for boid-style interaction.
src/Games/BoidGameController.h - Map Game · Map-based game controller using reference terrain data.
src/Games/MapGameController.h - Score Tracker · Tracks and persists sandbox game scores.
src/Games/SandboxScoreTracker.h - Reference Maps · Loads map references and matching settings for the map game.
src/Games/ReferenceMapHandler.h - Vehicle · Movement/object logic used by sandbox game modes.
src/Games/vehicle.h
Runtime Assets
Persistent settings, shaders, maps, and saved scores used at runtime.
- Settings · Persisted calibration and renderer settings.
bin/data/settings - Map Assets · Reference data and stored scores for the map game.
bin/data/mapGame - Shader Assets · GL2 and GL3 shader sources used by the surface renderer.
bin/data/shaders
Build Targets
Platform-specific project and build files for native C++ builds.
- Build Files · Native build definitions for Visual Studio, Xcode, Qbs, and Make.
Magic-Sand.sln
Connections 25
- main.cpp → ofApp: launches · Entry point constructs the application coordinator.
- main.cpp → SandSurfaceRenderer: draw callback · Main wires the projector-side render path.
- ofApp → KinectProjector: owns · The app drives the sensor/calibration boundary.
- ofApp → SandSurfaceRenderer: updates · The app feeds calibrated surface state into rendering.
- ofApp → Boid Game: switches mode · App can activate sandbox game logic.
- ofApp → Map Game: switches mode · App can activate map-game logic.
- KinectProjector → Grabber: acquires · Coordinates Kinect frame capture.
- KinectProjector → Frame Filter: filters · Applies temporal smoothing to sensor data.
- KinectProjector → Calibration: transforms · Maps between Kinect, world, and projector coordinates.
- KinectProjector → dlib vendored: uses · Depends on bundled geometry and matrix utilities.
- KinectProjector → SandSurfaceRenderer: provides terrain · Supplies calibrated elevation/height data to rendering.
- KinectProjector → Boid Game: provides terrain · Game logic reads the same calibrated surface data.
- KinectProjector → Map Game: provides terrain · Map game consumes calibrated surface data.
- SandSurfaceRenderer → ColorMap: maps colors · Converts height values to projected colors.
- SandSurfaceRenderer → GL2 shaders: uses · GL2 rendering path.
- SandSurfaceRenderer → GL3 shaders: uses · GL3 rendering path.
- SandSurfaceRenderer → Settings: loads config · Renderer settings are persisted in XML.
- Calibration → Settings: persists · Calibration data is stored in runtime settings.
- Boid Game → Vehicle: updates entities · Boid gameplay uses vehicle/object motion logic.
- Map Game → Reference Maps: loads references · Map game consumes reference-map assets and settings.
- Score Tracker → Map Assets: stores scores · Score data is saved under map-game runtime assets.
- Reference Maps → Map Assets: reads assets · Reference map handler loads external map data.
- SandSurfaceRenderer → Shader Assets: loads assets · Renderer uses shader sources from data files.
- ofApp → Settings: loads ui state · App starts in setup/calibration mode using persisted settings.
- ofApp → Build Files: builds with · Repository is packaged as native desktop project files.