How michaelt919/kintsugi3dbuilder works
1. What this repo is
- Kintsugi 3D Builder is a desktop Java 11 application for turning flash-photography image sets and photogrammetry reconstructions into empirically derived material textures.
- It is not a backend service or web app; the architecture is centered on a JavaFX UI plus an OpenGL rendering and optimization engine.
2. Top-level runtime shape
- The app starts in kintsugi3d.builder.app.Kintsugi3DBuilder / javafx.core.MainApplication.
- At runtime, JavaFX owns the windows and workflow UI, while a separate rendering layer drives OpenGL canvases and shader execution.
- The build/package setup also includes OS-specific launch and installer assets for Windows, macOS, and distribution packaging.
3. UI and workflow layer
- The JavaFX layer is organized by experience/workflow: create project, light calibration, mask options, specular basis fit, specular texture fit, export model, system settings, and help/about.
- Controllers live under src/kintsugi3d/builder/javafx/controllers/..., with FXML in src/main/resources/fxml/....
- The “paged” controller classes suggest wizard-like multi-step flows, while the experience package coordinates modal and workflow screens.
4. Application state and model boundary
- State is split between project data, scene configuration, and user/system preferences.
- The state and core packages define the canonical models: project, camera, lighting, environment, object pose, tabs/cards, and general settings.
- There are synchronized/observable wrappers under javafx/multithread and javafx/internal to bridge JavaFX thread updates with rendering/background work.
5. Project and source-data ingestion
- Input comes from photogrammetry outputs and capture metadata, mainly through readers for Agisoft Metashape, RealityCapture CSV, and a custom VSET format.
- The io layer also handles project directories, view-set serialization, LOD generation, image export, and texture export.
- This is the boundary where raw source data becomes the application’s internal ViewSet/ProjectModel representation.
6. Rendering subsystem
- Rendering is a substantial subsystem under src/kintsugi3d/builder/rendering, backed by the custom OpenGL framework in src/kintsugi3d/gl.
- The gl package provides the low-level abstraction layer for contexts, framebuffers, textures, shaders, vertex buffers, windowing, and event handling.
- Higher-level rendering components compose scene elements such as camera widgets, lights, backplates, environment maps, split-screen views, and lit/standard scenes.
7. Shader-driven visual pipeline
- The shaders/ tree contains the actual GPU programs used for display, reconstruction, color appearance, specular fitting, scene rendering, and test/synthetic cases.
- Architecturally, the shaders are the core execution units for both visualization and several processing stages; much of the “heavy lifting” is GPU-based rather than CPU-only.
- The shader groups map closely to functional domains: scene rendering, render modes/material display, specular fit, and helper/common math.
8. Optimization and reconstruction pipeline
- The optimization and builder/fit packages implement the scientific/reconstruction side of the app.
- Main stages include specular decomposition, normal optimization, roughness optimization, diffuse/albedo/ORM finalization, and reconstruction/error evaluation.
- These stages use least-squares, non-negative optimization, SVD, clustering, and shader-based error calculation to derive material parameters from the captured images.
9. Resource and asset management
- The app maintains explicit resource layers for images, cached graphics resources, preview generation, and stream-based render resources.
- resources/project and resources/project/specular manage cacheable textures and material resources tied to a project.
- This boundary separates transient GPU resources from durable project data and exportable outputs.
10. Interaction and manipulation tools
- User interaction with the 3D scene is modularized in builder/tools.
- Tools cover orbit/pan/dolly, first-person navigation, object orientation, light adjustment, environment rotation, picking, and parameter toggles.
- These tools act as input-to-state adapters: mouse/keyboard events feed the scene models, which then drive rendering updates.
11. Persistence and preferences
- User preferences and system settings are serialized separately from project content.
- The preferences package manages directory settings and global user preferences, with Jackson-based serialization support.
- Project state itself has its own serialization path through view-set and specular-fit serializers, which is important for drawing a diagram showing durable vs. runtime-only state.
12. Export and companion-tool integration
- Outputs include standard textures, GLTF-related data, screenshots, resampled views, orbit animations, and model exports.
- The repo is designed to interoperate with external viewers, especially the companion Kintsugi 3D Viewer application, and with downstream tools such as Sketchfab-style pipelines.
- Packaging scripts and installer configs reflect that the Builder is meant to produce artifacts consumed by another app or by standard 3D ecosystems.
13. Build/distribution structure
- Maven is the primary build system (pom.xml), with assembly descriptors for distribution and platform-specific launchers/installers.
- Windows and macOS packaging are first-class concerns, including bundled runtimes, NSIS installer support, and macOS app resources.
- The repository also includes sample test assets and documentation PDFs, but those are ancillary to the runtime architecture.
14. Suggested architectural diagram boundaries
- A good diagram should show four main zones: JavaFX UI/workflows, project/state models, rendering/OpenGL+shader pipeline, and fit/optimization/export pipeline.
- The key data flow is: imported photogrammetry/capture data → internal project/view-set models → interactive visualization and calibration → optimization/reconstruction → exported textures/materials/models.
Main entry points: MainApplication.java FX core · ObservableProjectModel.java UI state · ProjectRenderingEngine.java Render engine · LightingEnvironmentModel.java Scene state
Components 21
Runtime
Application startup, UI workflow, and interactive scene state
- App bootstrap · Desktop entrypoint and window synchronization setup.
src/kintsugi3d/builder/app/Kintsugi3DBuilder.java - FX core · Main application, window orchestration, and project/session wiring.
src/kintsugi3d/builder/javafx/core/MainApplication.java - Experiences · Modal and wizard-like workflow flows for create-project, calibration, fit, export, settings, and about.
src/kintsugi3d/builder/javafx/experience/Experience.java - Paged UI · Reusable page/controller framework for multi-step dialogs and data-bound flows.
src/kintsugi3d/builder/javafx/controllers/paged/Page.java - Scene UI · Root scene, camera, lights, environment, and object settings panels.
src/kintsugi3d/builder/javafx/controllers/scene/RootSceneController.java - UI state · JavaFX-facing observable and synchronized models bridging UI, background work, and rendering.
src/kintsugi3d/builder/javafx/internal/ObservableProjectModel.java - Project model · Canonical project, view-set, camera, lighting, environment, and pose state.
src/kintsugi3d/builder/state/project/ProjectModel.java - Scene state · Interactive camera, light, environment, and object pose models used by the viewer.
src/kintsugi3d/builder/state/scene/LightingEnvironmentModel.java - Interaction tools · Orbit, pan, dolly, first-person, picking, light, and object manipulation tools.
src/kintsugi3d/builder/tools/ToolBox.java
Pipeline
Import, reconstruction, optimization, and export
- Project IO · Project loading, recent projects, and session-level file operations.
src/kintsugi3d/builder/javafx/core/ProjectIO.java - Import IO · Readers and writers for Metashape XML, RealityCapture CSV, and VSET project data.
src/kintsugi3d/builder/io/ViewSetReader.java - Export IO · Texture, model, screenshot, animation, resample, and glTF export paths.
src/kintsugi3d/builder/io/gltf/GLTFExporter.java - Fit pipeline · Specular decomposition, normal/roughness optimization, and final material reconstruction.
src/kintsugi3d/builder/fit/SpecularFitProcess.java - Optimization · Least-squares, non-negative optimization, clustering, SVD, and shader-based error calculation.
src/kintsugi3d/optimization/ShaderBasedOptimization.java
Graphics
Rendering engine, OpenGL layer, and shader programs
- Render engine · Project rendering engine and reconstruction/scene viewport coordination.
src/kintsugi3d/builder/rendering/ProjectRenderingEngine.java - Scene render · Base scene, lit scene, split-screen, light calibration, and subject rendering components.
src/kintsugi3d/builder/rendering/components/StandardScene.java - OpenGL layer · Contexts, framebuffers, textures, shaders, vertex buffers, windows, and event plumbing.
src/kintsugi3d/gl/core/Context.java - Shaders · GPU programs for scene rendering, color appearance, render modes, and specular fitting.
shaders/subject/subjectMain.glsl - Resources · Project graphics resources, previews, image caches, and stream-based render resources.
src/kintsugi3d/builder/resources/project/GraphicsResources.java
Platform
Preferences, packaging, and launch assets
- Preferences · Global preferences, directory settings, and serialization for persistent user configuration.
src/kintsugi3d/builder/preferences/GlobalUserPreferencesManager.java - Packaging · Maven build, assembly descriptors, platform launchers, and installer resources.
pom.xml
Connections 21
- App bootstrap → FX core: starts · Bootstrap hands off to the JavaFX application and window lifecycle.
- FX core → Experiences: opens workflows · Main UI launches modal and wizard-like experiences.
- FX core → Paged UI: uses pages · Wizard dialogs are built from reusable paged controllers.
- FX core → Scene UI: hosts scene UI · Main window embeds camera, light, environment, and object panels.
- FX core → UI state: binds state · JavaFX models mirror application and project state across threads.
- UI state → Project model: wraps · Observable project state is a UI-facing view of canonical project data.
- UI state → Scene state: wraps · Observable scene models bridge interaction controls and rendering.
- Project IO → Import IO: loads · Project/session IO relies on source-data readers and writers.
- Import IO → Project model: builds · Imported capture data becomes internal project and view-set models.
- Fit pipeline → Optimization: drives · Reconstruction stages use the solver and error-evaluation core.
- Fit pipeline → Resources: uses caches · Optimization stages consume cached project graphics and specular resources.
- Export IO → Resources: exports from · Exporters read from project graphics/material resources.
- Render engine → Scene render: composes · The engine assembles the scene-specific render components.
- Render engine → OpenGL layer: runs on · Rendering is backed by the custom OpenGL abstraction layer.
- Scene render → Shaders: executes · Scene components are shader-driven GPU render passes.
- OpenGL layer → Shaders: loads · Programs and shader objects are built from repository shader sources.
- Interaction tools → Scene state: mutates · Interaction tools update camera, object, and lighting state.
- Scene state → Render engine: feeds · Manipulated scene state is rendered in the viewport and calibration views.
- Preferences → FX core: configures · User preferences affect application startup and runtime settings.
- Packaging → App bootstrap: bundles · Build and installer assets package the desktop entrypoint.
- Packaging → Preferences: ships · Distribution includes platform-specific configuration and persisted settings support.