How junyi42/loger works
1. What this repo is
- LoGeR is a PyTorch-based reimplementation of a long-video geometric reconstruction system.
- The central architectural idea is chunked video processing with hybrid memory: frames are handled in segments, and state is carried across chunks to preserve long-range geometric consistency.
2. Core model boundary
- The main model code lives in loger/models/.
- pi3.py, pi3x.py, and ttt.py are the top-level model variants/adaptations.
- This is the boundary where raw image/video inputs become geometry outputs such as depth, pose, or reconstruction features.
3. Backbone and feature extraction
- loger/models/dinov2/ vendors a DINOv2-style vision transformer backbone.
- It includes the transformer blocks, patch embedding, attention, MLP, heads, and config/utility code needed to run the backbone inside this repo.
- This backbone is the feature encoder feeding the downstream geometric heads.
4. Task-specific heads and geometric prediction
- loger/models/layers/ contains specialized output modules:
- camera_head.py for camera/pose-related prediction,
- conv_head.py and transformer_head.py for alternative decoding paths,
- pos_embed.py, attention.py, and block.py for transformer internals,
- prope.py for additional positional/propagation logic.
- These modules form the bridge from visual features to geometry-aware outputs.
5. Geometry and rotation utilities
- loger/utils/geometry.py, rotation.py, and basic.py provide the math used across model and evaluation code.
- These utilities are the shared layer for coordinate transforms, pose handling, and geometric conversions.
6. Checkpoint and configuration boundary
- Pretrained weights are stored outside the model code under ckpts/LoGeR/ and ckpts/LoGeR_star/.
- Each checkpoint folder has an original_config.yaml, indicating that runtime behavior depends on paired model weights plus configuration.
- The repo expects latest.pt in these directories, so loading is a distinct runtime boundary from code.
7. Demo/visualization path
- demo_run.sh and demo_viser.py are the interactive demo entry points.
- loger/utils/viser_utils.py and visual_util.py suggest a visualization pipeline built around Viser for inspecting outputs.
- The demo likely connects sample data from data/examples/office to the model and renders reconstruction results interactively.
8. Evaluation as a separate subsystem
- The eval/ tree is not just test code; it is a full benchmark harness with multiple task-specific pipelines.
- It contains its own launch scripts, metadata, dataset preprocessing, metrics, and adapters.
- This means the repo has a clear split between core inference code and evaluation orchestration.
9. Video depth and relative pose evaluation
- eval/video_depth/ handles depth estimation benchmarks, with dataset-specific run scripts for Bonn, KITTI, and Sintel.
- eval/relpose/ handles relative pose evaluation, including sequence reuse logic and EVO-based scoring utilities.
- These subsystems consume model predictions and compare them against benchmark ground truth with task-specific metrics.
10. Multiview reconstruction and long-benchmark tooling
- eval/mv_recon/ is a self-contained multiview reconstruction evaluation stack with its own data pipeline, criterion, metric utilities, and device/parallel helpers.
- eval/long_eval_script/ contains C++ benchmark code for KITTI and VBR, indicating some evaluations rely on compiled external benchmark executables.
- eval/datasets_preprocess/ standardizes datasets such as TUM, ScanNet, and Bonn before evaluation.
11. External model and benchmark integration
- The README notes the code is based on Pi3 and LaCT, and evaluation borrows from TTT3R, VBR, and Viser.
- eval/pi3_adapter.py is a compatibility layer that likely bridges LoGeR outputs to legacy Pi3-style evaluation or interfaces.
- Practically, this repo sits at the intersection of a custom long-context model and several established geometry benchmarks.
12. Overall data flow
- Input videos/images enter through demo or eval entry points.
- Data is chunked, encoded by the DINOv2 backbone, processed by LoGeR’s memory-aware model, and decoded by geometry heads.
- Outputs then branch into either visualization/demo rendering or one of the evaluation pipelines, where dataset-specific metrics and benchmark scripts consume them.
Main entry points: pi3.py PI3 · demo_viser.py Demo run · layers Geom heads · vision_transformer.py DINOv2
Components 20
Core model
Long-video reconstruction model, backbone, and geometry heads.
- PI3 · Main long-video reconstruction model variant.
loger/models/pi3.py - PI3X · Alternative model adaptation for long-context processing.
loger/models/pi3x.py - TTT · Third top-level model variant/adaptation.
loger/models/ttt.py - DINOv2 · Vendored transformer backbone for visual feature extraction.
loger/models/dinov2/models/vision_transformer.py - ViT layers · Attention, MLP, patch embedding, and transformer blocks for the backbone.
loger/models/dinov2/layers - Geom heads · Camera, convolutional, and transformer heads that map features to geometry outputs.
loger/models/layers - Model attention · Shared transformer attention, blocks, and positional modules used by the decoder side.
loger/models/layers/attention.py
Evaluation
Benchmark harnesses and task-specific evaluation pipelines.
- MV recon · Self-contained multiview reconstruction benchmark pipeline.
eval/mv_recon - Depth eval · Video depth evaluation across KITTI, Bonn, and Sintel.
eval/video_depth - Rel pose · Relative pose evaluation with sequence reuse and EVO scoring.
eval/relpose - Preprocess · Dataset standardization scripts for long-form evaluation inputs.
eval/datasets_preprocess - Long bench · Compiled benchmark executables for KITTI and VBR evaluation.
eval/long_eval_script - Pi3 adapter · Bridges LoGeR outputs to Pi3-style evaluation or interfaces.
eval/pi3_adapter.py
Demo & viz
Interactive demo entry points and visualization support.
- Viz utils · Rendering and Viser integration utilities for interactive inspection.
loger/utils/viser_utils.py - Demo run · Interactive demo script that drives inference and visualization.
demo_viser.py - Demo script · Shell entry point for running the demo workflow.
demo_run.sh
Assets
Pretrained checkpoints and example inputs.
- Checkpoints · Pretrained weights paired with original configuration files.
ckpts/LoGeR - Example data · Sample office scene used by the demo pipeline.
data/examples/office
Other components
- Geometry utils · Coordinate transforms, rotation handling, and shared geometric math.
loger/utils/geometry.py - Basic utils · General-purpose helper routines used across model and evaluation code.
loger/utils/basic.py
Connections 23
- Demo script → Demo run: launches · Shell script starts the interactive demo.
- Demo run → Example data: loads · Demo consumes sample inputs from the example scene.
- Demo run → Checkpoints: loads · Demo depends on pretrained weights and config.
- Demo run → PI3: runs · Interactive demo invokes the core model variant.
- PI3 → DINOv2: encodes · Model uses the backbone for visual feature extraction.
- PI3 → Geom heads: decodes · Model uses geometry heads to produce outputs.
- PI3 → Geometry utils: uses · Model relies on shared geometry math.
- PI3X → DINOv2: encodes · Variant reuses the same feature backbone.
- PI3X → Geom heads: decodes · Variant reuses geometry-aware heads.
- TTT → DINOv2: encodes · Variant reuses the same feature backbone.
- TTT → Geom heads: decodes · Variant reuses geometry-aware heads.
- DINOv2 → ViT layers: built from · Vision transformer is assembled from internal backbone layers.
- Geom heads → Model attention: includes · Decoder-side modules include attention and transformer internals.
- Geom heads → Geometry utils: uses · Heads use geometric conversions and pose math.
- MV recon → PI3: consumes · Evaluation stack runs model predictions through multiview metrics.
- Depth eval → PI3: consumes · Depth benchmarks evaluate model outputs on video datasets.
- Rel pose → PI3: consumes · Relative pose benchmarks evaluate model predictions.
- Preprocess → Depth eval: preps for · Dataset preprocessing feeds depth benchmark pipelines.
- Preprocess → Rel pose: preps for · Dataset preprocessing feeds pose benchmark pipelines.
- Long bench → MV recon: supports · Native benchmark executables support long-form evaluation.
- Pi3 adapter → Rel pose: adapts · Compatibility layer helps bridge to legacy evaluation interfaces.
- Viz utils → Demo run: renders · Visualization utilities support the interactive demo.
- Checkpoints → PI3: configures · Checkpoint folders pair weights with the expected runtime config.