Hybrid Exterior Reconstruction
Learning-guided camera and depth prediction, classical bundle adjustment, and occlusion-aware multiview fusion for quantitatively evaluated outdoor reconstruction.
Project overview
Feed-forward 3D models can recover cameras and dense geometry directly from sparse imagery, but their outputs may contain scale drift, pose inconsistencies, and unsupported depth. Classical structure-from-motion and bundle adjustment offer stronger geometric constraints, but are slower and depend on reliable feature matching.
This project combines both strengths: VGGT produces a learned scene initialization, bundle adjustment refines the camera geometry, and an adaptive multiview filter removes dense points that are inconsistent with overlapping views while preserving high-confidence geometry that cannot be fairly judged because of occlusion or limited overlap.
Milestone 1 — Camera reconstruction and refinement
I compared three reconstruction paths on the same 12 views: VGGT feed-forward, VGGT initialized bundle adjustment, and classical COLMAP. All methods registered all selected images. Pose estimates were aligned to the calibrated ETH3D camera model with a least-squares Sim(3) transformation before evaluation.
| Method | Translation RMSE | Mean rotation error |
|---|---|---|
| VGGT feed-forward | 3.87 cm | 1.19° |
| VGGT + bundle adjustment | 1.27 cm | 0.45° |
| Classical COLMAP | 0.85 cm | 0.33° |
Milestone 2 — Dense hybrid reconstruction
VGGT depth predictions were mapped into the bundle-adjusted coordinate system and unprojected using the refined cameras. Each observation was then tested in other views. Compatible depth provided support; visible disagreement counted as conflict; and points behind observed surfaces were treated as occluded rather than incorrectly rejected.
| Metric | Raw dense | Adaptive fusion |
|---|---|---|
| Voxelized points | 135,300 | 67,388 |
| Median point-to-scan error | 10.26 cm | 6.96 cm |
| Mean point-to-scan error | 14.02 cm | 9.66 cm |
| Precision within 5 cm | 29.1% | 38.7% |
| Precision within 10 cm | 49.1% | 61.5% |
| F1 within 20 cm | 73.1% | 72.4% |
Engineering contributions
- VGGT inference and COLMAP-compatible export
- Classical COLMAP sparse/dense baseline
- Camera correspondence and Sim(3) alignment
- Pose and sparse-geometry evaluation
- Depth-scale correction and dense re-unprojection
- Cross-view support, conflict, and occlusion reasoning
- Confidence-aware adaptive filtering
- Metric voxel fusion and PLY export
- ETH3D scan sampling and shared-crop evaluation
- JSON/CSV result packaging, checksums, and visualizations
Evaluation scope and limitations
The study uses one outdoor scene and a 12-view subset. Dense evaluation is performed against a sampled ETH3D laser scan inside a common visible-region crop. The results are controlled diagnostic measurements for this project, not official ETH3D leaderboard scores.
- Reflective surfaces, vegetation, thin structures, and low-overlap regions remain difficult.
- Adaptive filtering improves precision by removing uncertain observations and can reduce completeness.
- The current result is geometric; semantic façade decomposition and CAD-style primitives are future work.
- Scaling to larger image collections requires additional runtime and memory evaluation.
Next extension — structured property models
The next stage will add open-vocabulary façade segmentation, multiview semantic fusion, finite wall and roof plane extraction, structural edge validation, approximate measurements, and exterior material/design visualization.
Summary
On a 12-view ETH3D Courtyard subset, I combined VGGT dense depth with bundle-adjusted cameras and occlusion-aware multiview consistency filtering. The hybrid pipeline reduced median point-to-scan error from 10.26 cm to 6.96 cm and improved precision within 5 cm from 29.1% to 38.7%, while maintaining nearly the same 20 cm F1 score.