Back

DUSt3R: Geometric 3D Vision Made Easy

Shuzhe WangVincent LeroyYohann CabonBoris ChidlovskiiJérôme Revaud

article2023CVPR1,826 citations

Presents DUSt3R, a method that performs dense 3D reconstruction and camera pose estimation directly from arbitrary uncalibrated images by formulating multi-view geometry as pointmap regression.

Listen

Dense 3D reconstruction from photographs is vital for robotics, autonomous navigation, mapping, and digital preservation, but traditional methods rely on fragile multi-stage pipelines. Conventional workflows sequentially execute feature matching, camera calibration, and multi-view triangulation. When initial calibration fails or when image sets have limited visual overlap, these complex pipelines frequently break down. The article sets out to evaluate and demonstrate a unified framework called DUSt3R, which directly reconstructs dense 3D geometry from uncalibrated and unposed images without requiring prior camera parameters.

The authors developed a data-driven transformer model that casts stereo reconstruction as the direct regression of dense 3D pointmaps, representing points in the first camera's coordinate frame alongside confidence scores. The model was trained on roughly 8.5 million image pairs across eight diverse synthetic and real-world datasets, using self-supervised vision transformer pretraining without enforcing rigid geometric formulas during inference. To reconstruct scenes involving more than two images, the framework introduces a global alignment method that optimizes geometry directly in 3D space rather than minimizing traditional 2D reprojection errors.

The evaluation yielded several key findings. First, DUSt3R set new performance benchmarks in map-free visual localization on the Map-free benchmark, achieving a median translation error of 0.98 meters compared to 1.23 to 2.93 meters for existing state-of-the-art baselines. Second, in multi-view relative pose estimation, it achieved relative rotation and translation accuracies of 96.2% and 86.8% on the CO3Dv2 benchmark, outperforming dedicated models like PoseDiffusion. Third, the single model demonstrated strong zero-shot performance in both monocular and multi-view depth estimation across indoor and outdoor benchmarks. Finally, multi-view 3D alignment converged in seconds on standard hardware, running significantly faster than traditional bundle adjustment pipelines that require several minutes.

These findings indicate that 3D reconstruction can be consolidated into an end-to-end learning framework, bypassing fragile multi-step engineering pipelines. Operating without camera intrinsics or manual calibration significantly lowers operational risk, software complexity, and compute overhead for computer vision deployments. In dense multi-view reconstruction benchmarks, DUSt3R achieved a competitive overall average point error of 1.7 millimeters without task-specific tuning, demonstrating immediate practical utility.

Organizations developing spatial computing, robotics, or mapping workflows should pilot DUSt3R as a streamlined alternative or front-end pre-processor to conventional structure-from-motion tools. For production integration, engineering teams should assess the trade-off between direct pointmap regression speed and sub-pixel triangulation accuracy. Future development should evaluate model performance on non-standard optics, dynamic objects, and memory constraints during very large-scale scene captures.

The findings are supported by comprehensive testing across established public datasets. However, because predictions are regressed up to an arbitrary scale factor without ground-truth sensors, downstream metric localization still requires a reference depth scale. Users should note that while the method is robust across diverse indoor and outdoor environments, specialized models trained exclusively on single domains with ground-truth camera poses may still achieve finer sub-millimeter surface precision.

Aalto UniversityNaver Labs Europe3d modellingbinocularsdecodersunstructured multi-view stereomonocular depth estimationscene reconstruction
arXiv: 2312.14132naver/dust3r
Cover for DUSt3R: Geometric 3D Vision Made Easy

Abstract

Multi-view stereo reconstruction (MVS) in the wild requires to first estimate the camera parameters e.g. intrinsic and extrinsic parameters. These are usually tedious and cumbersome to obtain, yet they are mandatory to triangulate corresponding pixels in 3D space, which is the core of all best performing MVS algorithms. In this work, we take an opposite stance and introduce DUSt3R, a radically novel paradigm for Dense and Unconstrained Stereo 3D Reconstruction of arbitrary image collections, i.e. operating without prior information about camera calibration nor viewpoint poses. We cast the pairwise reconstruction problem as a regression of pointmaps, relaxing the hard constraints of usual projective camera models. We show that this formulation smoothly unifies the monocular and binocular reconstruction cases. In the case where more than two images are provided, we further propose a simple yet effective global alignment strategy that expresses all pairwise pointmaps in a common reference frame. We base our network architecture on standard Transformer encoders and decoders, allowing us to leverage powerful pretrained models. Our formulation directly provides a 3D model of the scene as well as depth information, but interestingly, we can seamlessly recover from it, pixel matches, relative and absolute camera. Exhaustive experiments on all these tasks showcase that the proposed DUSt3R can unify various 3D vision tasks and set new SoTAs on monocular/multi-view depth estimation as well as relative pose estimation. In summary, DUSt3R makes many geometric 3D vision tasks easy.

Topics

3d modellingbinocularsdecodersdepth estimationencoder-decoderencodersimage collectionsmulti-view stereonetwork architecturepose estimation

Keywords

unstructured multi-view stereomonocular depth estimationscene reconstructionpixel correspondencescamera localizationview synthesisvision transformersoptical flow

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Method
  • 3.1 Overview
  • 3.2 Training Objective
  • 3.3 Downstream Applications
  • 3.4 Global Alignment
  • 4 Experiments with DUSt3R
  • 4.1 Visual Localization
  • 4.2 Multi-view Pose Estimation
  • 4.3 Monocular Depth
  • 4.4 Multi-view Depth
  • 4.5 3D Reconstruction
  • 4.6 Ablations
  • 5 Conclusion
  • Qualitative results
  • Extended Related Work
  • Multi-view Pose Estimation
  • Visual localization
  • Training details
  • .1 Training data
  • .2 Training hyperparameters
  • References

Knowls

  1. Knowl 1 — Pairwise Pointmap Regression Architecture in DUSt3R

    model/method

    DUSt3R formulates dense stereo 3D reconstruction as a direct regression problem from an unconstrained image pair without requiring prior camera intrinsics or poses. Given two RGB images I1∈RW1×H1×3I^1 \in \mathbb{R}^{W_1 \times H_1 \times 3} and I2∈RW2×H2×3I^2 \in \mathbb{R}^{W_2 \times H_2 \times 3}, a neural network ff directly predicts two dense 3D pointmaps X1,1∈RW1×H1×3X^{1,1} \in \mathbb{R}^{W_1 \times H_1 \times 3} and X2,1∈RW2×H2×3X^{2,1} \in \mathbb{R}^{W_2 \times H_2 \times 3} along with associated confidence maps C1,1∈RW1×H1C^{1,1} \in \mathbb{R}^{W_1 \times H_1} and C2,1∈RW2×H2C^{2,1} \in \mathbb{R}^{W_2 \times H_2}. Crucially, both pointmaps are regressed in the common 3D coordinate frame of camera 1 (I1I^1).

    The model architecture is structured as follows:

    1. Encoder: A shared Vision Transformer (ViT) processes images I1I^1 and I2I^2 in a Siamese manner to extract token sequences F1=Encoder(I1)F^1 = \text{Encoder}(I^1) and F2=Encoder(I2)F^2 = \text{Encoder}(I^2).
    2. Decoder: Two transformer decoder branches with BB blocks exchange information through cross-attention. At decoder block i∈{1,…,B}i \in \{1, \dots, B\}, branch representations Gi1G_i^1 and Gi2G_i^2 are updated by cross-attending to tokens from the opposing branch: Gi1=DecoderBlocki1(Gi−11,Gi−12),Gi2=DecoderBlocki2(Gi−12,Gi−11)G_i^1 = \text{DecoderBlock}_i^1(G_{i-1}^1, G_{i-1}^2), \quad G_i^2 = \text{DecoderBlock}_i^2(G_{i-1}^2, G_{i-1}^1) where initial tokens are G01=F1G_0^1 = F^1 and G02=F2G_0^2 = F^2.
    3. Regression Heads: Dense Prediction Transformer (DPT) heads process the decoder token representations to output the pointmaps and pixel-wise confidence maps: X1,1,C1,1=Head1(G01,…,GB1),X2,1,C2,1=Head2(G02,…,GB2)X^{1,1}, C^{1,1} = \text{Head}^1(G_0^1, \dots, G_B^1), \quad X^{2,1}, C^{2,1} = \text{Head}^2(G_0^2, \dots, G_B^2)

  2. Knowl 2 — Scale-Invariant Confidence-Weighted 3D Regression Loss

    equation

    DUSt3R is trained using a scale-normalized, confidence-weighted Euclidean distance regression loss directly in 3D coordinate space. For an image pair with valid ground-truth pixel coordinate sets D1,D2⊆{1,…,W}×{1,…,H}\mathcal{D}^1, \mathcal{D}^2 \subseteq \{1, \dots, W\} \times \{1, \dots, H\} and ground-truth pointmaps Xˉ1,1,Xˉ2,1\bar{X}^{1,1}, \bar{X}^{2,1}, the regression error for a pixel i∈Dvi \in \mathcal{D}^v in view v∈{1,2}v \in \{1, 2\} is: ℓregr(v,i)=∥1zXiv,1−1zˉXˉiv,1∥\ell_{\text{regr}}(v, i) = \left\Vert \frac{1}{z} X_i^{v,1} - \frac{1}{\bar{z}} \bar{X}_i^{v,1} \right\Vert

    Here, zz and zˉ\bar{z} normalize the scale ambiguity by computing the mean distance of all valid 3D points to the origin: z(X1,1,X2,1)=1∣D1∣+∣D2∣∑v∈{1,2}∑i∈Dv∥Xiv,1∥,zˉ(Xˉ1,1,Xˉ2,1)=1∣D1∣+∣D2∣∑v∈{1,2}∑i∈Dv∥Xˉiv,1∥z(X^{1,1}, X^{2,1}) = \frac{1}{|\mathcal{D}^1| + |\mathcal{D}^2|} \sum_{v \in \{1,2\}} \sum_{i \in \mathcal{D}^v} \left\Vert X_i^{v,1} \right\Vert, \quad \bar{z}(\bar{X}^{1,1}, \bar{X}^{2,1}) = \frac{1}{|\mathcal{D}^1| + |\mathcal{D}^2|} \sum_{v \in \{1,2\}} \sum_{i \in \mathcal{D}^v} \left\Vert \bar{X}_i^{v,1} \right\Vert

    The full training loss Lconf\mathcal{L}_{\text{conf}} incorporates learned per-pixel confidence scores Civ,1C_i^{v,1}: Lconf=∑v∈{1,2}∑i∈DvCiv,1ℓregr(v,i)−αlog⁡Civ,1\mathcal{L}_{\text{conf}} = \sum_{v \in \{1,2\}} \sum_{i \in \mathcal{D}^v} C_i^{v,1} \ell_{\text{regr}}(v, i) - \alpha \log C_i^{v,1} where Civ,1=1+exp⁡(civ,1)>1C_i^{v,1} = 1 + \exp(c_i^{v,1}) > 1 is parameterized via unconstrained network logits civ,1∈Rc_i^{v,1} \in \mathbb{R}, and α>0\alpha > 0 is a regularization hyperparameter preventing the trivial solution of setting confidence to zero.

  3. Knowl 3 — Global Pointmap Optimization for Multi-View 3D Alignment

    algorithm

    To reconstruct scenes from arbitrary image collections {I1,I2,…,IN}\{I^1, I^2, \dots, I^N\}, DUSt3R aligns all pairwise predicted pointmaps into a unified world coordinate frame by optimizing 3D spatial alignment rather than minimizing 2D reprojection errors.

    Input: Images {I1,…,IN}\{I^1, \dots, I^N\}, connectivity graph G=(V,E)\mathcal{G} = (\mathcal{V}, \mathcal{E}), pairwise model ff
    Output: Globally aligned pointmaps {χn∈RW×H×3}n=1N\{\chi^n \in \mathbb{R}^{W \times H \times 3}\}_{n=1}^N, pairwise poses {Pe}\{P_e\}, scale factors {σe}\{\sigma_e\}
    for each edge e=(n,m)∈Ee = (n, m) \in \mathcal{E} do
        (Xn,e,Xm,e,Cn,e,Cm,e)←f(In,Im)(X^{n,e}, X^{m,e}, C^{n,e}, C^{m,e}) \leftarrow f(I^n, I^m)
    end for
    Initialize pairwise scale factors σe←1\sigma_e \leftarrow 1 for all e∈Ee \in \mathcal{E}
    Initialize rigid transformations Pe=[Re∣te]∈R3×4P_e = [R_e \mid t_e] \in \mathbb{R}^{3 \times 4} using pairwise relative poses propagated along a maximum spanning tree of G\mathcal{G}
    Initialize global pointmaps χn\chi^n for n=1,…,Nn = 1, \dots, N
    repeat
        Compute loss: Lalign=∑e∈E∑v∈e∑i=1HWCiv,e∥χiv−σePeXiv,e∥\mathcal{L}_{\text{align}} = \sum_{e \in \mathcal{E}} \sum_{v \in e} \sum_{i=1}^{HW} C_i^{v,e} \|\chi_i^v - \sigma_e P_e X_i^{v,e}\|
        Update {χn}\{\chi^n\}, {Pe}\{P_e\}, and {σe}\{\sigma_e\} via gradient descent to minimize Lalign\mathcal{L}_{\text{align}}
        Project {σe}\{\sigma_e\} to enforce the scale constraint: ∏e∈Eσe=1\prod_{e \in \mathcal{E}} \sigma_e = 1
    until convergence (typically a few hundred iterations)
    return globally aligned pointmaps {χn}n=1N\{\chi^n\}_{n=1}^N

    To optionally extract explicit camera poses {Pn}\{P_n\}, intrinsics {Kn}\{K_n\}, and depthmaps {Dn}\{D^n\}, global pointmaps are constrained to the standard pinhole model χi,jn=Pn−1h(Kn−1Di,jn[i,j,1]⊤)\chi_{i,j}^n = P_n^{-1} h(K_n^{-1} D_{i,j}^n [i, j, 1]^\top), where h(x,y,z)=(x,y,z,1)⊤h(x, y, z) = (x, y, z, 1)^\top denotes homogeneous coordinates.

  4. Knowl 4 — Estimating Camera Intrinsics and Relative Pose from Pointmaps

    model/method

    The pointmap representation output by DUSt3R allows extracting camera intrinsics and pairwise relative poses:

    1. Focal Length Estimation: Assuming square pixels and a centered principal point (W/2,H/2)(W/2, H/2), the focal length f1∗f_1^* for camera 1 is recovered from pointmap X1,1X^{1,1} and confidence map C1,1C^{1,1} by solving: f1∗=arg⁡min⁡f1∑i=1W∑j=1HCi,j1,1∥(i−W2,j−H2)−f1(Xi,j,01,1,Xi,j,11,1)Xi,j,21,1∥f_1^* = \arg\min_{f_1} \sum_{i=1}^{W} \sum_{j=1}^{H} C_{i,j}^{1,1} \left\Vert \left(i - \frac{W}{2}, j - \frac{H}{2}\right) - f_1 \frac{(X_{i,j,0}^{1,1}, X_{i,j,1}^{1,1})}{X_{i,j,2}^{1,1}} \right\Vert which is solved iteratively using the Weiszfeld algorithm. The focal length f2∗f_2^* for the second camera is estimated analogously by running inference on (I2,I1)(I^2, I^1) to obtain X2,2X^{2,2}.

    2. Relative Pose Estimation: The relative pose P∗=σ∗[R∗∣t∗]P^* = \sigma^* [R^* \mid t^*] between views 1 and 2 can be solved via weighted Procrustes alignment in closed form: P∗=arg⁡min⁡σ,R,t∑iCi1,1Ci1,2∥σ(RXi1,1+t)−Xi1,2∥2P^* = \arg\min_{\sigma, R, t} \sum_{i} C_i^{1,1} C_i^{1,2} \left\Vert \sigma (R X_i^{1,1} + t) - X_i^{1,2} \right\Vert^2 or via Perspective-n-Point with RANSAC (PnP-RANSAC) using 2D matches and recovered 3D point coordinates for enhanced robustness to outliers.

  5. Knowl 5 — Pixel Correspondences and Monocular/Multi-View Depth Extraction

    model/method

    Downstream geometric quantities are derived from DUSt3R pointmaps without task-specific architectural modifications:

    1. Dense Pixel Correspondences: Because X1,1X^{1,1} and X2,1X^{2,1} are predicted in the same coordinate frame, 2D correspondences are extracted via reciprocal (mutual) nearest neighbor (NN) matching in 3D coordinate space: M1,2={(a,b)∣a=NN1,2(b) and b=NN2,1(a)}\mathcal{M}_{1,2} = \{(a, b) \mid a = \text{NN}^{1,2}(b) \text{ and } b = \text{NN}^{2,1}(a)\} where NNn,m(a)=arg⁡min⁡b∥Xbn,1−Xam,1∥\text{NN}^{n,m}(a) = \arg\min_{b} \|X_b^{n,1} - X_a^{m,1}\|.

    2. Monocular Depth Estimation: For a single image II, depth is predicted by feeding identical pairs (I,I)(I, I) into the pairwise network f(I,I)f(I, I) and taking the zz-coordinate channel of the resulting pointmap X1,1X^{1,1}.

    3. Multi-View Depth Fusion: For multi-view inputs, predicted depthmaps (extracted from the zz-coordinates of pairwise pointmaps) are brought into a consistent scale through median or scale alignment, and aggregated across overlapping views via confidence-weighted averaging.

  6. Knowl 6 — Training Configuration and Architecture Setup of DUSt3R

    experimental setup

    DUSt3R is instantiated and trained with the following specifications:

    • Architecture: ViT-Large encoder (patch size 16×1616 \times 16) and ViT-Base decoder (patch size 16×1616 \times 16), followed by Dense Prediction Transformer (DPT) regression heads. Network weights are initialized using an off-the-shelf CroCo (Cross-View Completion) pretrained model.
    • Dataset Mixture: Trained on 8.5 million pairs sampled evenly across eight public datasets: Habitat, MegaDepth, ARKitScenes, Static Scenes 3D, Blended MVS, ScanNet++, CO3D-v2, and Waymo.
    • Training Schedule: Sequential resolution training starting at 224×224224 \times 224 pixels, followed by fine-tuning on images resized to 512 pixels along the largest dimension with diverse random aspect ratios (e.g., 16:9, 4:3).
    • Zero-Shot Evaluation: The single resulting model (DUSt3R 512) is evaluated across all downstream tasks without task-specific fine-tuning.

  7. Knowl 7 — Map-Free Visual Relocalization Performance

    data/table

    DUSt3R was evaluated zero-shot on the privately held-out test set of the Map-free visual relocalization benchmark against state-of-the-art matching and pose estimation baselines. Metric scale was recovered using off-the-shelf DPT-KITTI depth maps.

    Method VCRE (<90<90px) Pose Error (<25<25cm and 5∘5^\circ)
    Reproj. ↓\downarrow Prec. ↑\uparrow AUC ↑\uparrow Median Error ↓\downarrow Precision ↑\uparrow AUC ↑\uparrow
    RPR 147.1 px 40.2% 0.402 1.68m, 22.5∘^\circ 6.0% 0.060
    SIFT 222.8 px 25.0% 0.504 2.93m, 61.4∘^\circ 10.3% 0.252
    SP+SG 160.3 px 36.1% 0.602 1.88m, 25.4∘^\circ 16.8% 0.346
    LoFTR (DPT) 166.7 px 33.4% 0.618 2.31m, 39.4∘^\circ 9.8% 0.269
    LoFTR (KBR) 165.0 px 34.3% 0.634 2.23m, 37.8∘^\circ 11.0% 0.295
    RoMa 128.8 px 45.6% 0.669 1.23m, 11.1∘^\circ 22.8% 0.407
    FAR 137.0 px 44.2% 0.680 1.48m, 17.2∘^\circ 17.7% 0.392
    DUSt3R 115.8 px 50.3% 0.697 0.98m, 7.1∘^\circ 21.4% 0.393

    DUSt3R achieved the best Virtual Correspondence Reprojection Error (VCRE) AUC (0.697), the lowest reprojection error (115.8 px), and the lowest median pose error (0.98m translation, 7.1∘^\circ rotation), outperforming feature matching methods specifically trained for correspondence estimation.

  8. Knowl 8 — Multi-View Camera Pose Estimation Performance

    data/table

    Multi-view relative camera pose estimation performance evaluated on 10 random frames from sequences in CO3Dv2 and RealEstate10K using Relative Rotation Accuracy (RRA@15), Relative Translation Accuracy (RTA@15), and mean Average Accuracy (mAA@30).

    Methods Co3Dv2 RealEstate10K
    RRA@15 ↑\uparrow RTA@15 ↑\uparrow mAA(30) ↑\uparrow mAA(30) ↑\uparrow
    RelPose 57.1 - - -
    Colmap+SPSG 36.1 27.3 25.3 45.2
    PixSfM 33.7 32.9 30.1 49.4
    PosReg 53.2 49.1 45.0 -
    PoseDiffusion 80.5 79.8 66.5 48.0
    DUSt3R 512 (w/ PnP) 94.3 88.4 77.2 61.2
    DUSt3R 512 (w/ GA) 96.2 86.8 76.7 67.7

    DUSt3R with Global Alignment (GA) and DUSt3R with PnP-RANSAC both outperformed previous learning-based (PoseDiffusion) and structure-based (COLMAP+SPSG, PixSfM) approaches on both datasets by substantial margins without being trained on RealEstate10K.

  9. Knowl 9 — Multi-View Depth Estimation Benchmark Results

    data/table

    Multi-view stereo depth estimation performance across five datasets (KITTI, ScanNet, ETH3D, DTU, Tanks and Temples). Methods are evaluated by Absolute Relative Error (rel ↓\downarrow) and Inlier Ratio (τ>1.03\tau > 1.03 ↑\uparrow). Predictions without explicit metric scale are normalized using median ground-truth scaling.

    Method KITTI ScanNet ETH3D DTU TT Average
    rel τ\tau rel τ\tau rel τ\tau rel τ\tau rel τ\tau rel τ\tau
    COLMAP (Classical) 12.0 58.2 14.6 34.2 16.4 55.1 0.7 96.5 2.7 95.0 9.3 67.8
    COLMAP Dense 26.9 52.7 38.0 22.5 89.8 23.2 20.8 69.3 25.7 76.4 40.2 48.8
    DeMoN 15.5 15.2 12.0 21.0 17.4 15.4 21.8 16.6 13.0 23.2 16.0 18.3
    DeepV2D (KITTI) (3.1) (74.9) 23.7 11.1 27.1 10.1 24.8 8.1 34.1 9.1 22.6 22.7
    DeepV2D (ScanNet) 10.0 36.2 (4.4) (54.8) 11.8 29.3 7.7 33.0 8.9 46.4 8.6 39.9
    DUSt3R 224-NoCroCo 15.14 21.16 7.54 40.00 9.51 40.07 3.56 62.83 11.12 37.90 9.37 40.39
    DUSt3R 224 15.39 26.69 (5.86) (50.84) 4.71 61.74 2.76 77.32 5.54 56.38 6.85 54.59
    DUSt3R 512 9.11 39.49 (4.93) (60.20) 2.91 76.91 3.52 69.33 3.17 76.68 4.73 64.52

    DUSt3R 512 achieved state-of-the-art multi-view depth estimation on un-posed and un-calibrated inputs, attaining the lowest average relative error (4.73) and running in 0.13s per pair, significantly faster than classical pipelines such as COLMAP (≈200\approx 200s).

  10. Knowl 10 — Zero-Shot Monocular Depth Estimation Performance

    data/table

    Zero-shot monocular depth estimation evaluated using Absolute Relative Error (AbsRel ↓\downarrow) and threshold accuracy (δ1.25<1.25\delta_{1.25} < 1.25 ↑\uparrow) across outdoor and indoor datasets by feeding single images as duplicate pairs f(I,I)f(I, I).

    Method Setting DDAD KITTI BONN NYUD-v2 TUM
    Rel ↓\downarrow δ1.25↑\delta_{1.25} \uparrow Rel ↓\downarrow δ1.25↑\delta_{1.25} \uparrow Rel ↓\downarrow δ1.25↑\delta_{1.25} \uparrow Rel ↓\downarrow δ1.25↑\delta_{1.25} \uparrow Rel ↓\downarrow δ1.25↑\delta_{1.25} \uparrow
    DPT-BEiT Supervised 10.70 84.63 9.45 89.27 - - 5.40 96.54 10.45 89.68
    NeWCRFs Supervised 9.59 82.92 5.43 91.54 - - 6.22 95.58 14.63 82.95
    Monodepth2 Self-Sup. 23.91 75.22 11.42 86.90 56.49 35.18 16.19 74.50 31.20 47.42
    SC-SfM Self-Sup. 16.92 77.28 11.83 86.61 21.11 71.40 13.79 79.57 22.29 64.30
    SC-DepthV3 Self-Sup. 14.20 81.27 11.79 86.39 12.58 88.92 12.34 84.80 16.28 79.67
    SlowTv Zero-shot 12.63 79.34 (6.84) (56.17) - - 11.59 87.23 15.02 80.86
    DUSt3R 224 Zero-shot 16.32 77.58 16.97 77.89 11.05 89.95 10.28 88.92 17.61 75.44
    DUSt3R 512 Zero-shot 13.88 81.17 10.74 86.60 8.08 93.56 6.50 94.09 14.17 79.89

    DUSt3R performs zero-shot monocular depth estimation on both indoor and outdoor environments without task-specific training, outperforming self-supervised methods and performing on par with fully supervised task-specific baselines.

Coverage note — No substantial contributed material was omitted; the knowls cover the pointmap formulation, network architecture, confidence-weighted loss, multi-view global alignment, intrinsics/pose/matching extraction, and zero-shot empirical evaluations across visual localization, relative pose estimation, multi-view depth, monocular depth, and dense 3D reconstruction.

References

  1. 1.Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. Large-scale data for multiple-view stereopsis. IJCV, 2016.
  2. 2.Eduardo Arnold, Jamie Wynn, Sara Vicente, Guillermo Garcia-Hernando, Áron Monszpart, Victor Adrian Prisacariu, Daniyar Turmukhambetov, and Eric Brachmann. Map-free visual relocalization: Metric pose relative to a single image. In ECCV, 2022.
  3. 3.Daniel Barath, Dmytro Mishkin, Luca Cavalli, Paul-Edouard Sarlin, Petr Hruby, and Marc Pollefeys. Affineglue: Joint matching and robust estimation, 2023.
  4. 4.Axel Barroso-Laguna, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk. Key. net: Keypoint detection by handcrafted and learned cnn filters. In ICCV, pages 5836–5844, 2019.
  5. 5.Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In ECCV, pages 404–417. Springer, 2006.
  6. 6.Jia-Wang Bian, Huangying Zhan, Naiyan Wang, Tat-Jun Chin, Chunhua Shen, and Ian D. Reid. Auto-rectify network for unsupervised indoor depth estimation. IEEE Trans. Pattern Anal. Mach. Intell., 44(12):9802–9813, 2022.
  7. 7.Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother. DSAC - differentiable RANSAC for camera localization. In CVPR, 2017.
  8. 8.Eric Brachmann and Carsten Rother. Learning less is more - 6d camera localization via 3d surface regression. In CVPR, 2018.
  9. 9.Eric Brachmann and Carsten Rother. Neural-guided RANSAC: learning where to sample model hypotheses. In ICCV, pages 4321–4330. IEEE, 2019.
  10. 10.Eric Brachmann and Carsten Rother. Visual camera relocalization from RGB and RGB-D images using DSAC. PAMI, 2022.
  11. 11.Neill D. F. Campbell, George Vogiatzis, Carlos Hernández, and Roberto Cipolla. Using multiple hypotheses to improve depth-maps for multi-view stereo. In ECCV, 2008.
  12. 12.Carlos Campos, Richard Elvira, Juan J. Gómez Rodríguez, José M. M. Montiel, and Juan D. Tardós. Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam. IEEE Transactions on Robotics, 2021.
  13. 13.Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural slam. arXiv preprint arXiv:2004.05155, 2020.
  14. 14.Guanying Chen, Kai Han, Boxin Shi, Yasuyuki Matsushita, and Kwan-Yee K. Wong. Deep photometric stereo for nonlambertian surfaces. PAMI, 2022.
  15. 15.Hongkai Chen, Zixin Luo, Lei Zhou, Yurun Tian, Mingmin Zhen, Tian Fang, David McKinnon, Yanghai Tsin, and Long Quan. Aspanformer: Detector-free image matching with adaptive span transformer. ECCV, 2022.
  16. 16.Shuo Cheng, Zexiang Xu, Shilin Zhu, Zhuwen Li, Li Erran Li, Ravi Ramamoorthi, and Hao Su. Deep stereo using adaptive thin volume representation with uncertainty awareness. In CVPR, 2020.
  17. 17.Roberto Cipolla, Yarin Gal, and Alex Kendall. Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. In CVPR, 2018.
  18. 18.David Crandall, Andrew Owens, Noah Snavely, and Daniel Huttenlocher. SfM with MRFs: Discrete-continuous optimization for large-scale structure from motion. PAMI, 2013.
  19. 19.Hainan Cui, Xiang Gao, Shuhan Shen, and Zhanyi Hu. Hsfm: Hybrid structure-from-motion. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
  20. 20.Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richlyannotated 3d reconstructions of indoor scenes. In CVPR, 2017.
  21. 21.Amaury Dame, Victor A Prisacariu, Carl Y Ren, and Ian Reid. Dense reconstruction using 3d object shape priors. In CVPR, pages 1288–1295, 2013.
  22. 22.Afshin Dehghan, Gilad Baruch, Zhuoyuan Chen, Yuri Feigin, Peter Fu, Thomas Gebauer, Daniel Kurz, Tal Dimry, Brandon Joffe, Arik Schwartz, and Elad Shulman. ARKitScenes: A diverse real-world dataset for 3d indoor scene understanding using mobile RGB-D data. In NeurIPS Datasets and Benchmarks, 2021.
  23. 23.Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In CVPR Workshops, pages 224–236, 2018.
  24. 24.Siyan Dong, Shuzhe Wang, Yixin Zhuang, Juho Kannala, Marc Pollefeys, and Baoquan Chen. Visual localization via few-shot scene region classification. In 2022 International Conference on 3D Vision (3DV), pages 393–402. IEEE, 2022.
  25. 25.Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
  26. 26.Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-net: A trainable CNN for joint description and detection of local features. In CVPR, pages 8092–8101, 2019.
  27. 27.Johan Edstedt, Ioannis Athanasiadis, Mårten Wadenbäck, and Michael Felsberg. DKM: Dense Kernelized Feature Matching for Geometry Estimation. In CVPR, 2023.
  28. 28.Johan Edstedt, Qiyu Sun, Georg Bökman, Mårten Wadenbäck, and Michael Felsberg. RoMa: Robust Dense Feature Matching. In CVPR, 2024.
  29. 29.Martin A. Fischler and Robert C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981.
  30. 30.Qiancheng Fu, Qingshan Xu, Yew Soon Ong, and Wenbing Tao. Geo-neus: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction. In NeurIPS, 2022.
  31. 31.Yasutaka Furukawa and Carlos Hernández. Multi-view stereo: A tutorial. Found. Trends Comput. Graph. Vis., 2015.
  32. 32.Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. PAMI, 2010.
  33. 33.Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In ICCV, June 2015.
  34. 34.Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The KITTI dataset. Int. J. Robotics Res., 32(11):1231–1237, 2013.
  35. 35.Clément Godard, Oisin Mac Aodha, and Gabriel J. Brostow. Unsupervised monocular depth estimation with left-right consistency. In CVPR, 2017.
  36. 36.Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow. Digging into self-supervised monocular depth estimation. In ICCV, pages 3827–3837. IEEE, 2019.
  37. 37.Leonardo Gomes, Olga Regina Pereira Bellon, and Luciano Silva. 3d reconstruction methods for digital preservation of cultural heritage: A survey. Pattern Recognit. Lett., 2014.
  38. 38.Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3d packing for self-supervised monocular depth estimation. In CVPR, pages 2482–2491, 2020.
  39. 39.Chris Harris, Mike Stephens, et al. A combined corner and edge detector. In Alvey vision conference, volume 15, pages 10–5244. Citeseer, 1988.
  40. 40.Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2004.
  41. 41.Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
  42. 42.Nianjuan Jiang, Zhaopeng Cui, and Ping Tan. A global linear method for camera pose registration. In ICCV, 2013.
  43. 43.Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 36(4), 2017.
  44. 44.Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. Epnp: An accurate O(n) solution to the pnp problem. IJCV, 2009.
  45. 45.Vincent Leroy, Jean-Sébastien Franco, and Edmond Boyer. Volume sweeping: Learning photoconsistency for multiview shape reconstruction. IJCV, 2021.
  46. 46.Xiaotian Li, Shuzhe Wang, Yi Zhao, Jakob Verbeek, and Juho Kannala. Hierarchical scene coordinate classification and regression for visual localization. In CVPR, 2020.
  47. 47.Zhengqi Li and Noah Snavely. Megadepth: Learning singleview depth prediction from internet photos. In CVPR, pages 2041–2050, 2018.
  48. 48.Chen-Hsuan Lin, Chen Kong, and Simon Lucey. Learning efficient point cloud generation for dense 3d object reconstruction. In AAAI, 2018.
  49. 49.Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: bundle-adjusting neural radiance fields. In ICCV, 2021.
  50. 50.Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Marc Pollefeys. Pixel-perfect structure-from-motion with featuremetric refinement. In ICCV, 2021.
  51. 51.Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Pollefeys. Lightglue: Local feature matching at light speed. In ICCV, 2023.
  52. 52.Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot One Image to 3D Object. In CVPR, 2023.
  53. 53.David G. Lowe. Distinctive image features from scaleinvariant keypoints. IJCV, 2004.
  54. 54.Bin Luo and Edwin R. Hancock. Procrustes alignment with the EM algorithm. In Computer Analysis of Images and Patterns, CAIP, volume 1689 of Lecture Notes in Computer Science, pages 623–631. Springer, 1999.
  55. 55.Zeyu Ma, Zachary Teed, and Jia Deng. Multiview stereo with cascaded epipolar raft. In ECCV, 2022.
  56. 56.N. Mayer, E. Ilg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In CVPR, 2016.
  57. 57.Xiaoxu Meng, Weikai Chen, and Bo Yang. Neat: Learning neural implicit surfaces with arbitrary topologies from multiview images. In CVPR, 2023.
  58. 58.Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
  59. 59.Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics, 2015.
  60. 60.Michael Niemeyer, Lars M. Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR, 2020.
  61. 61.Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In ICCV, 2021.
  62. 62.Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
  63. 63.Onur Özyeşil, Vladislav Voroninski, Ronen Basri, and Amit Singer. A survey of structure from motion*. Acta Numerica, 26:305–364, 2017.
  64. 64.Emanuele Palazzolo, Jens Behley, Philipp Lottes, Philippe Giguère, and Cyrill Stachniss. Refusion: 3d reconstruction in dynamic environments for RGB-D cameras exploiting residuals. In 2IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7855–7862, 2019.
  65. 65.Rémi Pautrat, Iago Suárez, Yifan Yu, Marc Pollefeys, and Viktor Larsson. GlueStick: Robust image matching by sticking points and lines together. In ICCV, 2023.
  66. 66.Dario Pavllo, Jonas Kohler, Thomas Hofmann, and Aurélien Lucchi. Learning generative models of textured 3d meshes from real-world images. In ICCV, 2021.
  67. 67.Dario Pavllo, Graham Spinks, Thomas Hofmann, Marie-Francine Moens, and Aurélien Lucchi. Convolutional generation of textured 3d meshes. In NeurIPS, 2020.
  68. 68.Dario Pavllo, David Joseph Tan, Marie-Julie Rakotosaona, and Federico Tombari. Shape, pose, and appearance from a single image via bootstrapped radiance field inversion. In CVPR, 2023.
  69. 69.Rui Peng, Rongjie Wang, Zhenyu Wang, Yawen Lai, and Ronggang Wang. Rethinking depth estimation for multiview stereo: A unified representation. In CVPR, 2022.
  70. 70.MV Peppa, JP Mills, KD Fieber, I Haynes, S Turner, A Turner, M Douglas, and PG Bryan. Archaeological feature detection from archive aerial photography with a sfm-mvs and image enhancement pipeline. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 42:869–875, 2018.
  71. 71.Frank Plastria. The Weiszfeld Algorithm: Proof, Amendments, and Extensions, pages 357–389. Springer US, 2011.
  72. 72.Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pages 77–85, 2017.
  73. 73.René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, 2021.
  74. 74.René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. CoRR, 1907.01341/abs, 2020.
  75. 75.Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In ICCV, pages 10881–10891, 2021.
  76. 76.Jérôme Revaud, Yohann Cabon, Romain Brégier, Jong-Min Lee, and Philippe Weinzaepfel. SACReg: Scene-agnostic coordinate regression for visual localization. CoRR, abs/2307.11702, 2023.
  77. 77.Jérôme Revaud, César Roberto de Souza, Martin Humenberger, and Philippe Weinzaepfel. R2D2: reliable and repeatable detector and descriptor. In Neurips, pages 12405–12415, 2019.
  78. 78.Carlos Ricolfe-Viala and Alicia Esparza. The Influence of Autofocus Lenses in the Camera Calibration Process, 2024.
  79. 79.Chris Rockwell, Nilesh Kulkarni, Linyi Jin, Jeong Joon Park, Justin Johnson, and David F. Fouhey. Far: Flexible, accurate and robust 6dof relative camera pose estimation, 2024.
  80. 80.Edward Rosten and Tom Drummond. Machine learning for high-speed corner detection. In ECCV. Springer, 2006.
  81. 81.Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In CVPR, pages 4937–4946, 2020.
  82. 82.Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied ai research. In ICCV, 2019.
  83. 83.Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  84. 84.Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In ECCV, 2016.
  85. 85.Thomas Schöps, Johannes L. Schönberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. In CVPR, 2017.
  86. 86.Philipp Schröppel, Jan Bechtold, Artemij Amiranashvili, and Thomas Brox. A benchmark and a baseline for robust multi-view depth estimation. In 3DV, pages 637–645, 2022.
  87. 87.Qi Shan, Brian Curless, Yasutaka Furukawa, Carlos Hernández, and Steven M Seitz. Occluding contours for multi-view stereo. In CVPR, pages 4002–4009, 2014.
  88. 88.Daeyun Shin, Charless C. Fowlkes, and Derek Hoiem. Pixels, voxels, and views: A study of shape representations for single view 3d object shape prediction. In CVPR, 2018.
  89. 89.Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from RGBD images. In ECCV, pages 746–760, 2012.
  90. 90.Jaime Spencer, Chris Russell, Simon Hadfield, and Richard Bowden. Kick back & relax: Learning to reconstruct the world by watching slowtv. In ICCV, 2023.
  91. 91.Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of RGB-D SLAM systems. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 573–580. IEEE, 2012.
  92. 92.Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-free local feature matching with transformers. CVPR, 2021.
  93. 93.Libo Sun, Jia-Wang Bian, Huangying Zhan, Wei Yin, Ian Reid, and Chunhua Shen. Sc-depthv3: Robust self-supervised monocular depth estimation for dynamic scenes. CoRR, 2211.03660, 2022.
  94. 94.Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, June 2020.
  95. 95.Shitao Tang, Chengzhou Tang, Rui Huang, Siyu Zhu, and Ping Tan. Learning camera localization via dense scene matching. In CVPR, 2021.
  96. 96.Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers. ICLR, 2022.
  97. 97.Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Multi-view 3d models from single images with a convolutional network. In ECCV, 2016.
  98. 98.Zachary Teed and Jia Deng. Deepv2d: Video to depth with differentiable structure from motion. In ICLR, 2020.
  99. 99.Sebastian Thrun. Probabilistic robotics. Communications of the ACM, 45(3):52–57, 2002.
  100. 100.Engin Tola, Christoph Strecha, and Pascal Fua. Efficient large-scale multi-view stereo for ultra high-resolution image sets. Mach. Vis. Appl., 2012.
  101. 101.Michał Tyszkiewicz, Pascal Fua, and Eduard Trulls. Disk: Learning local features with policy gradient. Advances in Neural Information Processing Systems, 33:14254–14265, 2020.
  102. 102.Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, and Thomas Brox. DeMoN: Depth and motion network for learning monocular stereo. In CVPR, pages 5622–5631, 2017.
  103. 103.Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Pablo Speciale, and Marc Pollefeys. Patchmatchnet: Learned multi-view patchmatch stereo. In CVPR, pages 14194–14203, 2021.
  104. 104.Jianyuan Wang, Christian Rupprecht, and David Novotný. Posediffusion: Solving pose estimation via diffusion-aided bundle adjustment. In ICCV, 2023.
  105. 105.Jinglu Wang, Bo Sun, and Yan Lu. Mvpnet: Multi-view point regression networks for 3d object reconstruction from A single image. In AAAI, 2019.
  106. 106.Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurIPS, 2021.
  107. 107.Shuzhe Wang, Juho Kannala, Marc Pollefeys, and Daniel Barath. Guiding local feature matching with surface curvature. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 17981–17991, October 2023.
  108. 108.Shuzhe Wang, Zakaria Laskar, Iaroslav Melekhov, Xiaotian Li, and Juho Kannala. Continual learning for image-based camera localization. In ICCV, pages 3252–3262, 2021.
  109. 109.Shuzhe Wang, Zakaria Laskar, Iaroslav Melekhov, Xiaotian Li, Yi Zhao, Giorgos Tolias, and Juho Kannala. Hscnet++: Hierarchical scene coordinate classification and regression for visual localization with transformer. International Journal of Computer Vision, pages 1–21, 2024.
  110. 110.Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. Hfneus: Improved surface reconstruction using high-frequency details. In NeurIPS, 2022.
  111. 111.Yuesong Wang, Zhaojie Zeng, Tao Guan, Wei Yang, Zhuo Chen, Wenkai Liu, Luoyuan Xu, and Yawei Luo. Adaptive patch deformation for textureless-resilient multi-view stereo. In CVPR, 2023.
  112. 112.Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In ICCV, 2021.
  113. 113.Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and Jérôme Revaud. CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow. In ICCV, 2023.
  114. 114.Weinzaepfel, Philippe and Leroy, Vincent and Lucas, Thomas and Brégier, Romain and Cabon, Yohann and Arora, Vaibhav and Antsfeld, Leonid and Chidlovskii, Boris and Csurka, Gabriela and Revaud Jérôme. CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion. In NeurIPS, 2022.
  115. 115.Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from a single image. In CVPR, 2020.
  116. 116.Yuxi Xiao, Nan Xue, Tianfu Wu, and Gui-Song Xia. Level-S 2 fM: Structure From Motion on Neural Level Set of Implicit Surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  117. 117.Guangkai Xu, Wei Yin, Hao Chen, Chunhua Shen, Kai Cheng, and Feng Zhao. Frozenrecon: Pose-free 3d scene reconstruction with frozen depth models. In ICCV, 2023.
  118. 118.Qingshan Xu and Wenbing Tao. Learning inverse depth regression for multi-view stereo with correlation cost volume. In AAAI, 2020.
  119. 119.Jiayu Yang, Wei Mao, Jose M. Álvarez, and Miaomiao Liu. Cost volume pyramid based depth inference for multi-view stereo. In CVPR, pages 4876–4885, 2020.
  120. 120.Luwei Yang, Ziqian Bai, Chengzhou Tang, Honghua Li, Yasutaka Furukawa, and Ping Tan. Sanet: Scene agnostic network for camera localization. In ICCV, 2019.
  121. 121.Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In ECCV, 2018.
  122. 122.Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In CVPR, pages 1787–1796, 2020.
  123. 123.Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In NeurIPS, 2020.
  124. 124.Xinyi Ye, Weiyue Zhao, Tianqi Liu, Zihao Huang, Zhiguo Cao, and Xin Li. Constraining depth map geometry for multi-view stereo: A dual-depth approach with saddle-shaped depth cells. ICCV, 2023.
  125. 125.Zhichao Ye, Chong Bao, Xin Zhou, Haomin Liu, Hujun Bao, and Guofeng Zhang. Ec-sfm: Efficient covisibility-based structure-from-motion for both sequential and unordered images. CoRR, abs/2302.10544, 2023.
  126. 126.Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
  127. 127.Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. Lift: Learned invariant feature transform. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14, pages 467–483. Springer, 2016.
  128. 128.Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from a single image. In ICCV, 2023.
  129. 129.Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Simon Chen, Yifan Liu, and Chunhua Shen. Towards accurate reconstruction of 3d scene shape from a single monocular image, 2022.
  130. 130.Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Simon Chen, Yifan Liu, and Chunhua Shen. Towards accurate reconstruction of 3d scene shape from a single monocular image. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
  131. 131.Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to recover 3d scene shape from a single image. In CVPR, 2020.
  132. 132.Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. Neural window fully-connected crfs for monocular depth estimation. In CVPR, pages 3906–3915, 2022.
  133. 133.Zhaojie Zeng. OpenMVS. https://github.com/cdcseacave/openMVS, 2015. [Online; accessed 19-October-2023].
  134. 134.Jingyang Zhang, Shiwei Li, Zixin Luo, Tian Fang, and Yao Yao. Vis-mvsnet: Visibility-aware multi-view stereo network. Int. J. Comput. Vis., 131(1):199–214, 2023.
  135. 135.Jason Y. Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose: Predicting probabilistic relative rotation for single objects in the wild. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, ECCV, pages 592–611, 2022.
  136. 136.Zhe Zhang, Rui Peng, Yuxi Hu, and Ronggang Wang. Geomvsnet: Learning multi-view stereo with geometry perception. In CVPR, 2023.
  137. 137.Chen Zhao, Yixiao Ge, Feng Zhu, Rui Zhao, Hongsheng Li, and Mathieu Salzmann. Progressive correspondence pruning by consensus learning. In ICCV, 2021.
  138. 138.Chaoqiang Zhao, Youmin Zhang, Matteo Poggi, Fabio Tosi, Xianda Guo, Zheng Zhu, Guan Huang, Yang Tang, and Stefano Mattoccia. MonoViT: Self-supervised monocular depth estimation with a vision transformer. In International Conference on 3D Vision (3DV), sep 2022.
  139. 139.Huizhong Zhou, Benjamin Ummenhofer, and Thomas Brox. DeepTAM: Deep tracking and mapping with convolutional neural networks. Int. J. Comput. Vis., 128(3):756–769, 2020.
  140. 140.Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM Trans. Graph. (Proc. SIGGRAPH), 37, 2018.

Citation

MLA
Wang, S., et al. “DUSt3R: Geometric 3D Vision Made Easy”. arXiv, 2023, http://arxiv.org/abs/2312.14132v3.
APA
Wang, S., Leroy, V., Cabon, Y., Chidlovskii, B., & Revaud, J. (2023). DUSt3R: Geometric 3D Vision Made Easy. arXiv. http://arxiv.org/abs/2312.14132v3
Chicago
Wang, S., V. Leroy, Y. Cabon, B. Chidlovskii, and J. Revaud. 2023. “DUSt3R: Geometric 3D Vision Made Easy”. arXiv. http://arxiv.org/abs/2312.14132v3.
Harvard
Wang, S. et al. (2023) “DUSt3R: Geometric 3D Vision Made Easy”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2312.14132v3.
Vancouver
1. Wang S, Leroy V, Cabon Y, Chidlovskii B, Revaud J (2023) DUSt3R: Geometric 3D Vision Made Easy. arXiv

BibTeX

@article{wang2023dust3r,
  title = {DUSt3R: Geometric 3D Vision Made Easy},
  author = {Wang, Shuzhe and Leroy, Vincent and Cabon, Yohann and Chidlovskii, Boris and Revaud, Jerome},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2312.14132v3},
  eprint = {2312.14132}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE