Digging Into Self-Supervised Monocular Depth Estimation

Clément GodardOisin Mac AodhaGabriel Brostow

article2018ICCV2,551 citations

Demonstrates that self-supervised monocular depth estimation can achieve state-of-the-art accuracy through simple design choices, introducing a minimum reprojection loss to handle occlusions, full-resolution sampling to reduce visual artifacts, and auto-masking to ignore moving objects.

Listen

The article addresses the challenge of estimating dense depth from single color images, a task that is ill-posed without ground-truth depth data which is expensive to collect at scale. Self-supervised approaches that rely on image reconstruction from stereo pairs or monocular video offer a practical alternative, yet prior methods often produced artifacts around occlusions, moving objects, and low-texture regions, limiting their reliability for applications such as autonomous driving and augmented reality.

The work set out to evaluate whether a small set of targeted changes to the loss functions and training procedure could close much of the remaining gap between self-supervised and fully supervised depth estimation. The authors conducted experiments on the KITTI benchmark, training models with monocular video, stereo pairs, or both, and systematically ablated each proposed component.

Three modifications were introduced and validated: a per-pixel minimum reprojection loss that selects the best-matching source view to handle occlusions, an auto-masking term that automatically excludes pixels violating the static-scene or moving-camera assumptions, and a full-resolution multi-scale sampling strategy that computes photometric losses at native image resolution to reduce texture-copy artifacts. All models used a compact ResNet-18 encoder-decoder architecture initialized from ImageNet weights.

The resulting Monodepth2 models achieve state-of-the-art accuracy among self-supervised methods on the Eigen split, with absolute relative error dropping to 0.115 (monocular), 0.109 (stereo), and 0.106 (mixed) compared with 0.141–0.183 for prior leading approaches. Each component contributes measurable gains, and the full model remains competitive even when evaluated on higher-quality ground truth or at increased resolution. Qualitatively, depth maps are sharper at object boundaries and contain fewer holes around moving vehicles.

These results indicate that careful loss design can deliver high-quality depth without additional network complexity or ground-truth supervision, lowering the cost and data requirements for downstream tasks. The findings also suggest that many recent architectural elaborations may be unnecessary once the core training signals are made more robust.

The authors recommend adopting the three loss modifications as a new baseline for self-supervised depth work and note that further gains are possible by combining them with higher-resolution training or post-processing. Additional validation on diverse real-world video and integration with other perception tasks would strengthen deployment readiness.

The study is limited to the KITTI driving domain, relies on photometric consistency assumptions that break on reflective or saturated surfaces, and reports monocular results after median scaling to ground truth. Performance on unseen environments and under extreme lighting therefore remains to be quantified, warranting caution when extrapolating beyond the evaluated conditions.

California Institute of Technology Niantic University College London contrastive learning depth estimation image formations Monodepth2 Minimum Reprojection Loss Auto‑Masking Loss

arXiv: 1806.01260 nianticlabs/monodepth2

Paper: Unsupervised Monocular Depth Estimation with Left-Right Consistency, Clément Godard et al. (2016). Introduces the foundational unsupervised monocular depth estimation framework using stereo photometric consistency and left-right consistency that Monodepth2 directly builds upon and enhances.
Paper: Unsupervised Learning of Depth and Ego-Motion from Video, Tinghui Zhou et al. (2017). Establishes joint unsupervised learning of monocular depth and camera ego-motion from unconstrained video sequences, providing the foundational monocular training setup adapted by Monodepth2.
Paper: Depth Map Prediction from a Single Image using a Multi-Scale Deep Network, David Eigen et al. (2014). Pioneers deep single-image depth prediction and establishes the standard KITTI Eigen evaluation protocol used throughout self-supervised depth estimation research.

Paper: Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer, René Ranftl et al. (2019). Extends monocular depth estimation beyond dataset-specific self-supervision to robust zero-shot cross-dataset transfer by training across diverse datasets using invariant loss formulations.
Paper: Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data, Lihe Yang et al. (2024). Generalizes single-image depth prediction by leveraging massive collections of unlabeled images to build a robust foundation model that overcomes domain specificity.
Paper: Depth Pro: Sharp Monocular Metric Depth in Less Than a Second, Alexey Bochkovskiy et al. (2025). Advances monocular depth estimation towards high-resolution, sharp zero-shot metric depth prediction in sub-second runtimes.

Cover for Digging Into Self-Supervised Monocular Depth Estimation

Abstract

Per-pixel ground-truth depth data is challenging to acquire at scale. To overcome this limitation, self-supervised learning has emerged as a promising alternative for training models to perform monocular depth estimation. In this paper, we propose a set of improvements, which together result in both quantitatively and qualitatively improved depth maps compared to competing self-supervised methods.

Research on self-supervised monocular training usually explores increasingly complex architectures, loss functions, and image formation models, all of which have recently helped to close the gap with fully-supervised methods. We show that a surprisingly simple model, and associated design choices, lead to superior predictions. In particular, we propose (i) a minimum reprojection loss, designed to robustly handle occlusions, (ii) a full-resolution multi-scale sampling method that reduces visual artifacts, and (iii) an auto-masking loss to ignore training pixels that violate camera motion assumptions. We demonstrate the effectiveness of each component in isolation, and show high quality, state-of-the-art results on the KITTI benchmark.

Topics

contrastive learning depth estimation image formations binocular stereo monocular image object detection estimation method

Keywords

Monodepth2 Minimum Reprojection Loss Auto‑Masking Loss Full‑Resolution Multi‑Scale Sampling Self‑Supervised Monocular Depth Estimation KITTI Benchmark ResNet18 Encoder Pose Network Image Reconstruction Loss Depth Prediction

1 Introduction
2 Related Work
2.1 Supervised Depth Estimation
2.2 Self-supervised Depth Estimation
3 Method
3.1 Self-Supervised Training
3.2 Improved Self-Supervised Depth Estimation
3.3 Additional Considerations
4 Experiments
4.1 KITTI Eigen Split
4.1.1 KITTI Ablation Study
4.2 Additional Datasets
5 Conclusion
References
A Odometry Evaluation
B Network Details
C Additional Ablation Experiments
D Additional Evaluation
D.1 Improved Ground Truth
D.2 Single-Scale Evaluation
D.3 KITTI Evaluation Server Benchmark
E Additional Qualitative Comparisons
F Results with Post-Processing
G Effect of Image Resolution
H Comparison of Pose Encoder
I Supplementary Video Results

Knowls

Knowl 1 — Per-Pixel Minimum Reprojection Loss for Occlusion Handling
model/method
In self-supervised monocular depth estimation, a depth network predicts a depth map $D_t$ for a target image $I_t$ , while a pose network estimates the camera transformation $T_{t\to t'}$ to adjacent source frames $I_{t'}$ . Standard formulations average the photometric reprojection error across all available source images $t'$ . However, averaging penalizes correct depth predictions at pixels that are visible in the target frame $I_t$ but occluded or out-of-view in one of the source frames $I_{t'}$ , leading to blurred depth discontinuities at object boundaries.

To resolve this, Monodepth2 computes the per-pixel minimum reprojection loss across all candidate source views $t'$ :

$L_p = \min_{t'} pe(I_t, I_{t'\to t})$

where $I_{t'\to t} = I_{t'}\langle \text{proj}(D_t, T_{t\to t'}, K) \rangle$ denotes the source image $I_{t'}$ warped to the target coordinate frame using the projected depth $D_t$ , camera intrinsics $K$ , and relative camera pose $T_{t\to t'}$ , sampled via a differentiable bilinear sampler $\langle \cdot \rangle$ . The function $pe(I_a, I_b)$ is a photometric error function combining L1 loss and the Structural Similarity index (SSIM). By taking the minimum over all source images, the network only matches each pixel to the source view in which it is unoccluded and visible, substantially reducing border artifacts and sharpening depth boundaries.
Knowl 2 — Auto-Masking Stationary Pixels in Monocular Self-Supervised Depth Estimation
model/method
Self-supervised monocular depth estimation assumes a moving camera observing a static scene. When these assumptions fail—such as when the camera stops moving (e.g., at traffic lights) or when another object travels at the same relative velocity as the camera—the network incorrectly predicts infinite depth (depth 'holes') because zero apparent pixel motion is explained as an object infinitely far away.

Monodepth2 introduces a binary per-pixel auto-mask $\mu \in \{0, 1\}$ computed dynamically on the forward pass without requiring auxiliary motion segmentation networks. The mask includes a pixel in the loss if and only if the photometric error of the warped source image is strictly less than the photometric error of the unwarped source image:

$\mu = \left[ \min_{t'} pe(I_t, I_{t'\to t}) < \min_{t'} pe(I_t, I_{t'}) \right]$

where $[\,]$ denotes the Iverson bracket, $I_t$ is the target image, $I_{t'}$ is a candidate source image, $I_{t'\to t}$ is the source image warped to the target frame via predicted depth and pose, and $pe(\cdot, \cdot)$ is the photometric reconstruction error function. When the camera is stationary or an object moves at the same velocity as the camera, $pe(I_t, I_{t'}) \le pe(I_t, I_{t'\to t})$ , causing $\mu = 0$ and filtering out the stationary pixels or entire static frames from contaminating the training objective.
Knowl 3 — Full-Resolution Multi-Scale Appearance Matching Strategy
model/method
Conventional self-supervised depth decoders predict disparity maps at multiple decoder resolutions and calculate photometric reconstruction errors independently at each downsampled image scale. In large low-texture regions, downsampled images create ambiguous photometric errors, leading to depth 'holes' and texture-copy artifacts (where high-frequency image textures are erroneously transferred into the predicted depth map).

Monodepth2 decouples disparity map resolution from error evaluation resolution. Instead of computing photometric error on low-resolution source and target images, intermediate low-resolution depth/disparity predictions from the decoder are upsampled to the full input image resolution $H \times W$ via bilinear interpolation. The source images are then warped, resampled, and evaluated using the photometric error metric $pe$ at the original high resolution. This acts like patch matching, forcing intermediate decoder layers to optimize directly for full-resolution scene reconstruction and eliminating texture-copy artifacts.
Knowl 4 — Photometric and Smoothness Loss Formulation in Monodepth2
equation
The overall training objective $\mathcal{L}$ in Monodepth2 is a weighted sum of the auto-masked per-pixel photometric reprojection loss $L_p$ and an edge-aware smoothness regularization loss $L_s$ , averaged over all pixels, multi-scale decoder outputs, and batch samples:

$\mathcal{L} = \mu L_p + \lambda L_s$

where:
1. The photometric error function $pe(I_a, I_b)$ between two RGB images $I_a, I_b \in \mathbb{R}^{H \times W \times 3}$ is a linear combination of SSIM and L1 distance with weighting parameter $\alpha = 0.85$ :
$pe(I_a, I_b) = \frac{\alpha}{2}(1 - \text{SSIM}(I_a, I_b)) + (1 - \alpha)\|I_a - I_b\|_1$
1. The per-pixel minimum reprojection loss $L_p$ over source views $t'$ is:
$L_p = \min_{t'} pe(I_t, I_{t'\to t})$
1. The auto-mask $\mu$ is a binary Iverson bracket indicator:
$\mu = \left[ \min_{t'} pe(I_t, I_{t'\to t}) < \min_{t'} pe(I_t, I_{t'}) \right]$
1. The edge-aware smoothness loss $L_s$ regularizes inverse depth gradients weighted by image intensity gradients:
$L_s = |\partial_x d_t^*| e^{-|\partial_x I_t|} + |\partial_y d_t^*| e^{-|\partial_y I_t|}$

where $d_t^* = d_t / \bar{d}_t$ is the mean-normalized inverse depth (disparity) to prevent the predicted depth values from shrinking toward zero, $\partial_x$ and $\partial_y$ denote spatial image gradients, and $\lambda = 0.001$ is the smoothness regularization weight.
Knowl 5 — Monodepth2 Depth and Pose Network Architectures
model/method
Monodepth2 utilizes two separate convolutional neural networks for depth prediction and relative camera pose estimation:
1. Depth Estimation Network: Built on a U-Net architecture with a ResNet18 encoder (11M parameters) initialized with ImageNet pretrained weights. The decoder uses skip connections from encoder features, ELU nonlinearities, nearest-neighbor $2\times$ upsampling, and reflection padding (rather than zero padding) to suppress border artifacts. Output disparity layers produce a sigmoid activation $\sigma$ , converted into metric depth $D$ via:
$D = \frac{1}{a\sigma + b}$

where constants $a$ and $b$ constrain predicted depth $D \in [0.1, 100]$ meters.
1. Pose Estimation Network: A modified ResNet18 network accepting two concatenated temporal RGB frames ( $6$ input channels). When initialized with ImageNet pretraining, the initial $3 \times 64 \times 3 \times 3$ convolutional kernel is duplicated across the channel dimension to form a $6 \times 64 \times 3 \times 3$ weight tensor, with all weights divided by $2$ to preserve output numerical scale. The pose decoder applies a sequence of convolutions producing a 6-DoF relative camera transformation $T_{t\to t'}$ parameterized as an axis-angle rotation and a 3D translation vector, each scaled by a factor of $0.01$ .
Knowl 6 — Supervision Regimes: Monocular (M), Stereo (S), and Joint (MS)
model/method
Monodepth2 supports three self-supervised training regimes depending on the choice of candidate source images $I_{t'}$ relative to target frame $I_t$ :
1. Monocular Supervision (M): Source images are temporally adjacent video frames, $I_{t'} \in \{I_{t-1}, I_{t+1}\}$ . Camera poses $T_{t\to t-1}$ and $T_{t\to t+1}$ are predicted dynamically during training by the pose estimation network.
2. Stereo Supervision (S): The source image is the synchronized opposite view in a stereo pair, $I_{t'} = I_{\text{stereo}}$ , where the relative transformation $T_{t\to \text{stereo}}$ is a known fixed horizontal translation baseline.
3. Joint Monocular and Stereo Supervision (MS): Candidate source images comprise both temporal frames and the opposite stereo view, $I_{t'} \in \{I_{t-1}, I_{t+1}, I_{\text{stereo}}\}$ .
Under MS training, the per-pixel minimum reprojection loss automatically selects whichever view (temporal or stereo) provides an unoccluded correspondence for each pixel, improving robustness without requiring additional ground truth.

Knowl 7 — Depth Estimation Performance on the KITTI Eigen Split

data/table

The table below compares Monodepth2 variants against prior self-supervised and supervised methods on the standard KITTI 2015 Eigen test split, evaluated up to an 80m maximum depth threshold without post-processing.

Method	Train	Abs Rel	Sq Rel	RMSE	RMSE log	$\delta < 1.25$	$\delta < 1.25^2$	$\delta < 1.25^3$
Zhou et al.	M	0.183	1.595	6.709	0.270	0.734	0.902	0.959
GeoNet	M	0.149	1.060	5.567	0.226	0.796	0.935	0.975
DDVO	M	0.151	1.257	5.583	0.228	0.810	0.936	0.974
EPC++	M	0.141	1.029	5.350	0.216	0.816	0.941	0.976
Struct2depth	M	0.141	1.026	5.291	0.215	0.816	0.945	0.979
Monodepth2 w/o pt	M	0.132	1.044	5.142	0.210	0.845	0.948	0.977
Monodepth2	M	0.115	0.903	4.863	0.193	0.877	0.959	0.981
Monodepth2 ( $1024 \times 320$ )	M	0.115	0.882	4.701	0.190	0.879	0.961	0.982
Monodepth (R50)	S	0.133	1.142	5.533	0.230	0.830	0.936	0.970
3Net (ResNet50)	S	0.129	0.996	5.281	0.223	0.831	0.939	0.974
SuperDepth + pp ( $1024 \times 382$ )	S	0.112	0.875	4.958	0.207	0.852	0.947	0.977
Monodepth2	S	0.109	0.873	4.960	0.209	0.864	0.948	0.975
Monodepth2 ( $1024 \times 320$ )	S	0.107	0.849	4.764	0.201	0.874	0.953	0.977
EPC++	MS	0.128	0.935	5.011	0.209	0.831	0.945	0.979
Monodepth2	MS	0.106	0.818	4.750	0.196	0.874	0.957	0.979
Monodepth2 ( $1024 \times 320$ )	MS	0.106	0.806	4.630	0.193	0.876	0.958	0.980

Lower is better for error metrics (Abs Rel, Sq Rel, RMSE, RMSE log); higher is better for threshold accuracy metrics ( $\delta < 1.25^k$ ). Monocular models (M) use per-image median ground-truth scaling. The results show that Monodepth2 outperforms prior monocular methods across all metrics, and even matches or exceeds contemporary stereo-supervised models when trained using only monocular video.

Knowl 8 — Ablation Analysis of Architectural and Loss Components

data/table

Ablation results on the KITTI 2015 Eigen split show the isolated impact of Monodepth2's individual contributions on monocular self-supervised training.

Model	Auto-mask	Min. reproj.	Full-res m.s.	Pretrained	Abs Rel	Sq Rel	RMSE	$\delta < 1.25$
Baseline				✓	0.140	1.610	5.512	0.852
Baseline + min reproj.		✓		✓	0.122	1.081	5.116	0.866
Baseline + automasking	✓			✓	0.124	0.936	5.010	0.858
Baseline + full-res m.s.			✓	✓	0.124	1.170	5.249	0.865
Monodepth2 w/o min reproj.	✓		✓	✓	0.117	0.878	4.846	0.870
Monodepth2 w/o auto-masking		✓	✓	✓	0.120	1.097	5.074	0.872
Monodepth2 w/o full-res m.s.	✓	✓		✓	0.117	0.866	4.864	0.871
Monodepth2 with Zhou's mask	[mask]	✓	✓	✓	0.123	1.177	5.210	0.869
Monodepth2 (full)	✓	✓	✓	✓	0.115	0.903	4.863	0.877
Baseline w/o pt					0.150	1.585	5.671	0.827
Monodepth2 w/o pt	✓	✓	✓		0.132	1.044	5.142	0.845

Key takeaways from the ablation data include:

Each proposed component (minimum reprojection, auto-masking, full-resolution multi-scale) improves accuracy in isolation relative to the baseline.
Replacing auto-masking with Zhou et al.'s learned motion explanation mask degrades Abs Rel from 0.115 to 0.123 and increases RMSE from 4.863 to 5.210.
Pretraining on ImageNet reduces Abs Rel from 0.132 to 0.115, but Monodepth2 without pretraining still outperforms the pretrained baseline (0.132 vs 0.140).

Knowl 9 — Single-Scale Evaluation Protocol for Monocular Depth Estimation
model/method
Standard self-supervised monocular depth benchmarking rescales each predicted test depth map $D_{\text{pred}}^{(i)}$ independently by the ratio of its per-image median to the ground-truth median:

$s_i = \frac{\text{median}(D_{\text{gt}}^{(i)})}{\text{median}(D_{\text{pred}}^{(i)})}$

This per-image median scaling hides scale drift across frames and sequences, masking unstable scale estimation in both depth and visual odometry.

To evaluate scale stability fairly, the paper introduces a single-scale evaluation protocol: a single global scale factor $s^*$ is calculated by taking the median of all individual test set median ratios:

$s^* = \text{median}\left(\{s_1, s_2, \dots, s_N\}\right)$

Every predicted test depth map is multiplied by this single dataset-wide scale $s^*$ . Scale consistency is measured via the standard deviation of individual per-image scale factors, $\sigma_{\text{scale}}$ . On the KITTI Eigen split with improved ground truth, Monodepth2 achieves $\sigma_{\text{scale}} = 0.093$ (compared to Zhou et al.'s $0.210$ , DDVO's $0.108$ , and EPC++'s $0.123$ ), demonstrating higher scale consistency and stability across sequences.
Knowl 10 — Applicability of Test-Time Post-Processing to Monocular Depth Models
empirical result
Test-time post-processing is a technique where an input image $I$ is passed through the network alongside its horizontally flipped version $I_{\text{flip}}$ . The predicted disparity map for $I_{\text{flip}}$ is flipped back horizontally, and the two predictions are masked and averaged to generate the final disparity map:

$d_{\text{final}} = \frac{1}{2}\left(d(I) + \text{flip}\left(d(I_{\text{flip}})\right)\right)$

While previously applied exclusively to stereo-trained models to reduce left-right disparity artifacts, Monodepth2 demonstrates that test-time post-processing consistently improves quantitative performance for purely monocular ( $M$ ) and joint ( $MS$ ) self-supervised models.

For example, applying post-processing to Monodepth2 ( $M$ , $640 \times 192$ ) on the KITTI Eigen split reduces Abs Rel error from $0.115$ to $0.112$ , Sq Rel from $0.903$ to $0.851$ , and RMSE from $4.863$ to $4.754$ meters.
Knowl 11 — Limitations Induced by Photometric and Lambertian Assumptions
limitation
Because self-supervised training relies on photometric consistency ( $I_t \approx I_{t'\to t}$ ), the model fails in regions where underlying physical assumptions break down:
1. Non-Lambertian and Reflective Surfaces: Specular reflections, transparent windows, and wet surfaces violate brightness constancy, causing depth estimation failure.
2. Color Saturation and Distortion: Highly saturated, underexposed, or lens-distorted image areas provide incorrect photometric gradients, degrading depth predictions.
3. Complex Geometries and Thin Objects: Delineating fine, intricate structures (e.g., thin poles, tree branches) or objects with low boundary contrast remains challenging without explicit semantic or edge supervision.

Coverage note — None omitted; all core contributions, loss components (minimum reprojection, auto-masking, full-resolution multi-scale), architectures, training modes, ablation analyses, quantitative benchmark results, evaluation protocols, and limitations are fully covered.

References

1.Filippo Aleotti, Fabio Tosi, Matteo Poggi, and Stefano Mattoccia. Generative adversarial networks for unsupervised monocular depth prediction. In ECCV Workshops, 2018.
2.Amir Atapour-Abarghouei and Toby Breckon. Real-time monocular depth estimation using synthetic data with domain adaptation via image style transfer. In CVPR, 2018.
3.V Madhu Babu, Kaushik Das, Anima Majumdar, and Swagat Kumar. Undemon: Unsupervised deep network for depth and ego-motion estimation. In IROS, 2018.
4.Arunkumar Byravan and Dieter Fox. Se3-nets: Learning rigid body motion using deep neural networks. In ICRA, 2017.
5.Vincent Casser, Soeren Pirk, Reza Mahjourian, and Anelia Angelova. Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos. In AAAI, 2019.
6.Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-image depth perception in the wild. In NeurIPS, 2016.
7.Djork-Arne Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv, 2015.
8.David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In ICCV, 2015.
9.David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In NeurIPS, 2014.
10.Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In CVPR, 2018.
11.Yasutaka Furukawa and Carlos Hernández. Multi-view stereo: A tutorial. Foundations and Trends in Computer Graphics and Vision, 2015.
12.Ravi Garg, Vijay Kumar BG, and Ian Reid. Unsupervised CNN for single view depth estimation: Geometry to the rescue. In ECCV, 2016.
13.Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In CVPR, 2012.
14.Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
15.Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency. In CVPR, 2017.
16.Xiaoyang Guo, Hongsheng Li, Shuai Yi, Jimmy Ren, and Xiaogang Wang. Learning monocular depth by distilling cross-domain stereo networks. In ECCV, 2018.
17.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
18.Carol Barnes Hochberg and Julian E Hochberg. Familiar size and the perception of depth. The Journal of Psychology, 1952.
19.Derek Hoiem, Alexei A Efros, and Martial Hebert. Automatic photo pop-up. TOG, 2005.
20.Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. FlowNet2: Evolution of optical flow estimation with deep networks. In CVPR, 2017.
21.Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In NeurIPS, 2015.
22.Joel Janai, Fatma G¨uney, Anurag Ranjan, Michael Black, and Andreas Geiger. Unsupervised learning of multi-frame optical flow with occlusions. In ECCV, 2018.
23.Huaizu Jiang, Erik Learned-Miller, Gustav Larsson, Michael Maire, and Greg Shakhnarovich. Self-supervised relative depth learning for urban scene understanding. In ECCV, 2018.
24.Kevin Karsch, Ce Liu, and Sing Bing Kang. Depth transfer: Depth extraction from video using non-parametric sampling. PAMI, 2014.
25.Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. End-to-end learning of geometry and context for deep stereo regression. In ICCV, 2017.
26.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv, 2014.
27.KITTI Single Depth Evaluation Server. http://www.cvlibs.net/datasets/kitti/eval_depth.php?benchmark=depth_prediction. 2017.
28.Maria Klodt and Andrea Vedaldi. Supervising the new with the old: learning SFM from SFM. In ECCV, 2018.
29.Shu Kong and Charless Fowlkes. Pixel-wise attentional gating for parsimonious pixel labeling. arXiv, 2018.
30.Yevhen Kuznietsov, J¨org St¨uckler, and Bastian Leibe. Semisupervised deep learning for monocular depth map prediction. In CVPR, 2017.
31.Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 3DV, 2016.
32.Bo Li, Yuchao Dai, and Mingyi He. Monocular depth estimation with hierarchical fusion of dilated cnns and soft-weighted-sum inference. Pattern Recognition, 2018.
33.Ruihao Li, Sen Wang, Zhiqiang Long, and Dongbing Gu. UnDeepVO: Monocular visual odometry through unsupervised deep learning. arXiv, 2017.
34.Ruibo Li, Ke Xian, Chunhua Shen, Zhiguo Cao, Hao Lu, and Lingxiao Hang. Deep attention-based classification network for robust depth prediction. ACCV, 2018.
35.Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In CVPR, 2018.
36.Fayao Liu, Chunhua Shen, Guosheng Lin, and Ian Reid. Learning depth from single monocular images using deep convolutional neural fields. PAMI, 2015.
37.Miaomiao Liu, Mathieu Salzmann, and Xuming He. Discrete-continuous depth estimation from a single image. In CVPR, 2014.
38.Chenxu Luo, Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ram Nevatia, and Alan Yuille. Every pixel counts++: Joint learning of geometry and motion with 3D holistic understanding. arXiv, 2018.
39.Yue Luo, Jimmy Ren, Mude Lin, Jiahao Pang, Wenxiu Sun, Hongsheng Li, and Liang Lin. Single view stereo matching. In CVPR, 2018.
40.Reza Mahjourian, Martin Wicke, and Anelia Angelova. Unsupervised learning of depth and ego-motion from monocular video using 3D geometric constraints. In CVPR, 2018.
41.Nikolaus Mayer, Eddy Ilg, Philipp Fischer, Caner Hazirbas, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. What makes good synthetic training data for learning disparity and optical flow estimation? IJCV, 2018.
42.Nikolaus Mayer, Eddy Ilg, Philip H¨ausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In CVPR, 2016.
43.Ishit Mehta, Parikshit Sakurikar, and PJ Narayanan. Structured adversarial training for unsupervised monocular depth estimation. In 3DV, 2018.
44.Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. ORB-SLAM: a versatile and accurate monocular SLAM system. Transactions on Robotics, 2015.
45.Jogendra Nath Kundu, Phani Krishna Uppala, Anuj Pahuja, and R. Venkatesh Babu. AdaDepth: Unsupervised content congruent adaptation for depth estimation. In CVPR, 2018.
46.Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NeurIPS-W, 2017.
47.Sudeep Pillai, Rares Ambrus, and Adrien Gaidon. Superdepth: Self-supervised, super-resolved monocular depth estimation. In ICRA, 2019.
48.Andrea Pilzer, Dan Xu, Mihai Marian Puscas, Elisa Ricci, and Nicu Sebe. Unsupervised adversarial depth estimation using cycled generative networks. In 3DV, 2018.
49.Matteo Poggi, Filippo Aleotti, Fabio Tosi, and Stefano Mattoccia. Towards real-time unsupervised monocular depth estimation on cpu. In IROS, 2018.
50.Matteo Poggi, Fabio Tosi, and Stefano Mattoccia. Learning monocular depth estimation with unsupervised trinocular assumptions. In 3DV, 2018.
51.Anurag Ranjan, Varun Jampani, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black. Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. In CVPR, 2019.
52.Zhe Ren, Junchi Yan, Bingbing Ni, Bin Liu, Xiaokang Yang, and Hongyuan Zha. Unsupervised deep learning for optical flow estimation. In AAAI, 2017.
53.Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
54.Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
55.Ashutosh Saxena, Min Sun, and Andrew Ng. Make3d: Learning 3d scene structure from a single still image. PAMI, 2009.
56.Daniel Scharstein and Richard Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV, 2002.
57.Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
58.Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In CVPR, 2018.
59.Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger. Sparsity invariant CNNs. In 3DV, 2017.
60.Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, and Thomas Brox. DeMoN: Depth and motion network for learning monocular stereo. In CVPR, 2017.
61.Sudheendra Vijayanarasimhan, Susanna Ricco, Cordelia Schmid, Rahul Sukthankar, and Katerina Fragkiadaki. SfM-Net: Learning of structure and motion from video. arXiv, 2017.
62.Chaoyang Wang, Jose Miguel Buenaposada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using direct methods. In CVPR, 2018.
63.Yang Wang, Yi Yang, Zhenheng Yang, Liang Zhao, and Wei Xu. Occlusion aware unsupervised learning of optical flow. In CVPR, 2018.
64.Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 2004.
65.Jamie Watson, Michael Firman, Gabriel J Brostow, and Daniyar Turmukhambetov. Self-supervised monocular depth hints. In ICCV, 2019.
66.Yiran Wu, Sihao Ying, and Lianmin Zheng. Size-to-depth: A new perspective for single image depth estimation. arXiv, 2018.
67.Junyuan Xie, Ross Girshick, and Ali Farhadi. Deep3D: Fully automatic 2D-to-3D video conversion with deep convolutional neural networks. In ECCV, 2016.
68.Nan Yang, Rui Wang, J¨org St¨uckler, and Daniel Cremers. Deep virtual stereo odometry: Leveraging deep depth prediction for monocular direct sparse odometry. In ECCV, 2018.
69.Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, and Ram Nevatia. LEGO: Learning edge with geometry all at once by watching videos. In CVPR, 2018.
70.Zhenheng Yang, Peng Wang, Wei Xu, Liang Zhao, and Ramakant Nevatia. Unsupervised learning of geometry with edge-aware depth-normal consistency. In AAAI, 2018.
71.Zhichao Yin and Jianping Shi. GeoNet: Unsupervised learning of dense depth, optical flow and camera pose. In CVPR, 2018.
72.Jure ˇZbontar and Yann LeCun. Stereo matching by training a convolutional neural network to compare image patches. JMLR, 2016.
73.Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In CVPR, 2018.
74.Zhenyu Zhang, Chunyan Xu, Jian Yang, Ying Tai, and Liang Chen. Deep hierarchical guidance and regularization learning for end-to-end depth estimation. Pattern Recognition, 2018.
75.Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. Loss functions for image restoration with neural networks. Transactions on Computational Imaging, 2017.
76.Tinghui Zhou, Matthew Brown, Noah Snavely, and David Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, 2017.
77.Daniel Zoran, Phillip Isola, Dilip Krishnan, and William T Freeman. Learning ordinal relationships for mid-level vision. In ICCV, 2015.
78.Yuliang Zou, Zelun Luo, and Jia-Bin Huang. DF-Net: Unsupervised joint learning of depth and flow using cross-task consistency. In ECCV, 2018.

Citation

MLA

Godard, C., et al. “Digging Into Self-Supervised Monocular Depth Estimation”. arXiv, 2018, http://arxiv.org/abs/1806.01260v4.

APA

Godard, C., Aodha, O. M., Firman, M., & Brostow, G. (2018). Digging Into Self-Supervised Monocular Depth Estimation. arXiv. http://arxiv.org/abs/1806.01260v4

Chicago

Godard, C., O. M. Aodha, M. Firman, and G. Brostow. 2018. “Digging Into Self-Supervised Monocular Depth Estimation”. arXiv. http://arxiv.org/abs/1806.01260v4.

Harvard

Godard, C. et al. (2018) “Digging Into Self-Supervised Monocular Depth Estimation”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1806.01260v4.

Vancouver

1. Godard C, Aodha OM, Firman M, Brostow G (2018) Digging Into Self-Supervised Monocular Depth Estimation. arXiv

BibTeX

@article{godard2018digging,
  title = {Digging Into Self-Supervised Monocular Depth Estimation},
  author = {Godard, Clément and Aodha, Oisin Mac and Firman, Michael and Brostow, Gabriel},
  year = {2018},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1806.01260v4},
  eprint = {1806.01260}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE

Digging Into Self-Supervised Monocular Depth Estimation

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Per-Pixel Minimum Reprojection Loss for Occlusion Handling

Knowl 2 — Auto-Masking Stationary Pixels in Monocular Self-Supervised Depth Estimation

Knowl 3 — Full-Resolution Multi-Scale Appearance Matching Strategy

Knowl 4 — Photometric and Smoothness Loss Formulation in Monodepth2

Knowl 5 — Monodepth2 Depth and Pose Network Architectures

Knowl 6 — Supervision Regimes: Monocular (M), Stereo (S), and Joint (MS)