Joint bilateral upsampling
Johannes KopfMichael F. CohenDani LischinskiMatt Uyttendaele
Proposes a fast edge-preserving method that uses high-resolution guide images to accurately upsample low-resolution computational solutions in tone mapping, stereo depth, colorization, and segmentation tasks.
Modern digital image processing tasks—such as high dynamic range tone mapping, colorization, stereo depth estimation, and image compositing—increasingly handle multi-megapixel and gigapixel images. Computing complex pixel-level solutions at full resolution often requires excessive memory and processing time, leading to memory thrashing and slow performance. Downsampling images to compute solutions quickly at lower resolutions provides a workaround, but standard upsampling methods rely on basic smoothness assumptions that blur sharp boundaries, create halo artifacts, and cause color or exposure bleeding across object edges.
The article demonstrates and evaluates Joint Bilateral Upsampling, a method that uses the original high-resolution input image as a structural guide to accurately reconstruct high-resolution solutions from heavily downsampled computations.
The approach applies a joint bilateral filter across two different image resolutions simultaneously. It combines a spatial filter applied to the low-resolution computed solution with a range filter evaluated on the original high-resolution reference image. The evaluation tested this method against standard interpolation techniques—including nearest neighbor, bicubic, and Gaussian filters—across four diverse visual computing tasks downsampled by factors ranging from 2x2 to 32x32: exposure maps for tone mapping, chrominance channels for colorization, depth maps for stereo correspondence, and discrete label maps for graph-cut image stitching.
The key findings demonstrate that Joint Bilateral Upsampling consistently outperforms traditional interpolation methods, delivering results that are numerically closer to ground truth and visually sharper across all test cases. The relative performance advantage over traditional methods widens as the downsampling factor increases, effectively preventing edge bleeding, halos, and blocking artifacts. In terms of efficiency, execution time scales purely with output resolution and domain filter radius rather than the downsampling factor, requiring only about 2 seconds per output megapixel. This enables massive runtime reductions; for instance, tone mapping a 3.1-megapixel image required only 6 seconds using this approach compared to 80 seconds at full resolution, while colorization solvers that normally require several minutes run in seconds. Furthermore, because the operation is strictly local, it maintains a minimal memory footprint and processes massive multi-gigapixel images in a single streaming sweep without memory thrashing.
These findings mean organizations and developers can implement high-quality, interactive image processing tools on large visual datasets without investing in costly computational hardware or exceeding memory limits. The approach allows high-complexity algorithms to run on small representations while still delivering crisp, publication-quality full-resolution outputs, fundamentally altering the performance-versus-quality trade-off for resource-intensive graphic workflows.
Teams developing image analysis, photography, or rendering pipelines should adopt this upsampling technique to accelerate processing times and support very high-resolution workflows. When deploying the method, practitioners should tailor the range filter parameters to the specific application—such as using the standard deviation of luminance for unbounded exposure values or fixed color-distance thresholds for normalized colorization and depth data. As a next step, research and engineering teams can explore extending this guided upsampling framework to other computational domains, such as upsampling global illumination solutions from coarse geometric 3D meshes to fine surfaces.
Confidence in these findings is high for images where solution discontinuities align with visible edges in the guide image. However, practitioners should note that reconstruction quality naturally decreases at extreme downsampling factors, and low-resolution inputs must retain sufficient detail to support user-guided interactions or coarse algorithmic solvers. In addition, applications with non-continuous solutions, such as discrete graph-cut labeling, require voting schemes rather than direct interpolation.
- Paper: Gradient domain high dynamic range compression, Raanan Fattal et al. (2002). It introduces foundational multi-scale gradient attenuation methods for tone mapping high dynamic range scenes, which joint bilateral upsampling accelerates by computing on downsampled inputs.
- Paper: Poisson image editing, Patrick Pérez et al. (2003). It establishes gradient-domain Poisson image editing and guided interpolation principles that directly motivate edge-aware, guided upsampling for image enhancement and compositing tasks.
- Paper: Stereo Matching Using Belief Propagation, Jian Sun et al. (2002). It formulates dense stereo disparity estimation over pixel grids, representing the core computational bottleneck that joint bilateral upsampling solves by upsampling coarse disparity solutions using high-resolution color guidance.
- Paper: Mean Shift: A Robust Approach Toward Feature Space Analysis, Dorin Comaniciu et al. (2002). It provides the foundational joint spatial-range feature space formulation that underpins edge-preserving and bilateral filtering concepts.
- Paper: Image inpainting, Marcelo Bertalmio et al. (2000). It develops early boundary-preserving propagation techniques across pixel grids that provide background for guided image-completion and solution-upsampling tasks.
- Paper: Guided Image Filtering, Kaiming He et al. (2010). It introduces guided image filtering as a computationally faster, linear-time edge-preserving alternative that avoids the gradient-reversal artifacts seen in bilateral and joint bilateral upsampling.
- Paper: Image smoothing via L0 gradient minimization, Li Xu et al. (2011). It advances beyond local joint filtering by developing a global L0-norm gradient minimization technique for edge-preserving smoothing and artifact-free tone mapping.
- Paper: PatchMatch: a randomized correspondence algorithm for structural image editing, Connelly Barnes et al. (2009). It presents a fast randomized patch-matching algorithm that supersedes traditional pixel-grid guided sampling for complex structural image editing and retargeting.
