The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes
Germán RosLaura SellartJoanna MaterzynskaDavid VázquezAntonio M. López
Introduces a large-scale synthetic dataset of urban driving scenes with automatic pixel-level annotations, demonstrating that joint training on synthetic and real imagery significantly improves deep semantic segmentation accuracy across real-world benchmarks.
The article addresses the challenge of obtaining large volumes of pixel-level annotated images needed to train deep convolutional neural networks for semantic segmentation in urban driving scenes. Manual annotation is costly and time-consuming, limiting the scale and diversity of real-world datasets available for autonomous driving systems.
The article set out to determine whether realistic synthetic images generated in a virtual city could usefully supplement or replace real images when training such networks, and whether combining the two domains would improve segmentation accuracy on real test data.
The authors created the SYNTHIA dataset containing more than 213,400 synthetic frames rendered from a virtual urban environment, complete with automatic pixel-level labels for 13 classes and varying seasons, lighting, viewpoints, and dynamic objects. They trained two convolutional architectures on SYNTHIA alone and on balanced batches that mixed SYNTHIA with four existing real datasets, then measured performance on held-out real validation images.
Training solely on SYNTHIA produced reasonable accuracy on real test sets, sometimes matching or exceeding models trained on the smaller real datasets alone. Adding SYNTHIA to real training data raised average per-class accuracy by 7 to 18 points across the tested datasets and architectures, with the largest gains for pedestrians, cars, and cyclists. Global pixel accuracy also improved in most cases.
These results indicate that large-scale synthetic data can materially reduce reliance on expensive manual labeling while boosting the reliability of perception systems for autonomous driving. The approach offers a practical route to greater diversity in training data without proportional increases in cost or time.
Further work should focus on refining domain-adaptation techniques, testing at higher image resolutions to better capture small objects such as signs and poles, and extending the virtual environment to additional cities, weather conditions, and sensor configurations. The current experiments were conducted at low resolution, which limits recognition of fine details and may affect the generalizability of the reported gains.
- Paper: Fully convolutional networks for semantic segmentation, Jonathan Long et al. (2015). This seminal work introduced end-to-end fully convolutional networks for pixel-level semantic segmentation, establishing the core architectural foundation utilized and evaluated throughout the SYNTHIA dataset paper.
- Paper: SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation, Vijay Badrinarayanan et al. (2015). This paper establishes the SegNet deep convolutional encoder-decoder architecture, one of the primary baseline models evaluated to demonstrate the efficacy of synthetic training data in SYNTHIA.
- Paper: Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs, Liang-Chieh Chen et al. (2014). This research introduces DeepLab with atrous convolution for dense semantic segmentation, providing an essential model framework and baseline benchmark methodology for urban scene parsing.
- Book: Domain-Adversarial Training of Neural Networks, Yaroslav Ganin et al. (2016). This work formulates the fundamental principles of domain-adversarial training for bridging synthetic-to-real domain gaps, directly informing the cross-domain generalization motivations of SYNTHIA.
- Paper: Playing for Data: Ground Truth from Computer Games, Stephan R. Richter et al. (2016). This paper extends the synthetic data paradigm introduced by datasets like SYNTHIA by extracting dense pixel-level ground truth directly from commercial computer game graphics pipelines.
- Paper: Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks, Jun-Yan Zhu et al. (2017). CycleGAN provides an unpaired image-to-image translation methodology essential for closing the synthetic-to-real domain gap highlighted as an open challenge in the SYNTHIA study.
- Paper: Deep Visual Domain Adaptation: A Survey, Mei Wang et al. (2018). This survey provides a comprehensive synthesis of deep visual domain adaptation techniques, addressing the core challenges of transferring representations from synthetic environments to real-world target domains.
- Paper: High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs, Ting-Chun Wang et al. (2018). This work advances conditional generative modeling by synthesizing high-resolution, photorealistic urban scenes from semantic label maps, directly expanding upon synthetic data generation for visual perception.
- Paper: AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles, Shital Shah et al. (2017). AirSim develops a high-fidelity visual and physical simulation platform for autonomous vehicles, realizing an open-source virtual environment to generate rich synthetic training data alongside realistic dynamics.
- Paper: SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences, Jens Behley et al. (2019). SemanticKITTI expands the scope of driving scene understanding benchmarks from 2D RGB synthetic data to large-scale, densely labeled sequential 3D LiDAR point clouds.
- Paper: BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning, Fisher Yu et al. (2020). BDD100K extends urban perception benchmarks by supplying a massive, highly diverse real-world driving dataset to evaluate multitask learning and evaluate synthetic-to-real transfer across extensive environmental conditions.
