DT-R2S2R

Digital Twin-Driven Real2Sim2Real:
Simulator-Conditioned Generation via Paired Driving-Scene Reconstruction

Anonymous Authors
Anonymous Institution  ·  Under Review
Paper (Coming Soon) Datasets (Coming Soon) Demo: Method Demo: Long-tail Aug. Results Appendix
Overview of DT-R2S2R: real driving clips are reconstructed inside a georeferenced digital twin, a Sim2Real model is trained on the geometrically paired data, and the trained model then converts newly simulated scenes into photorealistic annotated training data.
Overview of DT-R2S2R. A Sim2Real model (DT-S2R) is trained on real driving images together with their geometrically paired digital-twin reconstructions (DT-R2S). The trained model then takes newly generated simulator data and produces photorealistic images that carry the simulator's annotations at no additional cost, which are used to train 3D detectors.

Abstract

Camera-based 3D perception for autonomous driving relies heavily on large annotated datasets, and deploying such a system to a new target region typically requires data collection and annotation. Generative augmentation has been proposed to reduce this cost, but existing approaches face a fundamental trade-off: label-conditioned methods consume the very annotations they aim to replace, while simulator-conditioned methods offer free annotations but lack visual grounding to specific real environments.

This work investigates the extent to which a digital-twin-driven Real2Sim2Real pipeline (DT-R2S2R) can substitute for target-region real data. By reconstructing recorded driving clips inside a georeferenced digital twin (DT-R2S), we condition a diffusion model on geometrically aligned simulator renderings, establishing a digital twin-grounded Sim2Real model (DT-S2R). As a result, DT-S2R synthesizes photorealistic driving images given low-cost yet georeferenced simulator data across both reconstructed and novel simulator scenes within digital-twin coverage.

The efficacy of generated data is verified on diverse 3D detectors. DETR3D, especially, reports 93.18% of mAP obtained by a target-region real-data oracle, without employing target images for detector training. Furthermore, simple co-training with existing out-of-target real data surpasses that oracle. Thus, DT-R2S2R can substantially alleviate the cost of manual on-site data collection and annotation for digital twin-available districts, offering a practical foundation for scaling 3D perception.

Key Contributions

01

Scene-level Real2Sim from a digital twin

Prior Real2Sim2Real pipelines reconstruct the dynamic agents and a coarse road layout. We reconstruct the entire scene inside a georeferenced twin — road network, buildings, terrain and dimension-matched vehicle assets — so that the generator receives the actual location as a condition rather than having to synthesize it.

02

The DT-rendered RGB can be a useful proxy for Sim2Real

Unlike depth and semantics, the twin's rendered RGB specifies what the location itself looks like — its road network, building facades and matched vehicle assets. Since that signal is already present in the condition, the generative backbone requires no architectural modification and remains frozen.

03

A labeled data engine, not an image translator

With zero target-region real images, DETR3D attains 93.18% of the real-data oracle. Co-training with out-of-target real data exceeds that oracle by +2.19 mAP.

Approach

The DT-R2S2R pipeline. (a) Real2Sim locates a clip inside the digital twin, replaces each annotated object with the closest library asset and recovers the sensor poses. (b) Sim2Real drives one ControlNet branch per rendered modality into a frozen SDXL backbone, with cross-view attention inside the branches.
The DT-R2S2R pipeline. (a) DT-R2S locates a clip inside the twin, replaces each annotated object with the closest library asset, and recovers the sensor poses. (b) DT-S2R drives one ControlNet branch per rendered modality — RGB, semantics and depth — into a frozen SDXL backbone, with cross-view attention placed inside the branches rather than in the backbone.

(a) DT-R2S — Paired driving-scene reconstruction

A recorded clip reproduced inside the digital twin. Top: the recording. Middle: the twin reconstruction rendered from the recovered sensor poses, each annotated vehicle replaced by its closest library asset. Bottom: the semantic annotation the simulator exports at no additional cost.

(b) DT-S2R — Photorealism enhancement

Frame-wise image translation across a scene. A georeferenced image is generated at every timestep, conditioned on the digital twin. Top: the real recording. Middle: the digital-twin rendering the generator is conditioned on. Bottom: the DT-S2R output.
Frame-wise image translation across a scene. A georeferenced image is generated at every timestep, conditioned on the digital twin. Top: the real recording. Middle: the digital-twin rendering the generator is conditioned on. Bottom: the DT-S2R output.
Frame-wise image translation across a scene. A georeferenced image is generated at every timestep, conditioned on the digital twin. Top: the real recording. Middle: the digital-twin rendering the generator is conditioned on. Bottom: the DT-S2R output.
Frame-wise image translation across a scene. A georeferenced image is generated at every timestep, conditioned on the digital twin. Top: the real recording. Middle: the digital-twin rendering the generator is conditioned on. Bottom: the DT-S2R output.
Frame-wise image translation across a scene. A georeferenced image is generated at every timestep, conditioned on the digital twin. Top: the real recording. Middle: the digital-twin rendering the generator is conditioned on. Bottom: the DT-S2R output.

Generation Beyond the Real Recordings

Dataset comparison (AI-Hub & ADC25)

One real and two synthetic datasets. The real set is an AI-Hub Korean urban driving dataset used to construct DT-R2S and train DT-S2R. See Appendix A for details. ADC25 is a simulation-only dataset that took no part in building the pipeline, meaning its frames do not correspond to any real recordings. Note that its scenarios originate from the Hyundai Motor Group Autonomous Driving Challenge 2025. ADC25 is used to evaluate whether DT-S2R generalizes to unseen routes and scenarios, and to generate additional training data for 3D detectors.

NotationDatasetDomainIn twinDT-S2R District Train
clips, frames		 Val
clips, frames
\(D^{\text{Real}}_{\text{OffTwin}}\)A AI-Hub (Off-Twin)Real✗N/A Seoul, Seongnam, Sejong, Daegu262, 47K66, 11K
\(D^{\text{Real}}_{\text{InTwin}}\)A AI-Hub (In-Twin)Real✓N/A Sangam, Pangyo134, 24K34, 6K
\(D^{\text{Real2Sim}}_{\text{InTwin}}\) DT-R2SSynthetic✓✗ Sangam, Pangyo134, 24K34, 6K
\(D^{\text{Sim2Real}}_{\text{InTwin}}\) DT-S2RSynthetic✓✓ Sangam, Pangyo134, 24K34, 6K
\(D^{\text{Sim}}_{\text{ADC25}}\) ADC25Synthetic✓✗ Sangam, Pangyo241, 28K—
\(D^{\text{Sim2Real}}_{\text{ADC25}}\) ADC25 (DT-S2R)Synthetic✓✓ Sangam, Pangyo241, 28K—

Route comparison of ADC25 against AI-Hub

The recordings visit only part of the twin. These interactive maps show which streets each source traverses and where they overlap — the disjoint part is what tests whether DT-S2R generalizes to streets it never saw.

DT-S2R results on ADC25

Top: Synthetic RGB images rendered based on ADC scenarios. Bottom: DT-S2R applied to the renderings. These routes were never recorded, so every frame here is conditioned on the twin alone.
Top: Synthetic RGB images rendered based on ADC scenarios. Bottom: DT-S2R applied to the renderings. These routes were never recorded, so every frame here is conditioned on the twin alone.
Top: Synthetic RGB images rendered based on ADC scenarios. Bottom: DT-S2R applied to the renderings. These routes were never recorded, so every frame here is conditioned on the twin alone.
Top: Synthetic RGB images rendered based on ADC scenarios. Bottom: DT-S2R applied to the renderings. These routes were never recorded, so every frame here is conditioned on the twin alone.
Top: Synthetic RGB images rendered based on ADC scenarios. Bottom: DT-S2R applied to the renderings. These routes were never recorded, so every frame here is conditioned on the twin alone.
Top: Synthetic RGB images rendered based on ADC scenarios. Bottom: DT-S2R applied to the renderings. These routes were never recorded, so every frame here is conditioned on the twin alone.
Top: Synthetic RGB images rendered based on ADC scenarios. Bottom: DT-S2R applied to the renderings. These routes were never recorded, so every frame here is conditioned on the twin alone.
Generation on foreseen and unforeseen routes within the digital-twin coverage of Pangyo, with a map showing the overlap between the real dataset routes and the ADC25 scenario routes.
Generation beyond the recorded routes. Roughly 53% of the ADC25 clips in Pangyo are spatially disjoint from the recordings. Buildings and road-network detail survive translation, and cross-view consistency holds on routes absent from training.

Quantitative Results

Deploying to a target region without target real images

Training data DETR3D PETR
mAP ↑NDS ↑30–50m ↑ mAP ↑NDS ↑30–50m ↑
\(D^{\text{Real}}_{\text{InTwin}}\) Oracle 48.8354.3929.86 55.2358.1037.94
\(D^{\text{Real}}_{\text{OffTwin}}\) Practical baseline 39.9848.9923.56 50.5955.3532.31
\(D^{\text{Sim2Real}}_{\text{ADC25}}\)no real data 29.3939.8615.25 34.4141.2917.96
\(D^{\text{Sim2Real}}_{\text{InTwin}} + D^{\text{Sim2Real}}_{\text{ADC25}}\) no real data 45.5052.2127.45 49.0753.6930.87
\(D^{\text{Sim2Real}}_{\text{InTwin}} + D^{\text{Sim2Real}}_{\text{ADC25}} + D^{\text{Real}}_{\text{OffTwin}}\) Co-training 51.0255.7733.55 55.7957.9537.57
Co-training vs. Oracle +2.19+1.38+3.69 +0.56−0.15−0.37
Co-training vs. Practical baseline +11.04+6.78+9.99 +5.20+2.60+5.26

Preliminary Attempt at Long-Tail Augmentation

Editing inside the twin rather than in image space lets a rare maneuver be composed onto a recorded scene and translated back — with the 3D annotation already attached, and without waiting for the event on the road.

Edge-case scenario augmentation: a real clip is reproduced in the digital twin, a normally-behaving car is edited to overtake and stop in front of the ego vehicle, and the result is translated back to photorealistic images.
A car exhibiting nominal behavior (red box) is edited to overtake and stop in front of the ego vehicle, while the original scenery is preserved (orange box).
The inserted maneuver stays consistent across all five views and across time.

Appendix

Appendix A: Customized Splits of AI-Hub

A.1  Why the official split cannot be used

The official AI-Hub split partitions each driving clip into training and validation frames (frame-wise). Because consecutive frames of one clip depict almost the same scene, such a partition would allow the Sim2Real generator to observe validation content during training, and would inflate detector scores for the same reason. We therefore repartition the data clip-wise, so that all frames of the same clip fall on one side of the split. Note that clips are divided such that the dynamic objects are distributed fairly in a polar coordinate system centered on the ego vehicle, keeping the range and azimuth distribution of annotated objects comparable across the train and validation sets.

A.2  Resulting splits

Every clip is labeled in-twin or off-twin by testing its GNSS trajectory against the digital-twin district hulls, yielding the following subsets.

SubsetNotationDistricts Train
clips, frames		 Val
clips, frames
In-twin\(D^{\text{Real}}_{\text{InTwin}}\) Sangam, Pangyo134, 24K34, 6K
Off-twin\(D^{\text{Real}}_{\text{OffTwin}}\) Seoul, Seongnam, Sejong, Daegu262, 47K66, 11K
Combined\(D^{\text{Real}}_{\text{All}}\) all of the above396, 71K100, 17K

Each clip is recorded with five surround-view cameras and one 128-channel LiDAR. Only the in-twin subset has a digital-twin counterpart and therefore participates in building the DT-R2S2R pipeline. The off-twin subset serves as the out-of-target real data.

Eight-panel comparison of the train and validation splits: location class, background panoptic composition, object distance, azimuth, orientation, per-split polar occupancy maps, and target class distribution.
The figure above depicts the train and validation sets after the repartition. The two distributions show comparable statistics in terms of object distance, azimuth, orientation and class balance, including the ego-centered polar maps.

A.3  Clip-level mapping

All 168 (train: 134, val: 34) in-twin clips carry 180 frames each, giving 24,120 training and 6,120 validation frames. The list below records which side of our split each clip lies on.

Full clip list — 168 clips
DistrictClipFramesSplit
PangyoClip_000180Val
PangyoClip_001180Train
PangyoClip_002180Val
PangyoClip_003180Train
PangyoClip_004180Train
PangyoClip_005180Train
PangyoClip_006180Train
PangyoClip_007180Train
PangyoClip_008180Train
PangyoClip_009180Val
PangyoClip_010180Train
PangyoClip_011180Val
PangyoClip_012180Train
PangyoClip_013180Train
PangyoClip_014180Train
PangyoClip_015180Train
PangyoClip_016180Train
PangyoClip_017180Train
PangyoClip_023180Train
PangyoClip_024180Train
PangyoClip_025180Train
PangyoClip_026180Train
PangyoClip_027180Train
PangyoClip_028180Train
PangyoClip_029180Train
PangyoClip_030180Val
PangyoClip_031180Train
PangyoClip_032180Train
PangyoClip_033180Val
PangyoClip_034180Train
PangyoClip_036180Train
PangyoClip_037180Train
PangyoClip_038180Train
PangyoClip_039180Val
PangyoClip_040180Train
PangyoClip_041180Train
PangyoClip_042180Train
PangyoClip_043180Train
PangyoClip_044180Train
PangyoClip_045180Train
PangyoClip_046180Val
PangyoClip_047180Train
PangyoClip_048180Train
PangyoClip_049180Train
PangyoClip_050180Train
PangyoClip_051180Train
PangyoClip_052180Train
PangyoClip_056180Train
PangyoClip_058180Train
PangyoClip_059180Train
PangyoClip_060180Train
PangyoClip_147180Train
PangyoClip_148180Val
PangyoClip_149180Train
PangyoClip_150180Train
PangyoClip_151180Train
PangyoClip_152180Train
PangyoClip_153180Train
PangyoClip_154180Val
PangyoClip_155180Val
PangyoClip_156180Train
PangyoClip_157180Train
PangyoClip_158180Train
PangyoClip_159180Val
PangyoClip_160180Train
PangyoClip_161180Train
PangyoClip_162180Train
PangyoClip_163180Train
PangyoClip_164180Train
PangyoClip_165180Train
PangyoClip_169180Train
PangyoClip_170180Train
PangyoClip_171180Train
PangyoClip_172180Val
PangyoClip_238180Val
PangyoClip_239180Train
PangyoClip_240180Train
SangamClip_061180Train
SangamClip_062180Train
SangamClip_063180Val
SangamClip_064180Train
SangamClip_065180Train
SangamClip_066180Train
SangamClip_067180Train
SangamClip_068180Val
SangamClip_069180Val
SangamClip_070180Val
SangamClip_071180Train
SangamClip_072180Val
SangamClip_073180Val
SangamClip_074180Train
SangamClip_075180Val
SangamClip_076180Val
SangamClip_077180Val
SangamClip_078180Val
SangamClip_079180Val
SangamClip_080180Train
SangamClip_081180Train
SangamClip_082180Val
SangamClip_083180Train
SangamClip_084180Val
SangamClip_085180Train
SangamClip_086180Val
SangamClip_087180Train
SangamClip_088180Train
SangamClip_089180Train
SangamClip_090180Val
SangamClip_091180Train
SangamClip_092180Train
SangamClip_093180Val
SangamClip_094180Train
SangamClip_095180Train
SangamClip_096180Train
SangamClip_097180Train
SangamClip_098180Train
SangamClip_099180Train
SangamClip_100180Train
SangamClip_101180Val
SangamClip_102180Train
SangamClip_103180Train
SangamClip_104180Train
SangamClip_105180Train
SangamClip_106180Train
SangamClip_107180Val
SangamClip_108180Train
SangamClip_109180Train
SangamClip_110180Train
SangamClip_111180Train
SangamClip_112180Train
SangamClip_113180Train
SangamClip_114180Train
SangamClip_115180Train
SangamClip_116180Val
SangamClip_117180Train
SangamClip_118180Train
SangamClip_119180Train
SangamClip_120180Train
SangamClip_121180Train
SangamClip_122180Train
SangamClip_123180Train
SangamClip_124180Val
SangamClip_125180Train
SangamClip_126180Train
SangamClip_127180Train
SangamClip_128180Train
SangamClip_129180Train
SangamClip_130180Train
SangamClip_131180Train
SangamClip_132180Train
SangamClip_133180Train
SangamClip_134180Train
SangamClip_135180Train
SangamClip_136180Train
SangamClip_137180Train
SangamClip_138180Train
SangamClip_139180Train
SangamClip_140180Train
SangamClip_141180Train
SangamClip_142180Train
SangamClip_143180Train
SangamClip_144180Train
SangamClip_145180Train
SangamClip_146180Train
SangamClip_232180Train
SangamClip_233180Train
SangamClip_243180Train
SangamClip_244180Train
SangamClip_245180Train
Appendix B: Reconstruction Details for DT-R2S

B.1  Object-to-asset mapping

Assets are ranked against the per-axis maximum box size by a weighted cost over the three dimensions. Each axis is weighted by its normalized coefficient of variation, so the more discriminative dimensions dominate the match.

AxisWeightRole
Height0.41most discriminative across the catalogue
Length0.35
Width0.24least discriminative — vehicle widths cluster tightly

The top-5 candidates by this cost are then re-ranked by appearance: a SAM-segmented crop of the object and each candidate asset are encoded with DINOv3, and the highest cosine similarity wins.

B.2  Sensor pose optimization

The LiDAR-to-ego transform is recovered with CMA-ES over the 8-corner box-matching objective. The initial pose comes from the dataset's LiDAR-to-GNSS extrinsics, and a RANSAC ground-plane fit on the real point cloud fixes the plausible margin for the mounting height. Every bound below is therefore an offset from that initial pose, and \(\sigma_0\) is the initial step size that sets how widely candidates are sampled around the current mean.

AxisInitial \(\sigma_0\)Search bound offset from init
x0.025 m±0.05 m
y0.025 m±0.05 m
z0.025 m±0.05 m
roll0.0175 rad (≈1°)±0.0523 rad (≈±3°)
pitch0.0175 rad (≈1°)±0.0523 rad (≈±3°)
yaw0.00875 rad (≈0.5°)±0.0175 rad (≈±1°)

Appendix C: Setup of the Compared Injection Mechanisms for DT-S2R

The simulator-rendered RGB carries a strong visual signal of what its real counterpart looks like. While designing the conditioning path we therefore compared three ways of injecting it into the diffusion backbone (SDXL). All three are built on the same Stage 1 configuration, so the comparison isolates the injection mechanism rather than the backbone. A dash marks a component that configuration does not have.

Hyperparameter Ours Stage 1 SDEdit T2I-Adapter
Way of using each condition
Simulator RGB residual ControlNet branch placed in the initial latent at \(t_0 = 0.5\) lightweight adapter
Semantics residual ControlNet branch residual ControlNet branch lightweight adapter
Depth residual ControlNet branch residual ControlNet branch lightweight adapter
Residual injection point summed per branch, then added to the UNet as down_block_additional_residuals and mid_block_additional_residual adapter features added to the encoder
Conditioning scale \(s_k\) [1.0, 1.0, 1.0] for RGB, semantic and depth — no modality is down-weighted a priori
Pre-trained initialization, per condition
Simulator RGB xinsir/controlnet-tile-sdxl-1.0 — (no trainable module) from scratch
Semantics from scratch from scratch from scratch
Depth diffusers/controlnet-depth-sdxl-1.0 diffusers/controlnet-depth-sdxl-1.0 TencentARC/t2i-adapter-depth-midas-sdxl-1.0
Optimization
Learning rate 5×10−5 5×10−5 1×10−5
OptimizerAdamW, weight decay 0.01
Scheduleconstant, 1,000 warm-up steps
Gradient clipping1.0
Optimization steps50,000
Precisionmixed
Batch per device2 frames × 5 cameras
Diffusion Backbone (Frozen)
Backbone (UNet)stabilityai/stable-diffusion-xl-base-1.0
VAEmadebyollin/sdxl-vae-fp16-fix
Text encodersCLIP ViT-L and OpenCLIP ViT-bigG, bundled with the SDXL base checkpoint
Shared protocol
Resolution576 × 1024
ObjectiveMSE + \(\lambda\) × depth-weighted MSE, \(\lambda = 0.1\), minimum depth clamp 0.02
Horizontal flipp = 0.5, applied jointly to the views and all conditions
Photometric jitterbrightness, contrast, saturation 0.1 each — simulator RGB only, the real target is left untouched
Condition dropoutdepth and semantics dropped independently at p = 0.1. Simulator RGB is never dropped
Text prompthighly detailed realistic driving scene in {district}, South Korea, dropped with p = 0.1
SamplingDDPM scheduler, 50 steps, guidance scale 5.0
Stage 2, applied on top of Ours only
Cross-view attention mid_block, down_block_2, down_block_1 of each branch. For each camera, the attention is applied to its two neighbors (left and right) in the ring topology. ——
Further training 40,000 steps from the Stage 1 checkpoint, \(\lambda\) raised to 0.3 ——

One asymmetry is worth stating explicitly. No SDXL segmentation or image-conditioned adapter is publicly available, so both of those T2I-Adapter branches start from scratch, whereas our RGB branch loads a pre-trained checkpoint.

BibTeX

@article{anonymous2026dtr2s2r,
  title   = {Digital Twin-Driven Real2Sim2Real: Simulator-Conditioned
             Generation via Paired Driving-Scene Reconstruction},
  author  = {Anonymous},
  journal = {Under review},
  year    = {2026}
}

Acknowledgments

This research (paper) used datasets from High-precision data collection vehicle daytime city road data. All data information can be accessed through AI-Hub.