Visuotactile sensors are indispensable for contact-rich robotic manipulation tasks. However, policy learning with tactile feedback in simulation, especially for online reinforcement learning (RL), remains a critical challenge, as it demands a delicate balance between physics fidelity and computational efficiency. To address this challenge, we present Tac2Real, a lightweight visuotactile simulation framework designed to enable efficient online RL training. Tac2Real integrates the Preconditioned Nonlinear Conjugate Gradient Incremental Potential Contact (PNCG-IPC) method with a multi-node, multi-GPU high-throughput parallel simulation architecture, which can generate marker displacement fields at interactive rates. Meanwhile, we propose a systematic approach, TacAlign, to narrow both structured and stochastic sources of domain gap, ensuring a reliable zero-shot sim-to-real transfer. We further evaluate Tac2Real on the contact-rich peg insertion task. The zero-shot transfer results achieve a high success rate in the real-world scenario, verifying the effectiveness and robustness of our framework.
Tac2Real uses PNCG-IPC, a physically consistent simulation method with robust contact handling and numerical stability, as visuotactile simulator. The visuotactile simulation is integrated into existing physics engine to support online RL training follows a plugin-based architecture. By extracting relative pose data during physics steps to simulate marker displacements, it forms a combined observation for RL agents. This framework has cross-engine compatibility and excellent multi-node multi-gpu parallelization performance.
To systematically mitigate the sim-to-real gap, we propose TacAlign within the Tac2Real framework, addressing both structured and stochastic discrepancies. For structured alignment, robot control, baseline IPC parameters and task-based parameters calibration are contained, combined with randomization to handle stochastic gap.
The visuotactile simulation results of 4 indenters ((a), (b), (c) and (d) corresponds to cube, cylinder, moon and triangle, respectively) under press, slide and rotate deformation. We also compare Tac2Real with Tacchi (material point method based) and TacSL (penalty-based) method. We observed that Tac2Real and Tacchi align relative closely with the real situation under such small deformation, whereas Tac2Real deviates significantly
Under large deformations, Tacchi suffers from particle splashing and numerical instability caused by adhesion deficiencies. In contrast, Tac2Real robustly handles large rotations and slips, with the sensor reliably recovering its original state post-contact, demonstrating superior stability for contact-rich RL tasks.
Benchmarked on a node with 16 RTX 4090 GPUs and 4090 environments, Tac2Real reaches 4,465 FPS, outperforming Tacchi. While TacSL achieves higher raw FPS by relying on simple SDF queries, it sacrifices physical fidelity.
Through the baseline and tasks-based simulation parameters calibration, the PNCG-IPC tactile displacements fields are in excellent agreement with the ground truth.
We perform RL training on 2 contact-rich tasks in simulation, random orientation peg insertion and random orientation nut threading. (a)and (c) are snapshots of inference in peg insertion and nut threading tasks with tactile simulation; (b) and (d) are comparisons of the learning curve among Tac2Real, TacSL, Tacchi and no tactile feedback. Tac2Real and TacSL exhibit similar good performance in simulation environments, both significantly outperforming Tacchi (many ineffective environments caused by numerical explosions of MPM) and the tactile-free case.
Zero-shot real-world deployment snapshots of peg insertion tasks under 0°,15° and -15°.
Tac2Real achieves a high zero-shot sim-to-real succerss rate of 91.7%. Unlike the results in the simulation, the deployment of TacSL only attains a 15% success rate, attributable to the substantial discrepancy between simulated and real marker displacement fields. We found that the task-based calibration, followed by robot control alignment, plays an important role in narrowing the sim-to-real gap, without which the success rate will decrease significantly.
@article{tac2real2026,
title = {Tac2Real: Reliable and GPU Visuotactile Simulation for Online Reinforcement Learning and Zero-Shot Real-World Deployment},
author = {Ningyu Yan, Shuai Wang, Xing Shen, Hui Wang, Hanqing Wang, Yang Xiang, Jiangmiao Pang},
journal = {arXiv},
year = {2026},
url = {Arxiv_Tac2Real.pdf}
}