Application of 3D Gaussian Splatting for Cinematic Anatomy on Consumer Class Devices

Simon Niedermayr¹, Christoph Neuhauser¹, Kaloian Petkov², Klaus Engel², Rüdiger Westermann¹

¹Technical University of Munich, ²Siemens Healthineers

Abstract

Interactive photorealistic rendering of 3D anatomy is used in medical education to explain the structure of the human body. It is currently restricted to frontal teaching scenarios, where even with a powerful GPU and high-speed access to a large storage device where the data set is hosted, interactive demonstrations can hardly be achieved. We present the use of novel view synthesis via compressed 3D Gaussian Splatting (3DGS) to overcome this restriction, and to even enable students to perform cinematic anatomy on lightweight and mobile devices.
Our proposed pipeline first finds a set of camera poses that captures all potentially seen structures in the data. High-quality images are then generated with path tracing and converted into a compact 3DGS representation, consuming < 70 MB even for data sets of multiple GBs. This allows for real-time photorealistic novel view synthesis that recovers structures up to the voxel resolution and is almost indistinguishable from the path-traced images.

Associated publications

Application of 3D Gaussian Splatting for Cinematic Anatomy on Consumer Class Devices
Simon Niedermayr, Christoph Neuhauser, Kaloian Petkov², Klaus Engel, Rüdiger Westermann
VMV 2024
[Preprint] [Project Page]