Interactive Segmentation of Radiance Fields

CVPR 2023

Rahul Goel, Dhawal Sirikonda, Saurabh Saini, P.J. Narayanan

BibTeX

@inproceedings{isrfgoel2023,
    title={{Interactive Segmentation of Radiance Fields}}, 
    author={Goel, Rahul and Sirikonda, Dhawal and Saini, Saurabh and Narayanan, P.J.},
    year={2023},
    booktitle = {{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}},
}

GUI Tool Demonstration on the GARDEN scene. This shows the demonstration of positive and negative strokes.

GUI Tool Demonstration on the KITCHEN scene. This shows the demonstration of multiple positive strokes from different views.

Abstract

Radiance Fields (RF) are popular to represent casually-captured scenes for new view synthesis and several applications beyond it. Mixed reality on personal spaces needs understanding and manipulating scenes represented as RFs, with semantic segmentation of objects as an important step. Prior segmentation efforts show promise but don't scale to complex objects with diverse appearance. We present the ISRF method to interactively segment objects with fine structure and appearance. Nearest neighbor feature matching using distilled semantic features identifies high-confidence seed regions. Bilateral search in a joint spatio-semantic space grows the region to recover accurate segmentation. We show state-of-the-art results of segmenting objects from RFs and compositing them to another scene, changing appearance, etc., and an interactive segmentation tool that others can use.

Video

pipeline diagram showing interactive segmentation

System overview: We capture a 3D scene of voxelized radiance field and distill the semantic feature into it. Once captured, the user can easily mark a regions using a brush tool on a reference view (green stroke). The features are collected corresponding to the marked pixels and clustered using K-Means. The voxel-grid is then matched using NNFM (nearest neighbor feature matching) to obtain a high confidence seed using a common tight threshold. The seed is then grown using bilateral filtering to smoothly cover the boundaries of the object, conditioning the growth in the spatio-semantic domain.

Detailed Results

Here, we show the segmentation of the Flower Pot in the GARDEN scene. Note the segmentation in the strands which is possible due to region growing.

This shows the scene with the pot removed.

This shows the pot and the granite part removed.

Here, the just the wooden tabletop has been segmented with the central granite removed.

Here, we remove the pot, the granite as well as the wooden tabletop.

Here, we show the segmentation of the lego JCB.

Here, we take the pot and place it on the ground underneath the table.

These two videos show composition of the two radiance fields. In the second video, we have applied style transfer
on just the segmented part to demonstrate appearance editing on desired objects.

Rendered Trajectory	N3F/DFF	Ours