StyleTRF: Stylizing Tensorial Radiance Fields

Rahul Goel, Dhawal Sirikonda, Saurabh Saini, P.J. Narayanan
CVIT, IIIT Hyderabad

ICVGIP 2022

Paper Arxiv Video Poster

StyleTRF applies stylization on radiance fields in less than 40 seconds!

Abstract

Stylized view generation of scenes captured casually using a camera has received much attention recently. The geometry and appearance of the scene are typically captured as neural point sets or neural radiance fields in the previous work. An image stylization method is used to stylize the captured appearance by training its network jointly or iteratively with the structure capture network. The state-of-the-art SNeRF method trains the NeRF and stylization network in an alternating manner. These methods have high training time and require joint optimization. In this work, we present StyleTRF, a compact, quick-to-optimize strategy for stylized view generation using TensoRF. The appearance part is fine-tuned using sparse stylized priors of a few views rendered using the TensoRF representation for a few iterations. Our method thus effectively decouples style-adaption from view capture and is much faster than the previous methods. We show state-of-the-art results on several scenes used for this purpose.

Video

pipeline diagram showing interactive segmentation

System overview: The pipeline diagram presents an overview of the strategy employed by our work. We first optimize Tensorial Radiance Fields for representation of the scene as proposed by Concurrently we optimize stylization module utilizing Johnson et al. method on COCO14-dataset. The stylization module is then used to stylize a sparse set of novel views generated by the pre-optimized TensoRF. These stylized views act as sparse style-priors and are used to fine-tune the appearance of the previously optimized scene representation. It is to be noted that we freeze the density terms of the TensoRF and only alter the appearance vectors which retains geometric while adapting the novel style.

Concurrent Works

ARF: Artistic Radiance Fields incorporates artistic brush-strokes using an NNFM based loss function.

BibTeX

@inproceedings{goel2022styletrf,
    author = {Goel, Rahul and Sirikonda, Dhawal and Saini, Saurabh and Narayanan, P. J.},
    title = {StyleTRF: Stylizing Tensorial Radiance Fields},
    year = {2022},
    doi = {10.1145/3571600.3571643},
    booktitle = {Proceedings of the Thirteenth Indian Conference on Computer Vision, Graphics and Image Processing},
    series = {ICVGIP '22}
}