TileGen: Tileable, Controllable Material Generation and Capture
Siggraph Asia 2022

Abstract

overview
overview

Recent methods (e.g. MaterialGAN) have used unconditional GANs to generate per-pixel material maps, or as a prior to reconstruct materials from input photographs. These models can generate varied random material appearance, but do not have any mechanism to constrain the generated material to a specific category or to control the coarse structure of the generated material, such as the exact brick layout on a brick wall. Furthermore, materials reconstructed from a single input photo commonly have artifacts and are generally not tileable, which limits their use in practical content creation pipelines. We propose TileGen, a generative model for SVBRDFs that is specific to a material category, always tileable, and optionally conditional on a provided input structure pattern. TileGen is a variant of StyleGAN whose architecture is modified to always produce tileable (periodic) material maps. In addition to the standard “style” latent code, TileGen can optionally take a condition image, giving a user direct control over the dominant spatial (and optionally color) features of the material. For example, in brick materials, the user can specify a brick layout and the brick color, or in leather materials, the locations of wrinkles and folds. Our inverse rendering approach can find a material perceptually matching a single target photograph by optimization. This reconstruction can also be conditional on a user-provided pattern. The resulting materials are tileable, can be larger than the target image, and are editable by varying the condition.

Supplementary Video

Fast Forward Video

BibTeX 

@inproceedings{zhou2022tilegen,
                  title={TileGen: Tileable, Controllable Material Generation and Capture},
                  author={Zhou, Xilong and Hasan, Milos and Deschaintre, Valentin and Guerrero, Paul and Sunkavalli, Kalyan and Kalantari, Nima Khademi},
                  booktitle={SIGGRAPH Asia 2022 Conference Papers},
                  pages={1--9},
                  year={2022}
                }

Acknowledgements

We thank the Siggraph Asia 2022 reviewers for their constructive comments. We thank Krishna Kumar Singh, Yijun Li and Jingwan Lu for help with CollageGAN set up and training details The website template was borrowed from Michael Gharbi.