PFStorer: Personalized Face Restoration and Super-Resolution
Varanka, Tuomas; Toivonen, Tapani; Tripathy, Soumya; Zhao, Guoying; Acar, Erman (2024-09-16)
IEEE
16.09.2024
T. Varanka, T. Toivonen, S. Tripathy, G. Zhao and E. Acar, "PFStorer: Personalized Face Restoration and Super-Resolution," 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2024, pp. 2372-2381, doi: 10.1109/CVPR52733.2024.00230.
https://rightsstatements.org/vocab/InC/1.0/
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
https://rightsstatements.org/vocab/InC/1.0/
© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
https://rightsstatements.org/vocab/InC/1.0/
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:oulu-202409276095
https://urn.fi/URN:NBN:fi:oulu-202409276095
Tiivistelmä
Abstract
Recent developments in face restoration have achieved remarkable results in producing high-quality and lifelike outputs. The stunning results however often fail to be faith-ful with respect to the identity of the person as the models lack necessary context. In this paper, we explore the poten-tial of personalized face restoration with diffusion models. In our approach a restoration model is personalized using a few images of the identity, leading to tailored restoration with respect to the identity while retaining fine-grained de-tails. By using independent trainable blocks for personal-ization, the rich prior of a base restoration model can be ex-ploited to its fullest. To avoid the model relying on parts of identity left in the conditioning low-quality images, a gener-ative regularizer is employed. With a learnable parameter, the model learns to balance between the details generated based on the input image and the degree of personalization. Moreover, we improve the training pipeline of face restoration models to enable an alignment-free approach. We showcase the robust capabilities of our approach in sev-eral real-world scenarios with multiple identities, demon-strating our method's ability to generate fine-grained de-tails with faithful restoration. In the user study we evalu-ate the perceptual quality and faithfulness of the generated details, with our method being voted best 61% of the time compared to the second best with 25% of the votes.
Recent developments in face restoration have achieved remarkable results in producing high-quality and lifelike outputs. The stunning results however often fail to be faith-ful with respect to the identity of the person as the models lack necessary context. In this paper, we explore the poten-tial of personalized face restoration with diffusion models. In our approach a restoration model is personalized using a few images of the identity, leading to tailored restoration with respect to the identity while retaining fine-grained de-tails. By using independent trainable blocks for personal-ization, the rich prior of a base restoration model can be ex-ploited to its fullest. To avoid the model relying on parts of identity left in the conditioning low-quality images, a gener-ative regularizer is employed. With a learnable parameter, the model learns to balance between the details generated based on the input image and the degree of personalization. Moreover, we improve the training pipeline of face restoration models to enable an alignment-free approach. We showcase the robust capabilities of our approach in sev-eral real-world scenarios with multiple identities, demon-strating our method's ability to generate fine-grained de-tails with faithful restoration. In the user study we evalu-ate the perceptual quality and faithfulness of the generated details, with our method being voted best 61% of the time compared to the second best with 25% of the votes.
Kokoelmat
- Avoin saatavuus [38865]