Physiological signals measurement and spoofing detection from face video
Yu, Zitong (2022-03-04)
https://urn.fi/URN:ISBN:9789526232379
Kuvaus
Tiivistelmä
Abstract
Human faces contain rich biometric and physiological clues. Thus, identity recognition and physiological state monitoring from face videos are feasible. On one hand, subtle color changes in the facial skin can reveal important information about the heart pulse of individuals, which works as the base for remote photoplethysmography (rPPG) signal measurement. Benefitting from computer vision technology, physiological signals can be reconstructed from face videos under laboratory-controlled conditions. On the other hand, face anti-spoofing (FAS) is vital for biometric security as face recognition systems are vulnerable to various presentation attacks.
In the first part of this thesis, three end-to-end spatio-temporal methods are presented for reliable rPPG signals recovery. To exploit efficient contextual clues from both spatial and temporal perspectives, several handcrafted and automatically searched spatio-temporal networks are proposed. Moreover, negative Pearson-based temporal loss and cross-entropy-based frequency constraints as well as rPPG-related auxiliary supervision (e.g., skin segmentation) are proposed for accurate rPPG signal recovery.
In the second part of this thesis, seven deep learning based FAS methods are presented to resolve the issue of intrinsic spoof representation, which is crucial to real-world deployment under unseen scenarios and attack types. On one side, novel convolutional operators as well as the networks are designed for generalized, lightweight, and multi-modal FAS. On the other side, several material-based pixel-wise supervision signals (e.g., depth and reflection) are proposed with an advanced pyramid supervision strategy.
Finally, with the evidence that the spoofings like a face mask cannot reflect live heart pulses, a novel facial rPPG-based method using a vision transformer is proposed to extract discriminative periodic liveness clues for challenging 3D mask attack detection.
Tiivistelmä
Ihmiskasvot sisältävät runsaasti biometrisiä ja fysiologisia vihjeitä, mikä mahdollistaa identiteetin tunnistamisen ja fysiologisen tilan seurannan kasvovideosta. Toisaalta kasvojen ihon hienovaraiset värimuutokset voivat paljastaa tärkeää tietoa yksilöiden sydämen sykkeestä, jonka perusteella voidaan mitata signaaleja etäfotopletysmografian (rPPG) keinoin. Tietokonenäön avulla fysiologiset signaalit voidaan rekonstruoida kasvovideoista laboratorio-olosuhteissa. Toisaalta kasvojen väärentämisen torjunta (face anti-spoofing, FAS) on oleellista biometrisen-turvallisuuden kannalta, koska kasvojentunnistusjärjestelmät ovat alttiita väärien kasvokuvien käytölle.
Opinnäytetyön ensimmäisessä osassa esitellään kolme päästä päähän ajallis-paikallista mallia rPPG-signaalien luotettavaa palautumista varten. Sekä paikan että ajan pohjalta saatavien tehokkaiden kontekstuaalisten vihjeiden hyödyntämiseksi ehdotetaan useita käsin laadittuja ja automaattisesti haettuja ajallis-paikallisia verkkoja. Lisäksi rPPG-signaalien tarkkaa palautumista varten ehdotetaan Pearsonin korrelaatiokertoimeen perustuvia negatiivisen ajallisen häviön ja ristientropiaan perustuvia taajuuden rajoitteita sekä rPPG-lukemiin perustuvaa lisävalvontaa (esim. ihon segmentointi).
Opinnäytetyön toisessa osassa esitellään seitsemän syväoppimiseen perustuvaa FAS-menetelmää, joilla ratkaistaan sisäisten väärennöksen piirteiden ongelma, mikä on ratkaisevan tärkeää todellisessa käyttöönotossa ennennäkemättömissä tilanteissa ja hyökkäystyypeissä. Toisaalta uudet konvolutionaaliset operaattorit ja verkot on suunniteltu yleistetyille, kevyille ja multimodaalisille FAS-järjestelmille. Toisaalta ehdotetaan useiden materiaalipohjaisten, pikselikohtaisten valvontasignaalien (esim. syvyys ja heijastuminen) käyttöä kehittyneellä pyramidivalvontastrategialla.
Lopuksi, koska on näyttöä siitä, että kasvomaskin kaltaiset väärennökset eivät voi heijastaa sydämen sykettä, ehdotetaan uudenlaista kasvojen rPPG-pohjaista menetelmää, jossa käytetään vision transformer -tietokonenäköteknologiaa, jotta voidaan irrottaa erottelevia, jaksoittaisia elävyysvihjeitä haastavien 3D-maskien avulla tehtyjen huijausyritysten havaitsemiseksi.
Original papers
Original papers are not included in the electronic version of the dissertation.
Yu, Z., Li, X., & Zhao, G. (2019). Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks. Proceedings of the British Machine Vision Conference (BMVC), 29.1-29.12. https://bmvc2019.org/wp-content/papers/0186.html
Yu, Z., Peng, W., Li, X., Hong, X., & Zhao, G. (2019). Remote heart rate measurement from highly compressed facial videos: An end-to-end deep learning solution with video enhancement. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 151–160. https://doi.org/10.1109/ICCV.2019.00024
Yu, Z., Li, X., Niu, X., Shi, J., & Zhao, G. (2020). AutoHR: A strong end-to-end baseline for remote heart rate measurement with neural searching. IEEE Signal Processing Letters, 27, 1245–1249. https://doi.org/10.1109/LSP.2020.3007086
Yu, Z., Zhao, C., Wang, Z., Qin, Y., Su, Z., Li, X., Zhou, F., & Zhao, G. (2020). Searching central difference convolutional networks for face anti-spoofing. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5294–5304. https://doi.org/10.1109/CVPR42600.2020.00534
Yu, Z., Qin, Y., Li, X., Wang, Z., Zhao, C., Lei, Z., & Zhao, G. (2020). Multi-modal face anti-spoofing based on central difference networks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2766–2774. https://doi.org/10.1109/CVPRW50498.2020.00333
Yu, Z., Qin, Y., Xu, X., Zhao, C., Wang, Z., Lei, Z., & Zhao, G. (2020). Auto-Fas: Searching lightweight networks for face anti-spoofing. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 996–1000. https://doi.org/10.1109/ICASSP40776.2020.9053587
Yu, Z., Li, X., Niu, X., Shi, J., & Zhao, G. (2020). Face anti-spoofing with human material perception. In A. Vedaldi, H. Bischof, T. Brox, & J.-M. Frahm (Eds.), Computer Vision – ECCV 2020 (Vol. 12352, pp. 557–575). Springer International Publishing. https://doi.org/10.1007/978-3-030-58571-6_33
Yu, Z., Li, X., Shi, J., Xia, Z., & Zhao, G. (2021). Revisiting pixel-wise supervision for face anti-spoofing. IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(3), 285–295. https://doi.org/10.1109/TBIOM.2021.3065526
Yu, Z., Qin, Y., Zhao, H., Li, X., & Zhao, G. (2021). Dual-cross central difference network for face anti-spoofing. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, 1281–1287. https://doi.org/10.24963/ijcai.2021/177
Yu, Z., Li, X., Wang, P., & Zhao, G. (2021). TransRPPG: Remote photoplethysmography transformer for 3d mask face presentation attack detection. IEEE Signal Processing Letters, 28, 1290–1294. https://doi.org/10.1109/LSP.2021.3089908
Osajulkaisut
Osajulkaisut eivät sisälly väitöskirjan elektroniseen versioon.
Yu, Z., Li, X., & Zhao, G. (2019). Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks. Proceedings of the British Machine Vision Conference (BMVC), 29.1-29.12. https://bmvc2019.org/wp-content/papers/0186.html
Yu, Z., Peng, W., Li, X., Hong, X., & Zhao, G. (2019). Remote heart rate measurement from highly compressed facial videos: An end-to-end deep learning solution with video enhancement. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 151–160. https://doi.org/10.1109/ICCV.2019.00024
Yu, Z., Li, X., Niu, X., Shi, J., & Zhao, G. (2020). AutoHR: A strong end-to-end baseline for remote heart rate measurement with neural searching. IEEE Signal Processing Letters, 27, 1245–1249. https://doi.org/10.1109/LSP.2020.3007086
Yu, Z., Zhao, C., Wang, Z., Qin, Y., Su, Z., Li, X., Zhou, F., & Zhao, G. (2020). Searching central difference convolutional networks for face anti-spoofing. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5294–5304. https://doi.org/10.1109/CVPR42600.2020.00534
Yu, Z., Qin, Y., Li, X., Wang, Z., Zhao, C., Lei, Z., & Zhao, G. (2020). Multi-modal face anti-spoofing based on central difference networks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2766–2774. https://doi.org/10.1109/CVPRW50498.2020.00333
Yu, Z., Qin, Y., Xu, X., Zhao, C., Wang, Z., Lei, Z., & Zhao, G. (2020). Auto-Fas: Searching lightweight networks for face anti-spoofing. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 996–1000. https://doi.org/10.1109/ICASSP40776.2020.9053587
Yu, Z., Li, X., Niu, X., Shi, J., & Zhao, G. (2020). Face anti-spoofing with human material perception. In A. Vedaldi, H. Bischof, T. Brox, & J.-M. Frahm (Eds.), Computer Vision – ECCV 2020 (Vol. 12352, pp. 557–575). Springer International Publishing. https://doi.org/10.1007/978-3-030-58571-6_33
Yu, Z., Li, X., Shi, J., Xia, Z., & Zhao, G. (2021). Revisiting pixel-wise supervision for face anti-spoofing. IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(3), 285–295. https://doi.org/10.1109/TBIOM.2021.3065526
Yu, Z., Qin, Y., Zhao, H., Li, X., & Zhao, G. (2021). Dual-cross central difference network for face anti-spoofing. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, 1281–1287. https://doi.org/10.24963/ijcai.2021/177
Yu, Z., Li, X., Wang, P., & Zhao, G. (2021). TransRPPG: Remote photoplethysmography transformer for 3d mask face presentation attack detection. IEEE Signal Processing Letters, 28, 1290–1294. https://doi.org/10.1109/LSP.2021.3089908
Kokoelmat
- Avoin saatavuus [37920]