SecNet: Semantic Eye Completion in Implicit Field

Yida Wang, Yiru Shen, David Joseph Tan, Federico Tombari, Sachin S. Talathi
Proceedings of The 1st Gaze Meets ML workshop, PMLR 210:241-256, 2023.

Abstract

If we take a depth image of an eye, noise artifacts and holes significantly affect the depth values on the eye due to the specularity of the sclera. This paper aims at solving this problem through semantic shape completion. We propose an end-to-end approach to train a neural network, called SecNet (semantic eye completion network), that predicts a point cloud with an accurate eye-geometry coupled with the semantic labels of each point. These labels correspond to the essential eye-regions, i.e. pupil, iris and sclera. Particularly, our work performs implicit estimation of the query points with semantic labels where both the semantic and occupancy predictions are trained in an end-to-end way. To evaluate the ap- proach, we then use the synthetic eye-scans rendered in UnityEyes simulator environment. Compared to the state of the art, the proposed method improves the accuracy for shape- completion for 3D eye-scan by 8.2%. In practice, we also demonstrate the application of our semantic eye completion for gaze estimation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v210-wang23a, title = {SecNet: Semantic Eye Completion in Implicit Field}, author = {Wang, Yida and Shen, Yiru and Tan, David Joseph and Tombari, Federico and Talathi, Sachin S.}, booktitle = {Proceedings of The 1st Gaze Meets ML workshop}, pages = {241--256}, year = {2023}, editor = {Lourentzou, Ismini and Wu, Joy and Kashyap, Satyananda and Karargyris, Alexandros and Celi, Leo Anthony and Kawas, Ban and Talathi, Sachin}, volume = {210}, series = {Proceedings of Machine Learning Research}, month = {03 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v210/wang23a/wang23a.pdf}, url = {https://proceedings.mlr.press/v210/wang23a.html}, abstract = {If we take a depth image of an eye, noise artifacts and holes significantly affect the depth values on the eye due to the specularity of the sclera. This paper aims at solving this problem through semantic shape completion. We propose an end-to-end approach to train a neural network, called SecNet (semantic eye completion network), that predicts a point cloud with an accurate eye-geometry coupled with the semantic labels of each point. These labels correspond to the essential eye-regions, i.e. pupil, iris and sclera. Particularly, our work performs implicit estimation of the query points with semantic labels where both the semantic and occupancy predictions are trained in an end-to-end way. To evaluate the ap- proach, we then use the synthetic eye-scans rendered in UnityEyes simulator environment. Compared to the state of the art, the proposed method improves the accuracy for shape- completion for 3D eye-scan by 8.2%. In practice, we also demonstrate the application of our semantic eye completion for gaze estimation.} }
Endnote
%0 Conference Paper %T SecNet: Semantic Eye Completion in Implicit Field %A Yida Wang %A Yiru Shen %A David Joseph Tan %A Federico Tombari %A Sachin S. Talathi %B Proceedings of The 1st Gaze Meets ML workshop %C Proceedings of Machine Learning Research %D 2023 %E Ismini Lourentzou %E Joy Wu %E Satyananda Kashyap %E Alexandros Karargyris %E Leo Anthony Celi %E Ban Kawas %E Sachin Talathi %F pmlr-v210-wang23a %I PMLR %P 241--256 %U https://proceedings.mlr.press/v210/wang23a.html %V 210 %X If we take a depth image of an eye, noise artifacts and holes significantly affect the depth values on the eye due to the specularity of the sclera. This paper aims at solving this problem through semantic shape completion. We propose an end-to-end approach to train a neural network, called SecNet (semantic eye completion network), that predicts a point cloud with an accurate eye-geometry coupled with the semantic labels of each point. These labels correspond to the essential eye-regions, i.e. pupil, iris and sclera. Particularly, our work performs implicit estimation of the query points with semantic labels where both the semantic and occupancy predictions are trained in an end-to-end way. To evaluate the ap- proach, we then use the synthetic eye-scans rendered in UnityEyes simulator environment. Compared to the state of the art, the proposed method improves the accuracy for shape- completion for 3D eye-scan by 8.2%. In practice, we also demonstrate the application of our semantic eye completion for gaze estimation.
APA
Wang, Y., Shen, Y., Tan, D.J., Tombari, F. & Talathi, S.S.. (2023). SecNet: Semantic Eye Completion in Implicit Field. Proceedings of The 1st Gaze Meets ML workshop, in Proceedings of Machine Learning Research 210:241-256 Available from https://proceedings.mlr.press/v210/wang23a.html.

Related Material