EyeGuide - From Gaze Data to Instance Segmentation

Jacqueline Kockwelp (University of Münster), Joerg Gromoll (CeRA), Joachim Wistuba (Centre of Reproductive Medicine and Andrology), Benjamin Risse (University of Münster)*
The 34th British Machine Vision Conference


Obtaining precise instance-level segmentations is a challenging task in machine learning. Especially for objects with complex and non-convex geometries or with partial occlusions scribble, bounding boxes or user clicks are often provided to guide the segmentation. In this paper, we explore the usage of a remote eye tracking system to generate gaze data as an additional input for object segmentation models (called EyeGuide). The gaze data is recorded during routine image inspections (i.e. without giving a particular task) and is used as an additional input to train neural networks. Our results indicate that the acquisition of gaze data is faster and more convenient than providing explicit user input, less annotations are necessary to generate equal or better segmentation results and also overall better generalisation capabilities on unseen classes compared to state-of-the-art techniques. In summary, EyeGuide is a simple yet powerful guidance strategy that can directly be integrated in image inspection routines and neural network architectures.



author    = {Jacqueline Kockwelp and Joerg Gromoll and Joachim Wistuba and Benjamin Risse},
title     = {EyeGuide - From Gaze Data to Instance Segmentation},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year      = {2023},
url       = {https://papers.bmvc2023.org/0719.pdf}

Copyright © 2023 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection