Modelling Gaze Behaviour in Subtitle Processing

The Effect of Structural and Lexical Properties




audiovisual translation, reception, subtitles, eye-tracking, linear mixed-effects models



One of the main rules of subtitling states that subtitles should be formatted and timed so that viewers have enough time to read and understand the text but also to follow the picture. In this paper we examine the factors that influence the time viewers spend looking at subtitles. We concentrate on the lexical and structural properties of subtitles. The participant group (= 14) watched a television documentary with Russian narration and Finnish subtitles (the participants’ native language), while their eye movements were tracked. Using a linear mixed-effects model, we identified significant effects of subtitle duration and character count on the time participants spent looking at the subtitles. The model also revealed significant inter-individual differences, despite the fact that the participant group was seemingly homogeneous. The findings underline the complexity of subtitled audiovisual material as a stimulus of cognitive processing. We provide a starting point for more comprehensive modelling of the factors involved in gaze behaviour when watching subtitled content.

Lay summary

Subtitles have become a popular method for watching foreign series and films even in countries that have traditionally used dubbing in this regard. Because subtitles are visible to the viewer a short, limited time, they should be composed so that they are easy to read, and that the viewer has time to also follow the image. Nevertheless, the factors that have impact the time it takes to read a subtitle is not very well known. We wanted to find out what makes people who are watching subtitled television shows spend more time gazing at the subtitles? To answer this question, we recorded the eye movements of 14 participants when they were watching a short, subtitled television documentary. We created a statistical model of the gaze behavior from the eye movement data and found that both the length of the subtitle and the time the subtitle is visible are separate contributing factors. We also discovered large differences between individual viewers. Our conclusion is that people process subtitled content in very different ways, but there are some common tendencies. Our model can be seen as solid starting point for comprehensive modelling of gaze behavior of people watching subtitled audiovisual material.


Download data is not yet available.

Author Biographies

Juha Lång, University of Eastern Finland

Juha Lång, MA, is a doctoral candidate at the Department of Humanities, Philosophical Faculty, University of Eastern Finland. His research concentrates on the reception of audiovisual translation, most notably subtitles. He is particularly interested in examining both the conscious and sub-conscious processing of subtitles with qualitative and quantitative methods in order to uncover the factors involved in the processing of subtitled audiovisual products.

Hana Vrzakova, University of Eastern Finland & Kuopio University Hospital

Hana Vrzakova, Ph.D., is a reasearcher at Microsurgery Training Center, Kuopio University Hospital, and an active member of the eye-tracking community. She received her PhD in eye tracking, machine learning, and HCI at University of Eastern Finland (Cor Baayen Award 2020, Honorable Mentioned). In her postdoctoral research at CU Boulder (Emotive Computing Lab), she investigated physiological and behavioral synchrony underlying remote collaborative problem solving and negotiation. She is currently Head of Medical R&D in Microsurgy Training Center, Kuopio University Hospital, and Research Director in the EMC Program, University of Eastern Finland.

Lauri Mehtätalo, Natural Resources Institute Finland

Lauri Mehtätalo, D.Sc., is research professor at Natural Resources Institute Finland and former professor in applied statistics at University of Eastern Finland. His research interests include modeling of grouped data using nonlinear, linear, generalized, and multivariate mixed-effects models. He is the main author of the textbook “Mehtätalo L., and Lappi, J. 2020. Biometry for forestry and environmental data: With examples in R” and the related R-package lmfor.




How to Cite

Lång, J., Vrzakova, H., & Mehtätalo, L. (2021). Modelling Gaze Behaviour in Subtitle Processing: The Effect of Structural and Lexical Properties . Journal of Audiovisual Translation, 4(1), 71–95.



Research articles