Testing readability and measuring it2020-05-22T17:49:31+10:00

Testing readability and measuring it: 2018-2020

Macquarie University and Biotext have collaborated in two research projects on testing the readability of health information (2018-20).  Both projects were funded by the Department of Industry, Innovation and Science as Innovation Connections Grants.

#1 The first Innovation Connections project (2018-19) was largely a scoping study, setting out the parameters of readability from the language perspective and different types of readers.  These correspond with the five major sections of the first project report (February 2019):

Section 1. Linguistic aspects of readability, health literacy and health communication

Section 2. Readability checkers used by Australian governments, and their scoring systems

Section 3. Benchmarking readability for Australian consumers of health information

Section 4. The reading skills of Australians who use English as a second language

Section 5. Readability and access to medical information across the health sector

A preliminary experiment was also conducted (2018-19) on the readability of content on a Department of Health immunisation website. The experiment involved 30 parents with children at the University’s child-care centres. All had professional training and occupations, and were either first- or second- language users of English. They were tasked with finding specific information on the website, and debriefed on their experience of using it.  Eye-tracking equipment was used to trace their explorations of webpages and mouseclicks as they moved to a new page.  The overall time they took to complete the 7 tasks, and the correctness of their answers was recorded, as was how they experienced the website’s features, in terms of likes/dislikes, and via elicited individual comments.

Statistically significant differences were found in the overall times taken by participants to complete the tasks.  Second-language users of English took more time to complete them, and were less successful in extracting the correct information from the website. The findings In terms of user experience included comments that the website’s graphics were insufficiently focused for navigation, and a very long FAQ list lacked any visible structure.

#2 In the second Innovation Connections project (2019-2020), a second eye-tracking experiment was conducted with 58 university students, both first- and second-language English, to test whether differing levels of readability could be measured by means of eye movements (fixation counts). The test materials consisted of short pieces of health information on dementia, diabetes and epilepsy, of which alternative high, medium and low difficulty versions were created experimentally, by modifying their language (choice of words and sentence patterns).  The participants each read just one of the modified pieces on each topic, its stylistic level selected randomly.  They then responded to a set of 5 short answer questions about the three topics, to test how accurately they could understand the information through the styiistic level at which it was delivered to them.  The time they took to read each text was also recorded.

Significant differences were found in the numbers of eye fixations: there were consistently higher fixation scores from low, medium to high style texts for both language groups, and in the overall times taken to read the texts at each level.  But the first-language participants registered consistently fewer eye fixations and took less time to read each piece than members of the second-language group. Researchers concluded these two (number of fixations, time taken to read) are robust measures of readability, which should be further tested on more diverse populations.

Pam-Peters

Pam Peters

Emeritus Professor, Adjunct Professor, Macquarie University

Pam Peter is a Fellow of the Australian Academy of the Humanities

Jan-Louis-Kruger

Jan-Louis Kruger

Head of Department and Associate Professor, Department of Linguistics

Jan-Louis is Head of the Department of Linguistics and a member of the Centre for Language Sciences