Testing: approaches and methods2020-11-09T15:26:09+11:00

Testing: approaches and methods

Macquarie University and Biotext have collaborated in two research projects on testing the readability of health information (2018-20).  Both projects were funded by the Department of Industry, Innovation and Science as Innovation Connections Grants.

#1 The first Innovation Connections project (2018-19) was firstly a scoping study in 5 phases, setting out the parameters of readability from the language perspective, and identifying the different types of readers of health information. Research on both perspectives and the interplay between them was presented in the final report in its 5 sections. Its key findings were:

  1. The oldest and widely used quantitative readability measure (Flesch-Kincaid) is insufficient for assessing the difficulty of health information for broad spectrum readerships in Australia, and for first- and second-language users of English.
  2. The Coh-Metrix measures of text ease/difficulty,, developed on linguistic and psycholinguistic principles, are more useful in identifying and quantifying the likely readability issues in health information for second-language readers and those with low reading skills.
  3. There is no one-size-fits-all readability level that works for all health information readers: instead websites need to have separate, clearly identified sections for basic and professional health content, so as to optimise the messaging for each group.

A summary of the full report can be accessed here.

A preliminary experiment was also conducted (2018-19) on the readability of content on a Department of Health immunisation website. The experiment involved 30 parents with children at the University’s child-care centres. All had professional training and occupations, and were either first- or second- language users of English. They were tasked with finding specific information on the website, and debriefed on their experience of using it.  Eye-tracking equipment was used to trace their explorations of webpages and mouseclicks as they moved to a new page.  The overall time they took to complete the 7 tasks, and the correctness of their answers was recorded, as was how they experienced the website’s features, in terms of likes/dislikes, and via elicited individual comments.

Statistically significant differences were found in the overall times taken by participants to complete the tasks.  Second-language users of English took more time to complete them, and were less successful in extracting the correct information from the website. The findings In terms of user experience included comments that the website’s graphics were insufficiently focused for navigation, and a very long FAQ list lacked any visible structure.

#2 In the second Innovation Connections project (2019-2020), a second eye-tracking experiment was conducted with 58 university students, both first- and second-language English, to test whether differing levels of readability could be measured by means of eye movements (fixation counts). The test materials consisted of short pieces of health information on dementia, diabetes and epilepsy, of which alternative high, medium and low difficulty versions were created experimentally, by modifying their language (choice of words and sentence patterns).  The participants each read just one of the modified pieces on each topic, its stylistic level selected randomly.  They then responded to a set of 5 short answer questions about the three topics, to test how accurately they could understand the information through the styiistic level at which it was delivered to them.  The time they took to read each text was also recorded.

Significant differences were found in the numbers of eye fixations: there were consistently higher fixation scores from low, medium to high style texts for both language groups, and in the overall times taken to read the texts at each level.  But the first-language participants registered consistently fewer eye fixations and took less time to read each piece than members of the second-language group. Researchers concluded these two (number of fixations, time taken to read) are robust measures of readability, which should be further tested on more diverse populations.

Pam-Peters

Pam Peters

Emeritus Professor, Adjunct Professor, Macquarie University

Pam Peter is a Fellow of the Australian Academy of the Humanities

Jan-Louis-Kruger

Jan-Louis Kruger

Head of Department and Associate Professor, Department of Linguistics

Jan-Louis is Head of the Department of Linguistics and a member of the Centre for Language Sciences