Dementia Awareness Summative Research
Summative evaluation of dementia awareness website.
Problem
My thesis project, conducted in two phases, investigates the effectiveness of allowing older adults to self-tailor, or personalize, a website's information architecture (IA) structure. The content used on the website are dementia awareness videos created for a separate research project in collaboration with Cyber-Seniors. View the entire thesis here.
This page describes the second phase of the project, during which I conducted a usability evaluation of the refined prototype design created in the first phase. The goal of the usability evaluation was to determine the impact that being able to self-tailor the IA structure had on the participants' experiences while using the website.
My Responsibilities
Prototyping, Research Study Design, Usability Evaluation
Tools
Framer
Zoom
Study Design
The usability evaluation required participants to complete a series of tasks on three different versions of the prototype: one which had a linear structure, one which had a matrix structure, and one which had the self-tailoring element, allowing participants to freely switch between a linear and matrix structure. The viewing and navigating experience of using the self-tailoring element was evaluated based on the participants’ satisfaction and engagement with the three different versions of the prototype, measured using a post-task Likert scale questionnaire and post-evaluation semi-structured interview.
The usability evaluation required participants to complete a series of tasks on three different versions of the prototype: one which had a linear structure, one which had a matrix structure, and one which had the self-tailoring element, allowing participants to freely switch between a linear and matrix structure. The viewing and navigating experience of using the self-tailoring element was evaluated based on the participants’ satisfaction and engagement with the three different versions of the prototype, measured using a post-task Likert scale questionnaire and post-evaluation semi-structured interview.
Instruments
Questionnaire
During the usability evaluation, each participant completed a post-task questionnaire after the first and second task. The purpose of the questionnaire was to inform which IA structure, linear or matrix, was preferred by the participants by identifying which of the two they found more engaging.
The questionnaire used was a modified version of the UES-SF developed by O’Brien et al. (2018), using a 5-point Likert scale.
Questionnaire Statements:
Using this website fully held my attention.
I was so involved in this experience that I lost track of time.
I was absorbed in this experience.
I felt frustrated while using this website.
I found this website confusing to use.
Using this website was taxing.
This website was attractive.
The screen layout of this website was visually appealing.
This website appealed to my senses.
Using this website was worthwhile.
My experience was rewarding.
I felt interested in this experience.
Post-Evaluation Interview
Following the completion of all three tasks in the usability evaluation, a semi-structured interview was conducted with each participant. Emphasis was placed on understanding the participants' experience using the self-tailoring feature on the website (task three), particularly for the questions that focused on functionality.
Interview Questions:
You were just shown three different versions of the same website. Overall, do you prefer one version of the website over the others? If yes, could you expand on why? If no, what are your thoughts on the different versions of the website?
The third version of the website gave you the option to switch between two different ways of navigating between pages. What are your thoughts are that option?
Does being able to switch how you navigate between pages seem useful? Why or why not?
Did you feel like you had enough information to decide which way of navigating, or going between pages, to select? Explain why.
Functionality
Do you think you would use this feature which lets you change how you navigate pages on another website? Why?
Does the way this feature works makes sense? Is there anything about it that was unclear?
Visual Design
Does the design of the personalization feature look nice? Why or why not?
Is there anything hard to read or understand because of the colours or shapes used? What about the size of the buttons and where they are located on the website?
What about the website as a whole, is there anything visually that is unappealing or should be changed?
Text & Language
Is there any other text or language, anywhere on the website, that should be changed or that I could improve on?
Procedure
Participants
A total of 12 older adults participated in the usability evaluation. In order to participate in the study, participants had to meet a set of inclusion criteria:
you are over the age of 65
you are fluent in English
you have previous experience using a computer and browsing websites (basic IT support to be provided by the researcher)
you have access to a desktop computer or laptop
you are interested in an online self-guided dementia awareness program
you do not have a form of dementia
you consent to have your audio and screen recorded
Additionally, participants were required to complete a consent form before they could participate in the study.
Evaluation Protocol
Before we start, please go over this consent form if you have not done so already. A few things to note is that this usability evaluation and follow up interview will take about 45 minutes and our audio as well as your screen will be recorded, and the audio will be transcribed. Your name and any identifying information will not be associated with the data. If at any point you would like to stop the usability evaluation or withdraw from the study just let me know. No explanations are needed. Additionally, if there is a topic that comes up that you do not wish to talk about, or a question you do not want to answer, just let me know and we can move on.
Take your time to read over the consent form and let me know if you have any questions. If you have no already signed the consent form, please do so and we will begin the usability evaluation.
[Consent Form]
You will be asked to perform three tasks using different versions of the same website. After the first two tasks, you will be asked to complete a short questionnaire based on your experience. Following the third and final tasks, I will ask you a few more questions about your experience using the different versions of the website.
This website is made up of pages with information about different risk factors for dementia. Each page has a video and some written text about each topic.
Task One
[Participant Completes Task]
Task Two
Task Three
[Post-Evaluation Interview]
Counterbalancing
Given that the usability evaluation was designed to be a within-subjects evaluation, the order in which the participants were exposed to the different versions of the website prototype was counterbalanced to account for potential order and sequence effects and improve the internal validity of the evaluation.
The evaluation had two instances of counterbalancing. First, the order of task one and task two were counterbalanced so half of the participants started with the linear-only structure and half started with the matrix-only structure.
The second instance of counterbalancing is during the third task, where half of the participants who started with the linear-only structure were presented with the prototype that allowed for tailoring but had the linear structure as the default and half started with the structure that allowed for tailoring but had the matrix structure as the default. The participants who started with the matrix-only structure are also counterbalanced in this manner.
