User testing and evaluation for Manager Dashboard
This article will introduce the fourth highlight of the project: a user testing I performed to evaluate the Manager Dashboard. The purpose of the user testing is to know the usability, ergonomics performance of the dashboard and to evaluate whether the design goals are met.
| Work Type: | Individual Work |
|---|---|
| Client: | KLM Royal Dutch Airlines |
| Publish Date: | August 2019 |
| Subject: | Manager Dashboard User Testing and Evaluation |
| Approach: | Prototype, Think-aloud, and Quantitative evaluation form |
Preface
The Manager Dashboard was designed based on the results of user research and stakeholder interviews. To evaluate the concept, five managers who belong to the target group were volunteered to participate the user
testing. There are two goals of the evaluation:
The first goal is to test the usability and cognitive ergonomics of the concept. The second goal is to know whether the concept meets the purpose of fulfilling the needs of managers and bringing out KLM values.
Research Approach & Setup
The user test took place at or close to the participantsʼ workplaces. The duration of the test was between 30 to 60 minutes. There were three parts in the user tests. First, managers were asked to think-aloud while using the prototypes. Second, they were asked to fill in an evaluation form. Third, a few more questions were asked verbally for more feedback and they are free to share more comments or remarks.
Part 1
Thinking-aloud
The thinking-aloud method encourages users to speak up their thoughts while using a computer system prototype/paper mock-up/documentation, i.e., prompted verbalising ideas, facts, expectations, doubt, etc. that comes to mind during the operation. It was utilised as a research method for evaluating the cognitive ergonomics score as it can bring timely, genuine and applicable feedback to the designers.
During the user test, each participant was asked to think-aloud while ‘interacting withʼ the prototype built with Keynote. The prototype is partial interactive, participants can control the tempo but cannot really click any button on the screen. Through the keynote animation, they were able to get the ideas of how the system works. To help the participants understand the usage of the prototype, contextual scenarios were given before a series of actions were displayed.
Part 2
Evaluation Form
After the operation, an evaluation form is provided for them to quantitatively evaluate the usability and ergonomics of the research, and to reflect whether the concept can fulfil their needs and bring out KLM values.
Part 3
Feedback & Remarks
Lastly, participants were asked to freely share their feedback and remarks on the concept.
Evaluation Result
All the managers who participated in the user test gave high regards to the concept. Both quantitative and qualitative evaluation results indicate that the concept has high usability, good cognitive ergonomics and it fulfils the needs of managers meanwhile bringing out the KLM values.
Feedbacks & Improvement
Based on the feedback, some designs have been improved, although further user tests or pilot test still need to be conducted to verify the adjustment. Take ‘Female rate’ on ‘Absenteeism dashboard’ as an example, some managers have doubts about whether it is important to show this information, others have unpleasant feelings about the way it is called ‘Female rate’ instead of ‘Gender Ratio’. Things like this need to be further tested.
