Questionnaire experience and the hybrid System Usability Scale: Using a novel concept to evaluate a new instrument

Baumgartner, Juergen; Ruettgers, Nicole; Hasler, Annigna; Sonderegger, Andreas; Sauer, Juergen (2021). Questionnaire experience and the hybrid System Usability Scale: Using a novel concept to evaluate a new instrument International Journal of Human-Computer Studies, 147(147), p. 102575. Elsevier 10.1016/j.ijhcs.2020.102575

Preview	Text Baumgartner_2021_Questionnaire experience and the hybrid System Usability Scale.pdf - Published Version Available under License Creative Commons: Attribution-Noncommercial-No Derivative Works (CC-BY-NC-ND). Download (2MB) | Preview
Preview	Text manuscript_hsus_revision_01.11.2020.pdf - Accepted Version Available under License Creative Commons: Attribution-Noncommercial-No Derivative Works (CC-BY-NC-ND). Download (1MB) | Preview

This article presents the concept of questionnaire experience (QX), intending to add a new element to the psychometric evaluation of questionnaires, which may eventually help increase the validity and reliability of instruments. The application of QX is demonstrated in the development of the Hybrid System Usability Scale (H-SUS), making use of items comprising pictorial and verbal elements to measure perceived usability. The H-SUS was modelled on the verbal version of the System Usability Scale (SUS). Since previous research showed advantages of pictorial scales over verbal scales (e.g., higher respondent motivation) but also disadvantages (e.g., longer completion times), we assumed that hybrid scales would combine the advantages of both scale types. The goal of this study was to compare the two instruments by assessing traditional psychometric criteria (convergent, divergent and criterion-related validity, reliability and sensitivity) and respondent-related aspects of QX (respondent workload, respondent motivation, questionnaire preference, and questionnaire completion time). An online experiment was carried out (N = 152), in which participants interacted with a smartphone prototype and subsequently completed the verbal SUS together with the H-SUS. Results indicate good psychometric properties of the H-SUS. Compared to the SUS, the H-SUS showed similar workload levels for questionnaire completion, higher levels of respondent motivation, but longer questionnaire completion time. Overall, the H-SUS is considered a promising alternative for the evaluation of perceived usability. Finally, QX can be considered a useful concept for identifying potential problems of psychometric instruments in a respondent-centred way, which may help improve the quality of future scales.

Item Type:	Journal Article (Original Article)
Division/Institute:	Business School > Institute for New Work Business School > Institute for New Work > New Forms of Work and Organisation
Name:	Baumgartner, Juergen; Ruettgers, Nicole; Hasler, Annigna; Sonderegger, Andreas0000-0003-0054-0544 and Sauer, Juergen
Subjects:	B Philosophy. Psychology. Religion > BF Psychology B Philosophy. Psychology. Religion > BH Aesthetics
ISSN:	1071-5819
Publisher:	Elsevier
Submitter:	Andreas Sonderegger
Date Deposited:	20 Jan 2021 10:01
Last Modified:	01 Apr 2023 02:30
Publisher DOI:	10.1016/j.ijhcs.2020.102575
ARBOR DOI:	10.24451/arbor.13934
URI:	https://arbor.bfh.ch/id/eprint/13934