Evaluation Metrics for Health Chatbots: A Delphi Study

Denecke, Kerstin; Abd-Alrazaq, Alaa; Househ, Mowafa; Warren, Jim (2021). Evaluation Metrics for Health Chatbots: A Delphi Study Methods of Information in Medicine, 60(05/06), pp. 171-179. Thieme 10.1055/s-0041-1736664

Text MIM_21010075.pdf - Published Version Restricted to registered users only Available under License Publisher holds Copyright. Download (1MB) | Request a copy

Background In recent years, an increasing number of health chatbots has been published in app stores and described in research literature. Given the sensitive data they are processing and the care settings for which they are developed, evaluation is essential to avoid harm to users. However, evaluations of those systems are reported inconsistently and without using a standardized set of evaluation metrics. Missing standards in health chatbot evaluation prevent comparisons of systems, and this may hamper acceptability since their reliability is unclear. Objectives The objective of this paper is to make an important step toward developing a health-specific chatbot evaluation framework by finding consensus on relevant metrics. Methods We used an adapted Delphi study design to verify and select potential metrics that we retrieved initially from a scoping review. We invited researchers, health professionals, and health informaticians to score each metric for inclusion in the final evaluation framework, over three survey rounds. We distinguished metrics scored relevant with high, moderate, and low consensus. The initial set of metrics comprised 26 metrics (categorized as global metrics, metrics related to response generation, response understanding and aesthetics). Results Twenty-eight experts joined the first round and 22 (75%) persisted to the third round. Twenty-four metrics achieved high consensus and three metrics achieved moderate consensus. The core set for our framework comprises mainly global metrics (e.g., ease of use, security content accuracy), metrics related to response generation (e.g., appropriateness of responses), and related to response understanding. Metrics on aesthetics (font type and size, color) are less well agreed upon—only moderate or low consensus was achieved for those metrics. Conclusion The results indicate that experts largely agree on metrics and that the consensus set is broad. This implies that health chatbot evaluation must be multiface- ted to ensure acceptability.

Item Type:	Journal Article (Original Article)
Division/Institute:	School of Engineering and Computer Science > Institute for Patient-centered Digital Health School of Engineering and Computer Science
Name:	Denecke, Kerstin0000-0001-6691-396X; Abd-Alrazaq, Alaa; Househ, Mowafa and Warren, Jim
Subjects:	Q Science > Q Science (General) T Technology > T Technology (General)
ISSN:	0026-1270
Publisher:	Thieme
Language:	English
Submitter:	Kerstin Denecke
Date Deposited:	12 Nov 2021 11:33
Last Modified:	25 Oct 2023 14:05
Publisher DOI:	10.1055/s-0041-1736664
ARBOR DOI:	10.24451/arbor.15682
URI:	https://arbor.bfh.ch/id/eprint/15682