Nyffenegger, Alex; Stürmer, Matthias; Niklaus, Joël (21 June 2024). Anonymity at Risk? Assessing Re-Identification Capabilities of Large Language Models in Court Decisions In: Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 2433-2462). Stroudsburg, PA, USA: Association for Computational Linguistics 10.18653/v1/2024.findings-naacl.157
|
Text
2024.findings-naacl.157.pdf - Published Version Available under License Creative Commons: Attribution (CC-BY). Download (5MB) | Preview |
Anonymity in court rulings is a critical aspect of privacy protection in the European Union and Switzerland but with the advent of LLMs, concerns about large-scale re-identification of anonymized persons are growing. In accordance with the Federal Supreme Court of Switzerland (FSCS), we study re-identification risks using actual legal data. Following the initial experiment, we constructed an anonymized Wikipedia dataset as a more rigorous testing ground to further investigate the findings. In addition to the datasets, we also introduce new metrics to measure performance. We systematically analyze the factors that influence successful re-identifications, identifying model size, input length, and instruction tuning among the most critical determinants. Despite high re-identification rates on Wikipedia, even the best LLMs struggled with court decisions. We demonstrate that for now, the risk of re-identifications using LLMs is minimal in the vast majority of cases. We hope that our system can help enhance the confidence in the security of anonymized decisions, thus leading the courts to publish more decisions.
Item Type: |
Conference or Workshop Item (Paper) |
|---|---|
Division/Institute: |
Business School > Institute for Public Sector Transformation Business School > Institute for Public Sector Transformation > Digital Sustainability Lab Business School |
Name: |
Nyffenegger, Alex; Stürmer, Matthias  0000-0001-9038-4041 and Niklaus, Joël 0000-0002-2779-1653 |
Subjects: |
K Law > K Law (General) Q Science > QA Mathematics > QA75 Electronic computers. Computer science |
ISBN: |
979-8-89176-119-3 |
Publisher: |
Association for Computational Linguistics |
Language: |
English |
Submitter: |
Matthias Stürmer |
Date Deposited: |
10 Sep 2024 15:18 |
Last Modified: |
10 Sep 2024 15:18 |
Publisher DOI: |
10.18653/v1/2024.findings-naacl.157 |
Related URLs: |
|
ARBOR DOI: |
10.24451/arbor.22330 |
URI: |
https://arbor.bfh.ch/id/eprint/22330 |
Actions (login required)
 |
View Item |