Semo, Gil; Bernsohn, Dor; Hagag, Ben; Hayat, Gila; Niklaus, Joël (1 November 2022). ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US In: Proceedings of the Natural Legal Language Processing Workshop 2022 (pp. 31-46). Association for Computational Linguistics 10.18653/v1/2022.nllp-1.3
|
Text
2211.00582.pdf Available under License Creative Commons: Attribution (CC-BY). Download (3MB) | Preview |
The research field of Legal Natural Language Processing (NLP) has been very active recently, with Legal Judgment Prediction (LJP) becoming one of the most extensively studied tasks. To date, most publicly released LJP datasets originate from countries with civil law. In this work, we release, for the first time, a challenging LJP dataset focused on class action cases in the US. It is the first dataset in the common law system that focuses on the harder and more realistic task involving the complaints as input instead of the often used facts summary written by the court. Additionally, we study the difficulty of the task by collecting expert human predictions, showing that even human experts can only reach 53% accuracy on this dataset. Our Longformer model clearly outperforms the human baseline (63%), despite only considering the first 2,048 tokens. Furthermore, we perform a detailed error analysis and find that the Longformer model is significantly better calibrated than the human experts. Finally, we publicly release the dataset and the code used for the experiments.
Item Type: |
Conference or Workshop Item (Paper) |
|---|---|
Division/Institute: |
Business School > Institute for Public Sector Transformation > Data and Infrastructure Business School |
Name: |
Semo, Gil; Bernsohn, Dor; Hagag, Ben; Hayat, Gila and Niklaus, Joël  0000-0002-2779-1653 |
Publisher: |
Association for Computational Linguistics |
Language: |
English |
Submitter: |
Safiya Verbruggen |
Date Deposited: |
25 Aug 2023 10:51 |
Last Modified: |
25 Aug 2023 10:51 |
Publisher DOI: |
10.18653/v1/2022.nllp-1.3 |
Related URLs: |
|
ARBOR DOI: |
10.24451/arbor.19710 |
URI: |
https://arbor.bfh.ch/id/eprint/19710 |
Actions (login required)
 |
View Item |