Lextreme: A multi-lingual and multi-task benchmark for the legal domain

Niklaus, Joël; Matoshi, Veton; Rani, Pooja; Galassi, Andrea; Stürmer, Matthias; Chalkidis, Ilias (10 December 2023). Lextreme: A multi-lingual and multi-task benchmark for the legal domain In: Findings of the Association for Computational Linguistics: EMNLP 2023. Ithaca, NY: Cornell University 10.18653/v1/2023.findings-emnlp.200

This is the latest version of this item.

[img]
Preview
 Text
2301.13126.pdf
Available under License Creative Commons: Attribution-Noncommercial-Share Alike (CC-BY-NC-SA).
Download (2MB) | Preview
[img]
Preview
 Text
2023.findings-emnlp.200.pdf - Published Version
Available under License Creative Commons: Attribution (CC-BY).
Download (2MB) | Preview

Lately, propelled by the phenomenal advances around the transformer architecture, the legal NLP field has enjoyed spectacular growth. To measure progress, well curated and challenging benchmarks are crucial. However, most benchmarks are English only and in legal NLP specifically there is no multilingual benchmark available yet. Additionally, many benchmarks are saturated, with the best models clearly outperforming the best humans and achieving near perfect scores. We survey the legal NLP literature and select 11 datasets covering 24 languages, creating LEXTREME. To provide a fair comparison, we propose two aggregate scores, one based on the datasets and one on the languages. The best baseline (XLM-R large) achieves both a dataset aggregate score a language aggregate score of 61.3. This indicates that LEXTREME is still very challenging and leaves ample room for improvement. To make it easy for researchers and practitioners to use, we release LEXTREME on huggingface together with all the code required to evaluate models and a public Weights and Biases project with all the runs.

Item Type:

 Conference or Workshop Item (Paper)

Division/Institute:

 Business School > Institute for Public Sector Transformation
Business School > Institute for Public Sector Transformation > Data and Infrastructure
Business School

Name:

 Niklaus, Joël0000-0002-2779-1653;
Matoshi, Veton0009-0002-6613-5701;
Rani, Pooja;
Galassi, Andrea;
Stürmer, Matthias0000-0001-9038-4041 and
Chalkidis, Ilias

Publisher:

 Cornell University

Language:

 English

Submitter:

 Joël Niklaus

Date Deposited:

 10 Sep 2024 15:28

Last Modified:

 10 Sep 2024 15:28

Publisher DOI:

 10.18653/v1/2023.findings-emnlp.200

Related URLs:

ARBOR DOI:

 10.24451/arbor.22335

URI:

 https://arbor.bfh.ch/id/eprint/22335

Available Versions of this Item

Actions (login required)

View Item View Item
Provide Feedback