Niklaus, Joël; Giofré, Daniele (13 July 2023). Can we Pretrain a SotA Legal Language Model on a Budget From Scratch? In: Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) (pp. 158-182). Association for Computational Linguistics 10.18653/v1/2023.sustainlp-1.11
|
Text
2023.sustainlp-1.11.pdf - Published Version Available under License Creative Commons: Attribution (CC-BY). Download (1MB) | Preview |
Even though many efficient transformers have been proposed, only few such models are available for specialized domains. Additionally, since the pretraining process is extremely costly in general – but even more so as the sequence length increases – it is often only in reach of large research labs. One way of making pretraining cheaper is the Replaced Token Detection (RTD) task, by providing more signal during training compared to MLM, since the loss can be computed over all tokens. In this work, we train Longformer models with the efficient RTD task on long-context legal data to showcase that pretraining efficient LMs is possibl using less than 12 GPU days. We evaluate the trained models on challenging summarization tasks requiring the model to summarize complex long texts. We find that both the small and base models outperform their baselines on the in-domain BillSum and out-of-domain PubMed tasks in their respective parameter range. We publish our models as a resource for researcher and practitioners.
Item Type: |
Conference or Workshop Item (Paper) |
|---|---|
Division/Institute: |
Business School > Institute for Public Sector Transformation > Data and Infrastructure Business School |
Name: |
Niklaus, Joël 0000-0002-2779-1653 and Giofré, Daniele |
ISBN: |
978-1-959429-79-1 |
Publisher: |
Association for Computational Linguistics |
Language: |
English |
Submitter: |
Safiya Verbruggen |
Date Deposited: |
25 Aug 2023 10:19 |
Last Modified: |
25 Aug 2023 10:19 |
Publisher DOI: |
10.18653/v1/2023.sustainlp-1.11 |
Related URLs: |
|
ARBOR DOI: |
10.24451/arbor.19709 |
URI: |
https://arbor.bfh.ch/id/eprint/19709 |
Actions (login required)
 |
View Item |