MuLD (Multitask Long Document Benchmark) is a set of 6 NLP tasks where the inputs consist of at least 10,000 words. The benchmark covers a wide variety of task types including translation, summarization, question answering, and classification. Additionally there is a range of output lengths from a single word classification label all the way up to an output longer than the input text.

This repo contains official code for the paper MuLD: The Multitask Long Document Benchmark.

The easiest method is to use the Huggingface Datasets library:

import datasets
ds = datasets.load_dataset("ghomasHudson/muld", "NarrativeQA")
ds = datasets.load_dataset("ghomasHudson/muld", "HotpotQA")
ds = datasets.load_dataset("ghomasHudson/muld", "Character Archetype Classification")
ds = datasets.load_dataset("ghomasHudson/muld", "OpenSubtitles")
ds = datasets.load_dataset("ghomasHudson/muld", "AO3 Style Change Detection")
ds = datasets.load_dataset("ghomasHudson/muld", "VLSP")

Or by cloning this repo:

import datasets
ds = datasets.load_dataset("./muld.py", "NarrativeQA")
...

If you prefer to download the data files yourself:

• NarrativeQA Train Val Test
  • Mirror: Train Val Test
• HotpotQA Train, Val
  • Mirror: Train Val
• Character Archetype Classification Train Val Test
  • Mirror: Train Val Test
• OpenSubtitles Train Test
  • Mirror: Train Test
• Style Change Train Val Test,
  • Mirror: Train Val Test
• VLSP Test
  • Mirror: Test

If you use our benchmark please cite the paper:

@InProceedings{hudson-almoubayed:2022:LREC,
  author    = {Hudson, George  and  Al Moubayed, Noura},
  title     = {MuLD: The Multitask Long Document Benchmark},
  booktitle = {Proceedings of the Language Resources and Evaluation Conference},
  month     = {June},
  year      = {2022},
  address   = {Marseille, France},
  publisher = {European Language Resources Association},
  pages     = {3675--3685},
  url       = {https://aclanthology.org/2022.lrec-1.392}
}

Additionally please cite the datasets we used (particularly NarrativeQA, HotpotQA, and Opensubtitles where we directly use their data with limited filtering).

The following table is necessary for this dataset to be indexed by search engines such as Google Dataset Search.