I’m attempting to use an
allennlp.data.dataset_readers.language_modeling.LanguageModelingReader to read data to train an
However, it appears that the APIs for these two classes do not quite agree: the
LanguageModelingReader prepares an instance with two keys
output_tokens, corresponding to the input and label sequences, whereas the
LanguageModel expects just a single field
source, which I assume should contain all the tokens in the document.
It seems like these classes should not be used together. What is the best practice work-around? Should I just write my own basic reader?