Training SRL with XLM-RoBERTa

Hi all,

I’m trying to build an SRL model using xlm-roberta-base. I’ve converted the semantic_role_labeling.py dataset reader to use the roberta-style special tags, roberta vocab pulls, and roberta tokenization. My vocabulary seems to build correctly and writes to disk the way I would expect. However, I’m still running into problems with training.

I adapted the srl_bert.py model:

self.bert_model = AutoModel.from_pretrained(bert_model)

I’m getting errors when I get here:

    bert_embeddings, _ = self.bert_model(
        input_ids=util.get_token_ids_from_text_field_tensors(tokens),
        token_type_ids=verb_indicator,
        attention_mask=mask,
    )

I ran it on my CPU to get the full error report that I’ll post below. Anybody have any ideas on what to do here? Thanks!

Traceback (most recent call last):
  File "./allennlp/__main__.py", line 22, in <module>
    run()
  File "./allennlp/__main__.py", line 18, in run
    main(prog="allennlp")
  File "/home/spenc/Dropbox/allennlp/allennlp/commands/__init__.py", line 93, in main
    args.func(args)
  File "/home/spenc/Dropbox/allennlp/allennlp/commands/train.py", line 145, in train_model_from_args
    dry_run=args.dry_run,
  File "/home/spenc/Dropbox/allennlp/allennlp/commands/train.py", line 204, in train_model_from_file
    dry_run=dry_run,
  File "/home/spenc/Dropbox/allennlp/allennlp/commands/train.py", line 267, in train_model
    dry_run=dry_run,
  File "/home/spenc/Dropbox/allennlp/allennlp/commands/train.py", line 464, in _train_worker
    metrics = train_loop.run()
  File "/home/spenc/Dropbox/allennlp/allennlp/commands/train.py", line 523, in run
    return self.trainer.train()
  File "/home/spenc/Dropbox/allennlp/allennlp/training/trainer.py", line 648, in train
    train_metrics = self._train_epoch(epoch)
  File "/home/spenc/Dropbox/allennlp/allennlp/training/trainer.py", line 423, in _train_epoch
    loss = self.batch_loss(batch, for_training=True)
  File "/home/spenc/Dropbox/allennlp/allennlp/training/trainer.py", line 333, in batch_loss
    output_dict = self._pytorch_model(**batch)
  File "/home/spenc/anaconda3/envs/xlmr/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/spenc/Dropbox/allennlp/allennlp/models/srl_xlmr.py", line 115, in forward
    attention_mask=mask,
  File "/home/spenc/anaconda3/envs/xlmr/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/spenc/anaconda3/envs/xlmr/lib/python3.7/site-packages/transformers/modeling_bert.py", line 783, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/home/spenc/anaconda3/envs/xlmr/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/spenc/anaconda3/envs/xlmr/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 65, in forward
    input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
  File "/home/spenc/anaconda3/envs/xlmr/lib/python3.7/site-packages/transformers/modeling_bert.py", line 175, in forward
    token_type_embeddings = self.token_type_embeddings(token_type_ids)
  File "/home/spenc/anaconda3/envs/xlmr/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/spenc/anaconda3/envs/xlmr/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/spenc/anaconda3/envs/xlmr/lib/python3.7/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to access index 1 out of table with 0 rows. at /opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418