RoBERTa Embedder Example

Hi!

Would it be possible to get an example config that uses RoBERTa as the embedder? I have code that uses the BERT embedder right now, and I’m having some trouble converting it over to the pretrained_transformer / use RoBERTa.

Thanks!

I know that Oyvind, Kevin (Lin), and Jiangming (and maybe Dheeru) have used roberta in reading comprehension models. I don’t know that any of them have accounts on here, so I can’t tag them, but you should know how to ping them. If you get something from them, could you post it here or push it to a github repo somewhere so that others can find it too?

I just talked to Kevin (who is working with Oyvind) on something. They didn’t use the RoBERTa embedder, and instead just directly instantiated the RoBERTa model / tokenizer within their dataset reader and model.

Anyway, I managed to get something to run, but it probably isn’t correct—the pytorch pretrained BERT code complains that WARNING - pytorch_transformers.modeling_roberta - A sequence with no speci al tokens has been passed to the RoBERTa model. This model requires special tokens in order to work. Please specify add_special_tokens=True in your encoding.

Thoughts on what I might be missing? I would have expected the indexer to handle this already, but I might be wrong…

See here: https://github.com/allenai/allennlp/issues/3356. Also related: https://github.com/allenai/allennlp/issues/3359 and https://github.com/allenai/allennlp/issues/3224.