Pretrain ELMo Model PubMed

Hi there,
One of the various pretrain models available from allenNLP is PubMed. Has the training of the model been done with exclusive use of the abstracts of the PubMed repository or have the Mesh terms also been included? Could this model and in particular the vectors it produces be used in a Mesh Indexing task?

Thanks

I’m not sure which model you are referring to. Can you post a link or something for me to identify it?

The ELMo PubMed found here

I’d be surprised if they used MeSH terms during pre-training. They don’t have a context, and the whole point of ELMo was to create better word embeddings by using context.

That said, it’s probably still your best bet, since the vocabulary will overlap quite a bit between abstracts and MeSH terms.

Thanks for your reply,
At least does any general information about how the PubMed model was trained exist?

It was trained as described in https://arxiv.org/abs/1802.05365, but with PubMed abstracts as text.