Question about calculating n-gram probabilities


I am new to Allen NLP. I would like to know if there is a class that would allow me to calculate the log probability of an n-gram given a particular corpus?


Hi @andrewfr, I don’t believe we have that functionality. We’re mostly focused on deep models. Perhaps could be of use?

Hi Brendan:

Thanks for the response. I found the following answer in Stackoverflow that addresses what I want:

The Moses statistical translation package also has the tools I need. A little inconvenient since I am programming in Python (I am using the constituency parser) but it should do what I want. Here is a link:

Maybe it would be nice to add to the Allen NLP?

Again, thanks!


Hi Andrew,

Glad you have something working! I think we’re unlikely to implement this in AllenNLP given that we’re focused on deep models and there’s already Python support from NLTK. That said, if you’re interested in making a feature request for this anything else you can file a Github issue here: