Saving predictions

Hi! In many setups, after we have successfully trained a model, we want to apply it (in batch-mode) and save the predictions to debug the model or run some official evaluation script etc. In such a setting, I think about the model as an annotator that reads a file that contains partial information (the predictions are missing) and produces a complete file. So far, I haven’t found an easy and clean solution to achieve this in general with AllenNLP.

Ideally, I’d imagine a command or something that reads a dataset, runs the model on it, takes the output from the model’s decode method, turns it back into an Instance, reorders them to undo batching effects, and then a custom “DatasetWriter” could write the model’s prediction to a file.

Does anything like this exist or how do you deal with such a setup?

Hi @namednil I don’t think we have any built-in functionality that does what you’re asking. Most of our effort to date has been around making it easier to build models rather than creating support for annotation pipelines. We’ve been interested in integrating our models with SpaCy’s API, but we haven’t yet found the time to do so.

That said–I’d love to hear if others have had this same problem and how they’ve addressed it in AllenNLP.

Hi @namednil, have you tried allennlp predict? See https://github.com/allenai/allennlp/blob/master/allennlp/commands/predict.py for details. You can use your existing dataset reader with the --use-dataset-reader flag and it also has options to batch and use a GPU. My only concern is the output format, which might not match what you need exactly, but it should be easy to modify the file I linked to get your desired behavior.

This won’t be as nice as the spaCy integrations @michaels alluded to, but maybe it’ll get you up and running. Good luck!

Thanks for your answers! I wasn’t aware of the --use-dataset-reader flag, that helps. It seems, some code of the Predictor class only gets invoked when that flag is not set. Perhaps that could use some documentation.

In case it helps anyone who might be looking at this: I ended up overwriting only predict_instance and predict_batch_instance in my subclass of Predictor to get readable predictions in the form of json dicts and I can now go from there.

1 Like