Hi! In many setups, after we have successfully trained a model, we want to apply it (in batch-mode) and save the predictions to debug the model or run some official evaluation script etc. In such a setting, I think about the model as an annotator that reads a file that contains partial information (the predictions are missing) and produces a complete file. So far, I haven’t found an easy and clean solution to achieve this in general with AllenNLP.
Ideally, I’d imagine a command or something that reads a dataset, runs the model on it, takes the output from the model’s decode method, turns it back into an Instance, reorders them to undo batching effects, and then a custom “DatasetWriter” could write the model’s prediction to a file.
Does anything like this exist or how do you deal with such a setup?