Unable to evaluate on metrics that did not train on

Hello! I’m pretty new to AllenNLP but am currently trying to use it for a project.

I have a few models that I have trained using specific metrics, and now realize that I would like to evaluate them on metrics not included in the training process. As of yet, I have not found a way to do this without retraining (which would not be ideal). What is the best way to do this? Is there something I’m missing? (If so, please send me info/a link to where I can find info!)

If more specific information is needed, please let me know and I can provide it.

Thank you!!

You might try out the allennlp evaluate command, which will run your model on an evaluation dataset. You can modify your code to include the new metric, then run that command.

Thank you for the response!

I’ve been using the evaluate command, as well as modifying my code to include the new metric and using the --overrides argument to include the new code. Currently it doesn’t seem like using the override command is changing anything.

As an example, one thing I am trying to do (along with the metric change mentioned above) is add a batchCallback in evaluation that allows me to save the predictions that the model makes after every batch of the evaluation. I have tried running the allennlp evaluate command with the following override to include the batch callback that I implemented, but nothing changes:
--overrides '{"trainer": {"batch_callbacks": [{"type": "savePreds_batchCallback"}]}}'

Any idea what is happening here? Am I missing something? Thanks!

Callbacks don’t get called in evaluate (or predict), so this is expected. If you want to save predictions, though, you can just use the predict command.