Tracking down warnings from logger

Hi, I’m getting this error periodically during training WARNING - root - NaN or Inf found in input tensor. But my loss isn’t diverging, so I don’t think there’s an issue in my modeling code. I see that there was a GitHub issue on this https://github.com/allenai/allennlp/issues/3116, but I’m using the latest code on Master and it’s still happening. Is there any way to track down where this message is coming from? I tried doing

import warnings
warnings.simplefilter("error")

so that the code would stop when this warning happened, but it didn’t work.

https://stackoverflow.com/questions/22373927/get-traceback-of-warnings looks pretty relevant.

You could also modify the source (I’m guessing that’s in PyTorch?) to dump out more information when that warning is triggered.

I found the issue: https://github.com/allenai/allennlp/blob/7afe9712f502dfdc6941bb03d3029a7eb6e0bca9/allennlp/training/tensorboard_writer.py#L10.
This fails if param.data.numel() is 0. This is admittedly a goofy edge case, because if there’s no elements why is this parameter part of the model? Still, I’m not sure this should throw a warning.
Is the warning the desired behavior? If not, I can make a PR.

We’d welcome a PR which avoids this warning in this edge case. Thanks!