Hi, I’m getting this error periodically during training
WARNING - root - NaN or Inf found in input tensor. But my loss isn’t diverging, so I don’t think there’s an issue in my modeling code. I see that there was a GitHub issue on this https://github.com/allenai/allennlp/issues/3116, but I’m using the latest code on Master and it’s still happening. Is there any way to track down where this message is coming from? I tried doing
import warnings warnings.simplefilter("error")
so that the code would stop when this warning happened, but it didn’t work.