Why do we build vocabulary from test dataset?

When I reading the source code, I notice that we build vocabulary from all datasets,which also include test dataset.
I think we should not include test dataset, because if we evaluate our model on test dataset, all words will be “known”.
That means no word will be “_oov_token”, I think this will cause some problems.
Do I understand right?