Which seeds have to be set where to realize 100% reproducibility of training results in tensorflow?
The best solution which works as of today with GPU is to install tensorflow-determinism with the following:
pip install tensorflow-determinism
Then include the following code to your code
import tensorflow as tf
import os
os.environ['TF_DETERMINISTIC_OPS'] = '1'
source: https://github.com/NVIDIA/tensorflow-determinism
One possible reason is that when constructing the model, there are some code using numpy.random module. So maybe you can try to set the seed for numpy, too.