How to "reset" tensorboard data after killing tensorflow instance

Solution 1:

Note: The solution you've posted (erase TensorBoard's log files and kill the process) will work, but it isn't preferred, because it destroys historical information about your training.

Instead, you can have each new training job write to a new subdirectory (of your top-level log directory). Then, TensorBoard will consider each job a new "run" and will create a nice comparison view so you can see how the training differed between iterations of your model.

In the following an example from https://www.tensorflow.org/tensorboard/get_started:

model = create_model()
...
model.compile(...)

log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

model.fit(..., callbacks=[tensorboard_callback])

Solution 2:

Yes, I believe ultimately this aspect is positive.
As an example, in my script I automate new run logs via datetime:

from datetime import datetime
now = datetime.now()
logdir = "tf_logs/.../" + now.strftime("%Y%m%d-%H%M%S") + "/"

Then when running TensorBoard you can click the different runs on and off provided you ran TensorBoard in the parent directory.

If you know you don't care about a previous run and want it out of your life, then yes, you need to remove the event files and restart TensorBoard.

Solution 3:

Ok, for some reason it didn't work before but now it did:

You must erase Tensorboard's log files AND kill its process

After killing the process run fuser 6006/tcp -k to free port 6006 (if you're in linux) and fire tensorboard again.

Solution 4:

I had a similar problem, but with a duplication of computational graphs: they've just added in tensorboard when I called

writer.add_graph(graph=sess.graph)

In my case, it wasn't about log files but about Jupyter Notebook context.

I figured out that after multiple runs of a Jupyter cell with a Graph definition the graph hasn't been reset but appeared in the context as a duplicate, so I added

tf.reset_default_graph()

before the start of building a computational graph.

Hope it will help.