What does the standard Keras model output mean? What is epoch and loss in Keras?

I have just built my first model using Keras and this is the output. It looks like the standard output you get after building any Keras artificial neural network. Even after looking in the documentation, I do not fully understand what the epoch is and what the loss is which is printed in the output.

What is epoch and loss in Keras?

(I know it's probably an extremely basic question, but I couldn't seem to locate the answer online, and if the answer is really that hard to glean from the documentation I thought others would have the same question and thus decided to post it here.)

Epoch 1/20
1213/1213 [==============================] - 0s - loss: 0.1760     
Epoch 2/20
1213/1213 [==============================] - 0s - loss: 0.1840     
Epoch 3/20
1213/1213 [==============================] - 0s - loss: 0.1816     
Epoch 4/20
1213/1213 [==============================] - 0s - loss: 0.1915     
Epoch 5/20
1213/1213 [==============================] - 0s - loss: 0.1928     
Epoch 6/20
1213/1213 [==============================] - 0s - loss: 0.1964     
Epoch 7/20
1213/1213 [==============================] - 0s - loss: 0.1948     
Epoch 8/20
1213/1213 [==============================] - 0s - loss: 0.1971     
Epoch 9/20
1213/1213 [==============================] - 0s - loss: 0.1899     
Epoch 10/20
1213/1213 [==============================] - 0s - loss: 0.1957     
Epoch 11/20
1213/1213 [==============================] - 0s - loss: 0.1923     
Epoch 12/20
1213/1213 [==============================] - 0s - loss: 0.1910     
Epoch 13/20
1213/1213 [==============================] - 0s - loss: 0.2104     
Epoch 14/20
1213/1213 [==============================] - 0s - loss: 0.1976     
Epoch 15/20
1213/1213 [==============================] - 0s - loss: 0.1979     
Epoch 16/20
1213/1213 [==============================] - 0s - loss: 0.2036     
Epoch 17/20
1213/1213 [==============================] - 0s - loss: 0.2019     
Epoch 18/20
1213/1213 [==============================] - 0s - loss: 0.1978     
Epoch 19/20
1213/1213 [==============================] - 0s - loss: 0.1954     
Epoch 20/20
1213/1213 [==============================] - 0s - loss: 0.1949

Solution 1:

Just to answer the questions more specifically, here's a definition of epoch and loss:

Epoch: A full pass over all of your training data.

For example, in your view above, you have 1213 observations. So an epoch concludes when it has finished a training pass over all 1213 of your observations.

Loss: A scalar value that we attempt to minimize during our training of the model. The lower the loss, the closer our predictions are to the true labels.

This is usually Mean Squared Error (MSE) as David Maust said above, or often in Keras, Categorical Cross Entropy

What you'd expect to see from running fit on your Keras model, is a decrease in loss over n number of epochs. Your training run is rather abnormal, as your loss is actually increasing. This could be due to a learning rate that is too large, which is causing you to overshoot optima.

As jaycode mentioned, you will want to look at your model's performance on unseen data, as this is the general use case of Machine Learning.

As such, you should include a list of metrics in your compile method, which could look like:

model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])

As well as run your model on validation during the fit method, such as:

model.fit(data, labels, validation_split=0.2)

There's a lot more to explain, but hopefully this gets you started.

Solution 2:

One epoch ends when your model had run the data through all nodes in your network and ready to update the weights to reach optimal loss value. That is, smaller is better. In your case, as there are higher loss scores on higher epoch, it "seems" the model is better on first epoch.

I said "seems" since we can't actually tell for sure yet as the model has not been tested using proper cross validation method i.e. it is evaluated only against its training data.

Ways to improve your model:

• Use cross validation in your Keras model in order to find out how the model actually perform, does it generalize well when predicting new data it has never seen before?
• Adjust your learning rate, structure of neural network model, number of hidden units / layers, init, optimizer, and activator parameters used in your model among myriad other things.

Combining sklearn's GridSearchCV with Keras can automate this process.