WARNING:tensorflow:sample_weight modes were coerced from ... to ['...']

Training an image classifier using .fit_generator() or .fit() and passing a dictionary to class_weight= as an argument.

I never got errors in TF1.x but in 2.1 I get the following output when starting training:

WARNING:tensorflow:sample_weight modes were coerced from
  ...
    to  
  ['...']

What does it mean to coerce something from ... to ['...']?

The source for this warning on tensorflow's repo is here, comments placed are:

Attempt to coerce sample_weight_modes to the target structure. This implicitly depends on the fact that Model flattens outputs for its internal representation.

Solution 1:

This seems like a bogus message. I get the same warning message after upgrading to TensorFlow 2.1, but I do not use any class weights or sample weights at all. I do use a generator that returns a tuple like this:

return inputs, targets

And now I just changed it to the following to make the warning go away:

return inputs, targets, [None]

I don't know if this is relevant, but my model uses 3 inputs, so my inputs variable is actually a list of 3 numpy arrays. targets is just a single numpy array.

In any case, it's just a warning. The training works fine either way.

Edit for TensorFlow 2.2:

This bug seems to have been fixed in TensorFlow 2.2, which is great. However the fix above will fail in TF 2.2, because it will try to get the shape of the sample weights, which will obviously fail with AttributeError: 'NoneType' object has no attribute 'shape'. So undo the above fix when upgrading to 2.2.

Solution 2:

I believe this is a bug with tensorflow that will happen when you call model.compile() with default parameter sample_weight_mode=None and then call model.fit() with specified sample_weight or class_weight.

From the tensorflow repos:

• fit() eventually calls _process_training_inputs()
• _process_training_inputs() sets sample_weight_modes = [None] based on model.sample_weight_mode = None and then creates a DataAdapter with sample_weight_modes = [None]
• the DataAdapter calls broadcast_sample_weight_modes() with sample_weight_modes = [None] during initialization
• broadcast_sample_weight_modes() seems to expect sample_weight_modes = None but receives [None]
• it asserts that [None] is a different structure from sample_weight / class_weight, overwrites it back to None by fitting to the structure of sample_weight / class_weight and outputs a warning

Warning aside this has no effect on fit() as sample_weight_modes in the DataAdapter is set back to None.

Note that tensorflow documentation states that sample_weight must be a numpy-array. If you call fit() with sample_weight.tolist() instead, you will not get a warning but sample_weight is silently overwritten to None when _process_numpy_inputs() is called in preprocessing and receives an input of length greater than one.