How to correctly save and load a model with custom CTC layer (Keras example)

The problem is not actually with Keras's saving methods. The characters set is inconsistent and does not keep ordering. Add the below code after creating the characters set to solve the issue:

characters = sorted(list(characters))

@Amirhosein, check out this function in the Horovod repository:

Serialize: https://github.com/horovod/horovod/blob/6f0bb9fae826167559501701d4a5a0380284b5f0/horovod/spark/keras/util.py#L115

Deserialize: https://github.com/horovod/horovod/blob/6f0bb9fae826167559501701d4a5a0380284b5f0/horovod/spark/keras/remote.py#L267

Example of use for deserialization: https://github.com/horovod/horovod/blob/6f0bb9fae826167559501701d4a5a0380284b5f0/horovod/spark/keras/remote.py#L118

If you are using custom objects like custom metrics or custom Loss function, you will need to use custom_object_scope as in the example.

It used a package called cloudpickle (https://pypi.org/project/cloudpickle/) under the hood to convert the KerasModel to a string and vice versa.