Why does it take a long time to get a response from the retrofit in some cases?

If sometimes it works fast sometimes with delay - it looks like issue is not in your codebase. Maybe the issue on the server side or in your network connection. To check this case you can use Fiddler or Charles to sniff your application traffic. And if response in the sniffing program will have the same time then issue is not related to your codebase.