Apple Books won't download any books/files saved in iCloud only on Mac (macOS Big Sur)

This seems to be related to the storage optimisation feature for iCloud (maybe it is somehow stuck or blocking these downloads). It can be found under System Preferences -> Apple ID -> Optimise Mac Storage. After turning the future off and on all downloads in Apple Books now work again.