How can I speed up a very slow Font Book

First of all don't go into that directory and start deleting fonts. Assuming SIP will even let you, those are the fonts that ship with the Mac. Unless... have you installed some fonts there yourself?

Fonts should be stored in /Users/[USERNAME]/Library/Fonts.

And you can safely remove any and all fonts from that directory, just leave the directory itself.

If you have been installing fonts in the /System folder you probably want to go in there and move the ones you installed there to your /Users fonts folder. In case you were wondering here is a list of fonts that macOS ships with.

Should you somehow mess things up, Font Book.app has a function in the File menu called "Restore Standard Fonts..." which can be used if you think you may have messed up and deleted fonts you should not have deleted.