ext4 listing of files very slow in one specific directory that contained lots of files before

Directories only ever grow in size, not shrink. Try moving all those files out into a a temporary directory (like log2) then rmdir the old directory and rename the temp one as the new permanent one.

From man e2fsck:

-D     Optimize directories in filesystem.  This option causes e2fsck to try to optimize all  directories,
either  by  reindexing  them if the filesystem supports directory indexing,  or by sorting and com‐
pressing directories for smaller directories, or for filesystems using traditional linear  directo‐
ries.

Even without the -D option, e2fsck may sometimes optimize a few directories --- for example, if di‐
rectory indexing is enabled and a directory is not indexed and would benefit from being indexed, or
if the index structures are corrupted and need to be rebuilt.  The -D option forces all directories
in the filesystem to be optimized.  This can sometimes make them  a  little  smaller  and  slightly
faster to search, but in practice, you should rarely need to use this option.

The  -D  option  will  detect  directory  entries with duplicate names in a single directory, which
e2fsck normally does not enforce for performance reasons.

In other words, if you have a file system volume you can afford to take offline you can simply run e2fsck -Df <block_device> on it and it will shrink all the directories. Including the root directory in the file system, which you can't delete otherwise without re-formatting the volume. In my case it successfully shrunk the root directory on the volume from something like 56MiB (used to have more than 1M of files) to about 220KiB (about 3-4K of files).