Why is L2ARC not giving more speedup for directory traversal?
If you are using ZFS on Linux, the slowness probably depends on xattr default setting (xattr=on), which stored extended attributes inside hidden directories/files.
Try setting chaning xattr to sa, issuing something like zfs set xattr=sa <pool/fs>. However, it affect only newly create files; so you need to remove/reload all files to get a significant speed up.
In other words, give it a small-scale try before reloading all your files.