How do I keep a large directory that is located inside WSL2, in sync with another one in Windows?

Try changing your rsync flags (man rsync)

rsync -hvrPt (found in this thread) may be suitable for your purposes

Using cp will work but tools like rsync are made with the purpose of backing up in mind, and typically speed things up if you configure them right (for example, only syncing files that have been modified).