How to delete all directories except directories pointed by symlink

I have below directory structure.

drwxr-xr-x 5 ec2-user ec2-user 4096 May 22 07:03 TEEST  
drwxr-xr-x 5 ec2-user ec2-user 4096 May 24 05:33 BEE-TES  
drwxr-xr-x 5 ec2-user ec2-user 4096 May 24 10:53 ONE  
drwxr-xr-x 5 ec2-user ec2-user 4096 May 25 09:50 TWO  
lrwxrwxrwx 1 ec2-user ec2-user   28 May 25 09:50 previous -> TWO  
drwxr-xr-x 5 ec2-user ec2-user 4096 Jun  1 11:00 TEMP  
lrwxrwxrwx 1 ec2-user ec2-user   28 Jun  1 11:00 current -> TEMP

How to remove all the directories, except those pointed by symlinks ?

{ find . -type l -exec readlink {} \; ; find . -type d -exec basename {} \; ; } | sort | uniq -u | tail -n +2 | xargs rmdir

In slow-mo:

• find . -type l -exec readlink {} \; finds the links and lists the corresponding directories
• find . -type d -exec basename {} \; finds and lists the directories
• the output of the commands in braces lists all directories once, except those that are the target of links that appear twice
• sort | uniq -u | tail -n +2 keeps only the unique names (so, those directories that are not linked to) and removes .
• xargs rmdir drops the directories. Replace with rm -r to remove their contents.