Why does Axii sign not work?

I've found that you have to charge the sign for it to work. Just hold Q until you see the Axii animation on the enemy finishing. If you stop before the sign is fully charged it just won't work.