iOS 11: Voice Control doesn't get turned off completely

Disable "Siri & Dictation" in restrictions.

1. Go to Settings > General > Restrictions.
2. Enable restrictions and set a passcode if you haven't already. Otherwise, enter your restrictions passcode.
3. Disable "Siri & Dictation".
   

I had the same problem with iOS 12 / iPhone X. Every time I tried to start/stop my music, Siri was activated when I pressed the headphone button a millisecond longer then normal. I even deactivated Siri in my settings!

So I activated Siri and noticed, there was another language/voice set than the one who appeared when I pressed the button.

The solution was to deactivate Siri with the exact voice/language setting, which was used when Siri showed up the whole time.