Is "Never mind" a phrasal verb?

'Never' is certainly not used with the usual meaning here; 'Never mind what he said to you last week' is very different from 'Never worry about what he said to you'. I'd agree that there is a cohesiveness between 'never' and 'mind'.

In fact, never mind is listed by the dictionaries I've checked in as a 'phrase' or an 'idiom', eg by the Cambridge Dictionary of American Idioms:

never mind (somebody/something): do not worry about someone or something

“I lost that wallet you gave me.” “Well, never mind, I can always buy you another one.”

Never mind Susan – she can get a ride home with someone else.

I avoid the term 'phrasal verb' as being too ambiguous, and this is different again from what most people using the terms would include as a 'phrasal verb' (* I never mind etc) but this is surely better regarded as a multi-word construction than 'adverb + verb'.