How do I permanently remove a default Powershell alias?

Skip this step if you already have a Powershell profile:

New-Item $profile -force -itemtype file

Then edit your profile:

notepad $profile

Add the following line to it:

remove-item alias:curl

Save, close notepad and reload the profile with the command below or close and open Powershell to apply the profile:

. $profile

I'd stick with using curl.exe instead of just curl. This approach doesn't rely on PowerShell running with particular profile (and PowerShell could be run with no profile at all using -NoProfile switch).