Changing character encoding from Unicode to Ascii

Solution 1:

With BBEdit this is easy. First open the file in BBEdit. If you let BBEdit install its command line tool you can even do this from Terminal with bbedit /path/to/filename. If the file has opened as the wrong encoding, select File > Reopen Using Encoding > correct encoding. I think it would be worth trying reopening using UTF-16 Little-Endian & UTF-16 Little-Endian, no BOM to see if either of those has the file open as desired. When you have the file correctly opened, select File > Save As.... In the Save As dialog box you can choose the desired encoding, & also the line ending type if that matters.

For dealing with inverted red question marks, probably null (ASCII 0), select Text > Zap Gremlins... for the dialog below...

Using the options shown there should give a state like what you see in TextEdit. Try different options on copies of a couple of your files.

Because BBEdit has a command line tool, you should be able script it once you have the correct options. BBEdit also works with AppleScript & Automator.

You can download BBEdit for free from the link I've given. It will start in demo mode, & when the demo mode expires it will continue to run in free mode where the features you need are still available.