Android: Unicode/Charset problems when sending an SMS (sendTextMessage)

Basically I have a working application that sends an SMS after receiving an SMS.

Everything works fine, except when the SMS text to send has "special chars", ie "é,à,í,ç", etc.

I've tried many things including charset conversion but I simply can't make it work... the msgText always comes back with charset encoding problems.

Here's the part where the message is sent:

if (msgText.length() > 160) {
    ArrayList msgTexts = sm.divideMessage(msgText);
    sm.sendMultipartTextMessage(PhoneNumber, null, msgTexts, null, null);
} else {
    try {
        sm.sendTextMessage(PhoneNumber, null, msgText, null, null);
    } catch (IllegalArgumentException e) {
        e.printStackTrace();
    }
}

Here's the charset conversion function I tried (but didn't help), that I applied on msgText:

public static String formatCharset(String txtInicial) {
    //-- Please notice this is just for reference, I tried every charset from/to conversion possibility. Even stupid ones and nothing helped.

    /*try {//-- Seems simpler, it should do the same as below, but didn't help
        msgText = new String(msgText.getBytes("UTF-8"), "ISO-8859-1");
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
    }*/

    Charset charsetOrigem = Charset.forName("UTF-8");
    CharsetEncoder encoderOrigem = charsetOrigem.newEncoder();
    Charset charsetDestino = Charset.forName("ISO-8859-1");
    CharsetDecoder decoderDestino = charsetDestino.newDecoder();

    String txtFinal = "";

    try {
        ByteBuffer bbuf = encoderOrigem.encode(CharBuffer.wrap( txtInicial ));
        CharBuffer cbuf = decoderDestino.decode(bbuf);
        txtFinal = cbuf.toString();
    } catch (CharacterCodingException e) {
        e.printStackTrace();
    }

    if (txtFinal.length() == 0) txtFinal = txtInicial;

    return txtFinal;
}

Near desperation I even tried the solution for unicode messaging in here (didn't help as well):

http://since2006.com/blog/android-send-unicode-message/

Anyway, here's the (cleaned up - package is com.THE.APPLICATION, main activity is MAINACT) LogCat for when it crashes (when trying to send the message, after receiving one):

WARN/dalvikvm(28218): threadid=1: thread exiting with uncaught exception (group=0x4001d7f0)
ERROR/AndroidRuntime(28218): FATAL EXCEPTION: main
ERROR/AndroidRuntime(28218): java.lang.RuntimeException: Error receiving broadcast Intent { act=android.provider.Telephony.SMS_RECEIVED (has extras) } in com.THE.APPLICATION.SMSReceiver@44acd880
ERROR/AndroidRuntime(28218):     at android.app.ActivityThread$PackageInfo$ReceiverDispatcher$Args.run(ActivityThread.java:905)
ERROR/AndroidRuntime(28218):     at android.os.Handler.handleCallback(Handler.java:587)
ERROR/AndroidRuntime(28218):     at android.os.Handler.dispatchMessage(Handler.java:92)
ERROR/AndroidRuntime(28218):     at android.os.Looper.loop(Looper.java:123)
ERROR/AndroidRuntime(28218):     at android.app.ActivityThread.main(ActivityThread.java:4627)
ERROR/AndroidRuntime(28218):     at java.lang.reflect.Method.invokeNative(Native Method)
ERROR/AndroidRuntime(28218):     at java.lang.reflect.Method.invoke(Method.java:521)
ERROR/AndroidRuntime(28218):     at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:868)
ERROR/AndroidRuntime(28218):     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:626)
ERROR/AndroidRuntime(28218):     at dalvik.system.NativeStart.main(Native Method)
ERROR/AndroidRuntime(28218): Caused by: java.lang.NullPointerException
ERROR/AndroidRuntime(28218):     at android.os.Parcel.readException(Parcel.java:1253)
ERROR/AndroidRuntime(28218):     at android.os.Parcel.readException(Parcel.java:1235)
ERROR/AndroidRuntime(28218):     at com.android.internal.telephony.ISms$Stub$Proxy.sendText(ISms.java:369)
ERROR/AndroidRuntime(28218):     at android.telephony.SmsManager.sendTextMessage(SmsManager.java:87)
ERROR/AndroidRuntime(28218):     at com.THE.APPLICATION.MAINACT.sendMessage(MAINACT.java:214)
ERROR/AndroidRuntime(28218):     at com.THE.APPLICATION.SMSReceiver.onReceive(SMSReceiver.java:24)
ERROR/AndroidRuntime(28218):     at android.app.ActivityThread$PackageInfo$ReceiverDispatcher$Args.run(ActivityThread.java:892)
ERROR/AndroidRuntime(28218):     ... 9 more

Sample of message text to send with issues:

VERBOSE/debug_tag(28218): msgText is: possível.

So, it reads possível when it should be possível

Please some enlightened soul help me out. He/She'll have a special place in my heart! :)

Edit: If the special place in my heart doesn't cut it, I'm willing to pay a few bucks for a working solution...

Solution 1:

Ok, this seems to have been solved by simply using sendMultipartTextMessage instead of sendTextMessage for the messages.

Who would've thought... it kind of makes sense because unicode characters use more "space" than "normal" ones.

Solution 2:

I have used this code to convert UTF-8 characters into ASCII. Then sending of SMS works and I can use 160 characters:

private static final String PLAIN_ASCII = "AaEeIiOoUu" // grave
        + "AaEeIiOoUuYy" // acute
        + "AaEeIiOoUuYy" // circumflex
        + "AaOoNn" // tilde
        + "AaEeIiOoUuYy" // umlaut
        + "Aa" // ring
        + "Cc" // cedilla
        + "OoUu" // double acute
;

private static final String UNICODE = "\u00C0\u00E0\u00C8\u00E8\u00CC\u00EC\u00D2\u00F2\u00D9\u00F9"
        + "\u00C1\u00E1\u00C9\u00E9\u00CD\u00ED\u00D3\u00F3\u00DA\u00FA\u00DD\u00FD"
        + "\u00C2\u00E2\u00CA\u00EA\u00CE\u00EE\u00D4\u00F4\u00DB\u00FB\u0176\u0177"
        + "\u00C3\u00E3\u00D5\u00F5\u00D1\u00F1"
        + "\u00C4\u00E4\u00CB\u00EB\u00CF\u00EF\u00D6\u00F6\u00DC\u00FC\u0178\u00FF"
        + "\u00C5\u00E5" + "\u00C7\u00E7" + "\u0150\u0151\u0170\u0171";

// remove accentued from a string and replace with ascii equivalent
public static String convertNonAscii(String s) {
    if (s == null)
        return null;
    StringBuilder sb = new StringBuilder();
    int n = s.length();
    for (int i = 0; i < n; i++) {
        char c = s.charAt(i);
        int pos = UNICODE.indexOf(c);
        if (pos > -1) {
            sb.append(PLAIN_ASCII.charAt(pos));
        } else {
            sb.append(c);
        }
    }
    return sb.toString();
}

Solution 3:

I am using this command line:

Html.fromHtml(new String(myString.getBytes("UTF-8"))).toString();

and my SMS message looks perfect.