Implementing Unicode SMS support

leadenmoth · April 13, 2020, 8:06pm

Hi, CircuitMess team! I was checking the source code to see what it’d take to implement 16-bit SMS encoding support (I get a bunch of texts in cyrillic, they come out as asterisks, and by the time they are saved into a json on the SD card, they are already irreversibly mis-parsed). Looks like you guys don’t use SIM800 native text mode and instead have your own PDU decoder. However, I wasn’t able to figure out exactly which module I need to modify, because these two seem to have some overlap in functionality (plus the arduino_pdu_decoder one seems quite a bit different from the version elsewhere on GitHub):

github.com

CircuitMess/CircuitMess-Ringo/blob/master/src/utility/arduino_pdu_decoder/pdu_decoder.cpp

//Implementation of a pdu decoder.

#include "Arduino.h"
#include "pdu_decoder.h"
struct MapAsciiToGSM7bit
{
	char const* GSM7bitValue;
	char AsciiValue;
};
MapAsciiToGSM7bit my_map[93] ={
  {"21", '!'},
  {"22", '"'},
  {"23", '#'},
  {"02", '$'},
  {"25", '%'},
  {"26", '&'},
  {"27", '\''},
  {"28", '('},
  {"29", ')'},
  {"2A", '*'},

This file has been truncated. show original

github.com

CircuitMess/CircuitMess-Ringo/blob/master/src/utility/pduParse/pduParse.cpp

/* 
Copyright (C) 2010- Alexander Chudnovets <effractor@gmail.com>

Based on SMS Server Tools 3
Copyright (C) 2006- Keijo Kasvi
http://smstools3.kekekasvi.com/

This program is free software unless you got it under another license directly
from the author. You can redistribute it and/or modify it under the terms of
the GNU General Public License as published by the Free Software Foundation.
Either version 2 of the License, or (at your option) any later version.
*/
#include <Arduino.h>
#include <stdlib.h>
#include <stdio.h>
#include <iostream>
#include <fstream>
#include <string.h>
#include <ctype.h>
#include <iconv.h>

This file has been truncated. show original

Could you give me some pointers?
Also, I’m guessing after I figure out this part, I’ll also need to figure out the fonts somewhere?..

emilgajsak · April 14, 2020, 3:38pm

Greetings,

I’m just going to get straight to the point.
The SIM800 PDU parser works fine for UTF8 characters, but if you have a Unicode character in the SMS, the SIM800 module reverts back to reporting the raw SMS PDU code. We needed to make our own custom PDU parsing functions, mostly by modifying this project’s code.

We refrained from using UCS2 (Unicode) encoding in our code, mostly because of the font limitations.

A nice place to start would be the MAKERphone::pduDecode function, mostly the parts where we’re decoding the PDU in a different way depending on the codingScheme flag (it shows what coding scheme the SMS is written in).

To save a Unicode SMS you should take a look at the block of code starting from line 6473 in MAKERphone.cpp. There we purposefully refrained from saving any Unicode chars and replacing them with asterisks.

The font, on the other hand, seems like a much more limiting factor, but there should be some Cyrillic fonts somewhere on the web.

Thanks for taking an interest in the project. I hope this gives you an idea of where to start.

Cheers

leadenmoth · June 28, 2020, 8:30pm

Sorry for disappearing, only recently got the time to look into this. Thanks for the write-up, with it I managed to at least successfully save incoming unicode messages to messages.json to read on PC.
Now, the font part is indeed trickier. To minimize changes, I’d prefer to just generate a replacement for Font7.c with more characters. However, Adafuit’s fontconverter doesn’t accept multiple subsets. Without understanding the structure of .c/.h font I can’t make a reasonably small file that would cover both Latin and Cyrillic characters. On the other hand, taking all of the first 1327 characters (all Latin, Greek and Cyrillic blocks) nets a 50kb+ file. LittleVGL’s converter allows multiple subsets, but the output format seems different from Adafruit’s, and again, I don’t understand these files enough to tweak manually. Any help?