I was looking at MessagePack for communicating to and from my STM32F1-based microcontroller project from the PC controller software I'd be writing. At least the official C library was not optimized for memory usage and code size. I also considered BSON, but it also lacked suitable libraries.
So I ended up using JSON. Yes the message sizes are larger in byte size with JSON but using the jsmn[1] parser I could avoid dynamic memory usage and code size was small. The jsmn parser outputs an array of tokens that point to the buffer holding the message (ie start and end of key name etc), so overhead is quite limited.
For JSON output I modified json-maker[2]. It already allowed for static memory usage and rather small code size, but I changed it to support a write-callback so I could send output directly over the data link, so I didn't have to buffer the whole message. This is nice when sending larger arrays of data for example.
Combined it took about 10kB of program (flash) memory, of which float to string support is about 50%. Memory usage is determined by how larger incoming messages I'd need, for now 1kB is plenty.
A nice advantage of using JSON is that it's very easy to debug over UART.
Though having compact messages would be nice for wireless stuff and similar, so does anyone know of a MessagePack C/C++ library that is microcontroller friendly?
It can be built to a very small code size, especially when you disable libc, allocations, etc. There are some people using it on embedded devices like Arduino. There's someone working on a port to 8-bit microcontrollers that don't have a 64-bit float, so you may want to look into that as well; see the open issue for it on the GitHub link above.
Protobufs/nanopb would be my go-to for minimal message size.
If you want small code size, CBOR seems like a good bet:
> The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation. [1]
This [2] C-implementation fits in under 1KiB of ARM code.
CBOR is also used on WebAuthn, usage in a web spec means to me that someone smart considered it a sane choice -- and more importantly that the format is here to stay.
It's great CBOR is accepted in wider area, but I am personally curious why WebAuthn choose CBOR instead of JSON. WebAuthn is a web browser feature, and why W3C would introduce a new data exchange format in their specs? Maybe WebAuthn needed a binary data type?
That’s strange because CBOR is almost literally msgpack that got an RFC and has extensions. I cant remember what MsgPack does for online streaming and indefinite lengths.
Looked at it again, seems memory management is a bit of an issue, it supports memory allocation callback but not just handing it a buffer to work with (though I guess allocation should be predictable).
Also I don't know how they got "code sizes appreciably under 1 KiB". On my STM32F1 release mode with -Os it adds about 12kB.
For reference, I’m using TimyCBOR because it’s include with Amazon FreeRTOS.
You’re on your own for malloc, which for me is great because FreeRTOS Heap4 management is quite good. So I malloc an object I’m decoding into and parse away.
There are two options parsing arrays and strings/bytestrings and I just chose the option where I specify the pointer to use, vs them using normal malloc then free() later.
I really like this setup. I made a deinit(bad_message) that works anywhere it failed (parse, validate, eval, etc), goes through and looks for pointers that I previously would have malloc’ed.
There is another popular library but I forget what it’s called.
- libmpack serialization/deserialization API is callback-based, making it simple to serialize/deserialize directly from/to application-specific objects
- libmpack does no allocation at all, and provides some helpers to simplify dynamic allocation by the user, if required.
This library is very small and you need to implement your own I/O via well-defined functions. The parser itself does not use any library (including libc):
So I ended up using JSON. Yes the message sizes are larger in byte size with JSON but using the jsmn[1] parser I could avoid dynamic memory usage and code size was small. The jsmn parser outputs an array of tokens that point to the buffer holding the message (ie start and end of key name etc), so overhead is quite limited.
For JSON output I modified json-maker[2]. It already allowed for static memory usage and rather small code size, but I changed it to support a write-callback so I could send output directly over the data link, so I didn't have to buffer the whole message. This is nice when sending larger arrays of data for example.
Combined it took about 10kB of program (flash) memory, of which float to string support is about 50%. Memory usage is determined by how larger incoming messages I'd need, for now 1kB is plenty.
A nice advantage of using JSON is that it's very easy to debug over UART.
Though having compact messages would be nice for wireless stuff and similar, so does anyone know of a MessagePack C/C++ library that is microcontroller friendly?
[1]: https://github.com/zserge/jsmn
[2]: https://github.com/rafagafe/json-maker