GNU ARM Embedded Toolchain
distributions include a non-polluting reduced-size runtime library called newlib (or newlib-nano for
the smallest variant).
Unfortunately, newlib internally uses free storage (malloc/free)
in startling places within the C runtime library.
Thus, newlib free storage routines get dragged in and used unexpectedly.
Ooops: Not MISRA-compliant, if you believe in that stuff.
The most common functions that bite unsuspecting embedded developers are
Here's a list of newlib functions with reentrancy support (and hence using malloc).
newlib requires platform-specific support to:
- provide thread-safety,
- switch thread contexts, and
- obtain memory to be doled out and managed
by malloc/free/etc (
Here's the detailed list of support functions required by newlib.
If this support is properly implemented, newlib works well in a threaded environment like FreeRTOS. CubeMX does not properly set up these support functions for a FreeRTOS project.
See: Bug Report to STM regarding CubeMX FreeRTOS projects and the STM bug discussion below.
Example newlib use of malloc/malloc_r:
Placing a breakpoint within malloc_r (innermost malloc routine) shows the following:
- newlib 3.0 does not use malloc before main() is called (unlike earlier NXP/newlib 2.5 startup, which malloc'd 5k). Verified for all four newlib variants.
- With simple decimal output, sprintf does not call malloc
- For %f output, sprintf does
4 malloc totalling ~200 bytes
(only the first time it is called per task/thread).
%f also requires proper linker arguments for float support.
- printf or similar function expected to do IO (as opposed to string operation) allocates an IO control structure of 428 bytes.
Tip: How can I see who is calling malloc ?
The basic steps are:
- create a wrapper function malloc_r,
- add the wrapper options to the linker, and
- place a breakpoint in malloc_r.
A complete example with instructions is included in the code below.
newlib under the hood - concurrency
newlib maintains information it needs to support each separate context (thread/task/ISR) in a reentrancy structure. This includes things like a thread-specific errno, thread-specific pointers to allocated buffers, etc. The active reentrancy structure is pointed at by global pointer _impure_ptr, which initially points to a statically allocated structure instance. If you use no threading or tasks, and you don't use malloc/free/etc or any of the reentrancy-dependent functions in multiple execution contexts, nothing more is required to use newlib in your application (newlib will maintain its required info in the single static structure). If your application or any library you use requires malloc/free/etc or any of the reentrancy-dependent functions in multiple contexts, newlib requires:
- concurrency protection for malloc/free/etc. The free storage pool will be corrupted if multiple threads call into these functions concurrently! To prevent reentrant execution of malloc/free/etc routines newlib requires hook procedures __malloc_lock/unlock If an RTOS-based application does not replace newlib's complete internal malloc family (FreeRTOS does not), _malloc_lock/unlock must be provided for thread safety.
- multiple reentrancy structures (one per context), and a mechanism to create, initialize, and cleanup these structures, plus switching _impure_ptr to point at the correct reentrancy structure each time the context changes.
WTF? Why are newlib's dtoa, ftoa, and sprintf calling malloc???
To quote Steele and White, Isn't it a pain when you ask a computer to divide 1.0 by 10.0 and it prints 0.0999999? To summarize and over-simplify a very long story, in 1990 my friend Will Clinger published a seminal paper giving a solution to this problem, but Will's solution required arbitrary precision and hence storage. See Will's retrospective on his seminal 1990 paper and subsequent developments. David Gay published an improved implementation (based on Will's paper), which was extremely widely adopted. newlib uses David Gay's code for dtoa and ftoa (using malloc), and sprintf uses dtoa and ftoa. Keith Packard's picolibc implements the no-malloc precise Ryū algorithm. There has been some recent discussion on the newlib mailing list including details of Gay's algorithm and maybe incorporating no-malloc Ryū algorithm from picolibc into newlib.
Really now, aren't you sorry you asked me why?
FreeRTOS support for newlibFreeRTOS provides support for newlib's context management. In
#define configUSE_NEWLIB_REENTRANT 1 // Required for thread-safety of newlib sprintf, strtok, etc...
With this option FreeRTOS does the following (in task.c):
- For each task, allocate and initialize a newlib reentrancy structure in the task control block (TCB).
This adds 96* bytes overhead per task (* size hugely dependant on newlib build options), plus anything else newlib allocates as needed.
- Each task switch, set _impure_ptr to point to the newly active task's reentrancy structure.
- On task destruction, clean up the reentrancy structure (help newlib free any associated memory).
Careful: Depending on how your supplier built newlib, and whether you are using nano,
configUSE_NEWLIB_REENTRANT can chew up considerable memory!
Check carefully if this is OK for your application!
FreeRTOS memory management
FreeRTOS internally uses its own memory management scheme and API (implemented by your choice of heapxx.c routines). Some FreeRTOS applications only use FreeRTOS-provided memory management. Unfortunately many libraries use the standard "C"
malloc-family routines internally,
including newlib itself
(for example STM's USB stack LL and HAL routines).
For these cases I've implemented the FreeRTOS memory management API
on top of the "C" standard (using newlib) in the
module heap_useNewlib.c below.
Bugs in projects generated by STM's CubeMX
Note as of spring 2020: ST has attempted to solve some of problems discussed below, but rather than just redistributing heap_useNewlib (as NXP and others have done), ST has tried to redo the implementation - and failed to do it correctly! Best you delete their unfortunate failed attempts and use proven working code provided here.
From earlier versions: There are a number of problems in projects generated with STM's CubeMX tool, which lead to memory corruption and other unfortunate happenings:
- STM does not provide __malloc_lock/unlock.
Hence memory management is not thread-safe.
In a stunning bit of coding malpractice, STM's USB stack calls
mallocfrom within an ISR (see for example function
USBD_CDC_Init). Normally thread-safety is provided by suspending task-switching during memory management. Because of this incompetent use of
mallocinside an ISR, interrupts need to be suspended during memory management. Unfortunately this can greatly increase interrupt latency.
ToDo: Verify priority set by
taskENTER_CRITICAL_FROM_ISRactually blocks USB interrupt.
newlib's free storage requires an external implementation of sbrk
to provide the heap storage then doled out by the malloc-family
newlib sbrk requirements).
sbrkimplementation provided by STM cannot work with FreeRTOS.
mallocrequest after the scheduler starts will fail.
STM did not enable (in
#define configUSE_NEWLIB_REENTRANT 1 // Required for thread-safety of newlib sprintf, strtok, etc...
Using newlib safely with FreeRTOS - Possible Approaches
If you are certain your application does not use any
newlib functions or any library that internally use
the malloc-family and/or depend on thread-specific reentrant context, you could do nothing.
But, are you really sure???
What about the libraries you use (for example STM's USB stack LL and HAL routines)?
For some hints, see this list of newlib functions with reentrancy support.
The only really safe way to preclude accidental use is to
provide HCF stubs for every one of these functions.
Alternatively you can provide a linker -wrap command for each forbidden
function but don't implement the wrappers, so link fails if forbidden routines are ever referenced.
You really need to be sure!
To avoid using newlib's printf and dragging in newlib reentrancy components,
there are many cut-down printf implementations available (that do not use malloc).
Won't help if you're using
strtok or others!
Example light-weight printf implementations:
- Mario Viara's light-weight printf (optional floating point and reentrancy), used in MCU on Eclipse Processor Expert.
- mpaland printf (optional floating point).
- printf-stdarg.c distributed in the FreeRTOS Lab TCPIP example; this one only uses stack storage but does not implement floating point.
- https://github.com/ksstech/support_common .
Another option is wrap newlib's malloc-family to use FreeRTOS free storage (ie heap_4.c), and specify newlib support for FreeRTOS. Tell the linker to wrap all newlib's malloc-family functions (using -Xlinker --wrap=malloc etc.), and provide a wrapper function that calls the FreeRTOS functions. I tried that, but newlib's printf family uses realloc, which is not supported in FreeRTOS heap implementations.
In the end (thanks to Richard Damon for encouraging this approach), I implemented the FreeRTOS memory API on top of newlib's malloc family, and provided all the hooks newlib's malloc family requires.
Using newlib safely with FreeRTOS - Recommended Solution Details
If your application needs a complete
malloc family implementation,
or you are using any newlib functions that require malloc
printf family or
do the following
(I've provided an implementation below):
- Implement the hooks required by newlib (
Make sure your linker file matches the
- Provide a heap implementation that implements the FreeRTOS memory API using the malloc family of newlib.
To use the implementation I've provided in your project:
Exclude from all builds any current FreeRTOS heap implementation, typically something like:
Exclude from all builds any current sbrk implementation.
In older versions of RubeMX, ST generated a stand-alone sbrk.c module.
Later versions hide it in
syscalls.cto obfuscate their messes. Get rid of it!
Add the module
If in multiple tasks your application needs a complete
dtoa>, or other newlib functions requiring reentrancy support, or you're not really sure...
Configure FreeRTOS for newlib support. In
FreeRTOSconfig.h, add the line:
#define configUSE_NEWLIB_REENTRANT 1
In RubeMX FreeRTOS options, select configUSE_NEWLIB_REENTRANT + deselect "FW pack heap file".
- If are sure you're only going to access newlib's reentrant routines from a single FreeRTOS task and want to skip configUSE_NEWLIB_REENTRANT (with its attendent overhead)? Ensure malloc is never called except during startup and from the designated single task. Verify with a check in malloc_lock (sorry, I did not provide example code).
Unfortunately some vendors distribute tool chains with incorrect examples of FreeRTOS/newlib.
However, with a bit of care newlib works well and safely with FreeRTOS.
Best Regards, Dave
PS: newlib provides facilities for wrapping stdio functions, not covered in this article. You will want to use these, for example, if your application uses posix IO functions to read and write to a USB stick using a local FAT implementation.
PS: Hope Richard Barry and the FreeRTOS team will add
heap_useNewlib.c to FreeRTOS ;-)