newlib requirements
GNU ARM Embedded Toolchain
distributions include a non-polluting reduced-size runtime library called newlib (or newlib-nano for
the smallest variant).
Unfortunately, newlib internally uses free storage (malloc/free)
in startling places within the C runtime library.
Thus, newlib free storage routines get dragged in and used unexpectedly.
Ooops: Not MISRA-compliant, if you believe in that stuff.
The most common functions that bite unsuspecting embedded developers are
sprintf
using %f
, dtoa
, ftoa
, rand
, and strtok
.
Here's a list of newlib functions with reentrancy support (and hence using malloc).
newlib requires platform-specific support to:
- provide thread-safety,
- switch thread contexts, and
- obtain memory to be doled out and managed
by malloc/free/etc (
sbrk
).
Here's the detailed list of support functions required by newlib.
If this support is properly implemented, newlib works well in a threaded environment like FreeRTOS.
This support is not properly implemented for FreeRTOS projects in
Freescale KDS and NXP MCUXpresso until spring 2020, and in some current MCUXpresso examples.
See: BUG: malloc
overruns heap, returns invalid pointer, and corrupts memory,
from a 1-line application printf'ing a float
Example newlib use of malloc/malloc_r:
Placing a breakpoint within malloc_r (innermost malloc routine) shows the following:
- newlib 3.0 does not use malloc before main() is called (unlike earlier NXP/newlib 2.5 startup, which malloc'd 5k). Verified for all four newlib variants.
- With simple decimal output, sprintf does not call malloc
- For %f output, sprintf does
4 malloc totalling ~200 bytes
(only the first time it is called per task/thread).
%f also requires proper linker arguments for float support. - printf or similar function expected to do IO (as opposed to string operation) allocates an IO control structure of 428 bytes.
Tip: How can I see who is calling malloc ?
The basic steps are:
- create a wrapper function malloc_r,
- add the wrapper options to the linker, and
- place a breakpoint in malloc_r.
A complete example with instructions is included in the code below.
newlib under the hood - concurrency
newlib maintains information it needs to support each separate context (thread/task/ISR)
in a reentrancy structure.
This includes things like a thread-specific errno,
thread-specific pointers to allocated buffers, etc.
The active reentrancy structure is pointed at by global pointer _impure_ptr,
which initially points to a statically allocated structure instance.
If you use no threading or tasks, and you don't use
malloc/free/etc or any of the reentrancy-dependent functions
in multiple execution contexts, nothing more is required to use newlib in your application
(newlib will maintain its required info in the single static structure).
If your application or any library you use requires
malloc/free/etc or any of the reentrancy-dependent functions
in multiple contexts, newlib requires:
- concurrency protection for malloc/free/etc. The free storage pool will be corrupted if multiple threads call into these functions concurrently! To prevent reentrant execution of malloc/free/etc routines newlib requires hook procedures __malloc_lock/unlock If an RTOS-based application does not replace newlib's complete internal malloc family (FreeRTOS does not), _malloc_lock/unlock must be provided for thread safety.
- multiple reentrancy structures (one per context), and a mechanism to create, initialize, and cleanup these structures, plus switching _impure_ptr to point at the correct reentrancy structure each time the context changes.
WTF? Why are newlib's dtoa, ftoa, and sprintf calling malloc???
To quote Steele and White, Isn't it a pain when you ask a computer
to divide 1.0 by 10.0 and it prints 0.0999999?
To summarize and over-simplify a very long story,
in 1990 my friend Will Clinger published a seminal paper giving
a solution to this problem, but Will's solution required arbitrary precision and hence storage.
See
Will's retrospective on his seminal 1990 paper and subsequent developments.
David Gay published an improved implementation (based on Will's paper), which was extremely widely adopted.
newlib uses David Gay's code for dtoa and ftoa (using malloc), and sprintf uses dtoa and ftoa.
Keith Packard's picolibc implements the no-malloc precise Ryū algorithm.
There has been some recent discussion on the newlib mailing list including
details of Gay's algorithm and maybe incorporating no-malloc Ryū algorithm from picolibc into newlib.
Really now, aren't you sorry you asked me why?
FreeRTOS support for newlib
FreeRTOS provides support for newlib's context management. InFreeRTOSconfig.h
, add: #define configUSE_NEWLIB_REENTRANT 1 // Required for thread-safety of newlib sprintf, strtok, etc...
With this option FreeRTOS does the following (in task.c):
- For each task, allocate and initialize a newlib reentrancy structure in the task control block (TCB).
This adds 96* bytes overhead per task (* size hugely dependant on newlib build options), plus anything else newlib allocates as needed. - Each task switch, set _impure_ptr to point to the newly active task's reentrancy structure.
- On task destruction, clean up the reentrancy structure (help newlib free any associated memory).
Careful: Depending on how your supplier built newlib, and whether you are using nano,
configUSE_NEWLIB_REENTRANT
can chew up considerable memory!
Check carefully if this is OK for your application!
FreeRTOS memory management
FreeRTOS internally uses its own memory management scheme and API
(implemented by your choice of heapxx.c routines).
Some FreeRTOS applications only use FreeRTOS-provided memory management.
Unfortunately many libraries use the standard "C" malloc
-family routines internally,
including newlib itself.
For these cases I've implemented the FreeRTOS memory management API
on top of the "C" standard (using newlib) in the
module heap_useNewlib.c below.
Using newlib safely with FreeRTOS - Possible Approaches
If you are certain your application does not use any
newlib functions or any library that internally use
the malloc-family and/or depend on thread-specific reentrant context, you could do nothing.
But, are you really sure???
What about the libraries you use?
For some hints, see this list of newlib functions with reentrancy support.
The only really safe way to preclude accidental use is to
provide HCF stubs for every one of these functions.
Alternatively you can provide a linker -wrap command for each forbidden
function but don't implement the wrappers, so link fails if forbidden routines are ever referenced.
You really need to be sure!
To avoid using newlib's printf and dragging in newlib reentrancy components,
there are many cut-down printf implementations available (that do not use malloc).
Won't help if you're using dtoa
or strtok
or others!
Example light-weight printf implementations:
- Mario Viara's light-weight printf (optional floating point and reentrancy), used in MCU on Eclipse Processor Expert.
- mpaland printf (optional floating point).
- printf-stdarg.c distributed in the FreeRTOS Lab TCPIP example; this one only uses stack storage but does not implement floating point.
- https://github.com/ksstech/support_common .
Another option is wrap newlib's malloc-family to use FreeRTOS free storage (ie heap_4.c), and specify newlib support for FreeRTOS. Tell the linker to wrap all newlib's malloc-family functions (using -Xlinker --wrap=malloc etc.), and provide a wrapper function that calls the FreeRTOS functions. I tried that, but newlib's printf family uses realloc, which is not supported in FreeRTOS heap implementations.
In the end (thanks to Richard Damon for encouraging this approach), I implemented the FreeRTOS memory API on top of newlib's malloc family, and provided all the hooks newlib's malloc family requires.
Using newlib safely with FreeRTOS - Recommended Solution Details
If your application needs a complete malloc
family implementation,
or you are using any newlib functions that require malloc
(for example printf
family or strtok
),
do the following
(I've provided an implementation below):
- Implement the hooks required by newlib (
sbrk, __malloc_lock/unlock
).
Make sure your linker file matches thesbrk
implementation! - Provide a heap implementation that implements the FreeRTOS memory API using the malloc family of newlib.
To use the implementation I've provided in your project:
-
Exclude from all builds any current FreeRTOS heap implementation, typically something like:
Middlewares\Third_Party\FreeRTOS\Source\portable\MemMang\heap_4.c
- Exclude from all builds any current sbrk implementation.
-
Either add the module
heap_useNewlib_NXP.c
I've provided, or the version now distributed by NXP. - Make sure your LD files provide the symbols required (defining the heap boundaries and size). The example LD file below show how, and also includes some helpful space-used computations, accessible with a post-build-step grep DRN on the map file.
-
If in multiple tasks your application needs a complete
snprintf
implementation,strtok
,dtoa>
, or other newlib functions requiring reentrancy support, or you're not really sure...
Configure FreeRTOS for newlib support. InFreeRTOSconfig.h
, add the line:
#define configUSE_NEWLIB_REENTRANT 1 - If are sure you're only going to access newlib's reentrant routines from a single FreeRTOS task and want to skip configUSE_NEWLIB_REENTRANT (with its attendent overhead)? Ensure malloc is never called except during startup and from the designated single task. Verify with a check in malloc_lock (sorry, I did not provide example code).
Note: Latest code and further documentation is on Github
Summary
Unfortunately some vendors distribute toolchains with incorrect examples of FreeRTOS/newlib.
However, with a bit of care newlib works well and safely with FreeRTOS.
Enjoy!
Best Regards, Dave
PS: newlib provides facilities for wrapping stdio functions, not covered in this article. You will want to use these, for example, if your application uses posix IO functions to read and write to a USB stick using a local FAT implementation.
PS: Hope Richard Barry and the FreeRTOS team will add heap_useNewlib.c
to FreeRTOS ;-)
Additional References
http://www.billgatliff.com/newlib.html
http://wiki.osdev.org/Porting_Newlib
http://www.embecosm.com/appnotes/ean9/ean9-howto-newlib-1.0.html