newlib is the C runtime library supplied with many GNU bare-metal embedded toolchains, including the
ARM GNU Embedded Toolchain (arm-none-eabi),
which is in turn redistributed by MCU vendors such as NXP as part of their development toolkits. Important notes:
Internally newlib has facilities making it suitable for embedded applications using FreeRTOS or another OS... but ONLY when newlib is correctly built and integrated with the OS.
This article attempts to explain:
_Thread_local, __thread, and C++ thread_local) fit into this?Multiple threads (tasks in FreeRTOS) often need their own thread-specific private copy of some data, in addition to data on their private stack.
In a C library, thread-specific instances are needed for all kinds of things (so that different threads don't stomp on each other), for example "globals" such as:
errno, which must have a value specific to the calling thread (even though application code accesses errno as if it were a global variable),asctime that return pointers to these static strings (though sometimes libraries use TLS for these buffers),strtok internal state (pointer to the next parse point),If the RTOS provides preemption, a task can be suspended in the middle of a library call and another task can enter the same or another library function. With SMP, library functions can additionally execute simultaneously on different processors. The library must therefore ensure that task-specific state is not inadvertently shared, and that genuinely shared mutable state is properly protected. If one thread is suspended in the middle of a library function, a second thread must not be able to stomp on that function's thread-specific internal data by calling a library function from the second thread.
Proper integration of the library and RTOS is required to support the library's thread-specific state. This is distinct from protecting shared mutable resources: per-thread state requires a reentrancy mechanism, while shared resources (especially the heap and other global facilities) require proper locking so accesses from multiple threads do not collide. Locking is a separate part of the thread-safety story.
struct _reent implementationTraditionally, newlib places thread-specific library state in a struct _reent.
For a non-threaded application only one struct _reent instance is needed. For a multi-threaded application, the OS must provide a struct _reent for each thread and arrange for newlib to obtain the instance belonging to the currently executing thread.
Within the classic newlib model, the macro _REENT is used to obtain the current struct _reent.
_impure_ptrThe traditional implementation has a global pointer _impure_ptr. newlib code ultimately obtains the current reentrancy context through _REENT; without dynamic reentrancy, _REENT resolves directly to _impure_ptr.
On a single processor, an RTOS can therefore change _impure_ptr on every context switch so it points at the struct _reent belonging to the task being switched in.
To enable FreeRTOS built-in newlib context support, #define configUSE_NEWLIB_REENTRANT 1 in your FreeRTOSConfig.h.
struct _reent directly in the TCBIn FreeRTOS versions through V10.4.6 (still widely used), enabling configUSE_NEWLIB_REENTRANT places a struct _reent named xNewLib_reent directly in every task's TCB.
FreeRTOS:
xNewLib_reent when a task is created,_impure_ptr to point to the newly selected task's xNewLib_reent each context switch,This implementation does not depend on GCC compiler TLS, nor on FreeRTOS's later generic C-runtime TLS abstraction. It is simply a struct _reent stored directly in each TCB, selected by changing _impure_ptr.
struct _reent in FreeRTOS-specific TLSStarting with FreeRTOS V10.5.0, FreeRTOS provides a TLS mechanism, enabled with configUSE_C_RUNTIME_TLS_SUPPORT, which stores one runtime-specific block in each task's TCB. In this article this mechanism is referred to as FreeRTOS-specific-TLS.
Despite the name TLS, FreeRTOS-specific-TLS is a FreeRTOS facility and must not be confused with GCC compiler TLS (_Thread_local, __thread, or C++ thread_local).
For newlib, configUSE_NEWLIB_REENTRANT is now implemented using FreeRTOS-specific-TLS.
When configUSE_NEWLIB_REENTRANT is enabled, FreeRTOS automatically enables FreeRTOS-specific-TLS and, by default, configures its per-task block as a struct _reent.
FreeRTOS-specific-TLS uses these configuration macros:
configTLS_BLOCK_TYPE
configINIT_TLS_BLOCK(...)
configSET_TLS_BLOCK(...)
configDEINIT_TLS_BLOCK(...)
FreeRTOS stores one xTLSBlock of configTLS_BLOCK_TYPE in each TCB and invokes these hooks as tasks are initialized, switched in, and destroyed.
Current newlib-freertos.h supplies these defaults
(unless the application has already defined the corresponding configuration macros, in which case you've got a different problem):
#define configUSE_C_RUNTIME_TLS_SUPPORT 1
#define configTLS_BLOCK_TYPE struct _reent
#define configINIT_TLS_BLOCK(...) _REENT_INIT_PTR(...)
#define configSET_TLS_BLOCK(...) (_impure_ptr = ...)
#define configDEINIT_TLS_BLOCK(...) _reclaim_reent(...)
Though _reent is stored in a different place, just as in versions prior to V10.5.0, FreeRTOS:
_reent when a task is created,_impure_ptr to point to the newly selected task's _reent each context switch,Both approaches provide one newlib struct _reent per task and select it through _impure_ptr.
There is only one FreeRTOS-specific-TLS xTLSBlock in each task's TCB, with one configTLS_BLOCK_TYPE and one set of init/set/deinit hooks. Therefore, when configUSE_NEWLIB_REENTRANT uses that block for struct _reent, this FreeRTOS-specific-TLS facility is unavailable for other use.
Using xTLSBlock to hold struct _reent does not consume FreeRTOS's separate application thread-local-storage pointer facility, controlled by #define configNUM_THREAD_LOCAL_STORAGE_POINTERS n.
Those pointers remain available for arbitrary per-task application storage through the FreeRTOS APIs vTaskSetThreadLocalStoragePointer() and pvTaskGetThreadLocalStoragePointer().
The TCB is laid out thus:
FreeRTOS task TCB
+--------------------------------------+
| xTLSBlock |
| struct _reent | <- one FreeRTOS-specific-TLS block
+--------------------------------------+
| pvThreadLocalStoragePointers[...] | <- separate application pointers
+--------------------------------------+
It would be possible to make the single FreeRTOS-specific-TLS block a custom composite type containing state required by more than one runtime, with custom init/set/deinit hooks. However, that is then application/port-specific integration rather than two independently configured FreeRTOS-specific-TLS blocks.
_impure_ptr is not sufficient for SMPOn an SMP, tasks can execute simultaneously on different processors, and each task requires its specific context. A single global _impure_ptr can't be correct for more than one task...
__DYNAMIC_REENT__ and __getreent()newlib also supports dynamic reentrancy.
When newlib is built with __DYNAMIC_REENT__, the _REENT macro is defined to obtain the current reentrancy object through the function struct _reent *__getreent(void);
(instead of directly using _impure_ptr).
For an OS to use the dynamic reentrancy option:
__getreent()
so __getreent() returns the current task's struct _reent.
This supports SMP because __getreent() can determine which task is running on the calling processor rather than depending on one process-wide global pointer.__getreent() implementation is adequate
provided FreeRTOS is correctly switching _impure_ptr for each task;
the default simply returns _impure_ptr,
so building newlib with __DYNAMIC_REENT__ does not by itself make the library SMP-safe.
The SMP OS integration must provide a suitable per-current-task implementation of __getreent() and context switching etc...--enable-newlib-reent-thread-local reentrancySeparately from the classic struct _reent mechanism, newlib 4.3.0
added the configuration option --enable-newlib-reent-thread-local.
This is incompatible with the classic __DYNAMIC_REENT__ model:
with thread-local reentrancy, struct _reent, _impure_ptr, and __getreent() are not the mechanism used to obtain per-thread state.
This configuration option defines _WANT_REENT_THREAD_LOCAL configuration and
replaces the traditional struct _reent members with dedicated C _Thread_local objects.
In this configuration, struct _reent no longer exists.
Conceptually:
Task A compiler TLS Task B compiler TLS
+-------------------+ +-------------------+
| errno | | errno |
| strtok state | | strtok state |
| asctime state | | asctime state |
| ... | | ... |
+-------------------+ +-------------------+
This is a fundamentally different mechanism from both:
xNewLib_reent member, andxTLSBlock when that block contains a struct _reent.The newlib _Thread_local configuration requires genuine compiler/runtime TLS support from the toolchain and appropriate RTOS integration.
It has two important consequences:
_impure_ptr to select the task's thread-specific state.This is therefore naturally compatible with SMP provided the compiler TLS implementation itself is correctly integrated with the SMP RTOS.
Ooops!
FreeRTOS, as explained below, does not directly implement GCC language-level TLS,
so its built-in newlib integration does not support newlib built with the configuration option --enable-newlib-reent-thread-local.
GCC/G++ supports the language-level TLS declarations (these are compiler/language TLS, nothing to do with FreeRTOS-specific-TLS):
_Thread_local int c_variable; // C11
__thread int gcc_variable; // GCC specific
thread_local int cpp_variable; // standard C++ since C++11
For native ELF TLS, the compiler and linker create one TLS layout/template for the application.
TLS variables are placed in TLS sections such as .tdata (initialized TLS data)
and .tbss (zero-initialized TLS data), and the ELF file may contain a PT_TLS program header
describing the complete TLS template, its total per-thread size, and required alignment.
The linker does not create a separate .tdata or .tbss for every thread.
At run time, the OS/runtime must allocate a separate TLS block for each thread/task, using that one linked TLS layout as the template. The initialized portion is copied from the application's TLS template, the zero-initialized portion is cleared, and each task gets its own thread-pointer value referring to its own TLS block.
The linker assigns every TLS object a fixed location/offset in the application's TLS layout.
The exact generated address sequence depends on the TLS model, but ultimately selects that object's instance in the current thread's TLS block.
Each task needs its own runtime TLS block large enough for all TLS objects in the application.
When a task's TLS block is created, the initialized bytes from the single .tdata template are copied into it,
and the .tbss portion is zero-initialized.
C++ thread_local objects requiring dynamic initialization are different:
their constructors are not found by scanning .tdata.
G++ emits TLS initialization/wrapper code and per-thread guard state for such objects.
On first use requiring initialization, the generated wrapper runs the constructor for the current task's instance.
Thus the same generated initialization code is used by every task,
but it operates on the object selected by that task's current thread pointer.
For a non-trivial thread_local object, G++ normally arranges for its destructor to be registered
when that task's instance is constructed, using the C++ runtime's thread-exit mechanism,
commonly __cxa_thread_atexit().
A FreeRTOS integration must provide the required C++ runtime support and invoke the registered destructors
when the corresponding task terminates; FreeRTOS does not do this automatically.
Every access to a TLS object is generated relative to a thread pointer (TP) supplied by the execution environment. The linker assigns each TLS symbol a fixed offset within the per-thread TLS layout. Thus a TLS access is conceptually:
address of this thread's variable = current thread pointer + link-time TLS offset
Every thread has a different TLS block/thread-pointer value,
but uses the same link-time offset for a given TLS variable.
Thus, if asctime() uses a _Thread_local object, the same asctime() code can run in many tasks:
each execution obtains the current task's TP and consequently accesses that task's own instance.
On 32-bit Arm GCC, the -mtp=soft model obtains the current thread pointer
by calling __aeabi_read_tp().
FreeRTOS (or an integration layer) must make that function return the TP
corresponding to the task currently executing.
FreeRTOS does not automatically supply the complete GCC/Arm TLS ABI simply because configNUM_THREAD_LOCAL_STORAGE_POINTERS is non-zero.
The compiler's TLS block must be created for each task and the compiler's thread-pointer mechanism must return the TLS context belonging to the currently executing task.
The statically initialized TLS template must be copied/zeroed when the task's TLS block is created.
C++ thread_local objects requiring dynamic initialization are normally constructed by G++'s generated per-thread initialization wrappers when first required by the C++ initialization rules, not simply at task creation.
Their destructors must be registered for that task/thread and invoked when that task/thread terminates.
In My Humble Opinion, there are a number of truly unfortunate implementation decisions (and poor documentation) that have contributed to enormous confusion and countless buggy applications using newlib (with FreeRTOS anyway). As a developer who has had to debug why my application crashed because of this stuff, more than once, I'm all too familiar with the consequences of these screw-ups. Since I published newlib and FreeRTOS these tools have been used in probably thousands of applications - only because the distributed toolchains improperly integrate newlib. Blame for this falls on newlib and FreeRTOS developers, not just clueless marginally-qualified 'developers' at the MCU vendors. Making me a cranky curmudgeon...
By including no-op stubs for required integration pieces inside the provided newlib library, users have often missed important required bits. newlib should provide the same no-op stubs,
as well-documented source code users must build into their applications, making it much more obvious how to safely use newlib. For example, required locking hooks include __malloc_lock() / __malloc_unlock() for the malloc pool, __env_lock() / __env_unlock() for the environment, and __tz_lock() / __tz_unlock() for shared time-zone state.
Depending on how newlib was configured, the more general __retarget_lock_*() interface may instead be used for library locks. Dynamic reentrancy is a separate issue: when __DYNAMIC_REENT__ is enabled, the OS must provide an appropriate __getreent() if newlib's default implementation (which simply returns _impure_ptr) is not suitable.
A debug version of newlib should provide context checking when TLS is accessed. We have seen numerous cases (especially in code from MCU vendors) where library routines needing task-context are used within ISRs. newlib should have a debug build where such library function use, or (access to memory management) immediately causes an assertion and stops the application, in hopes that cluelessness or accidents are caught ASAP.
WTF version of newlib am I using?
Not just the version number exposed by <newlib.h> (via <_newlib_version.h>),
but the complete set of options newlib was built with.
<newlib.h> exposes many build-time configuration definitions,
but does not tell me everything needed to determine the reentrancy integration,
such as whether __DYNAMIC_REENT__ is defined for the target.
For an SMP application I may have to inspect the library to determine whether the default __getreent() implementation (which simply returns _impure_ptr) was provided!
THIS SUCKS!
Documentation for TLS replacing struct _reent is difficult to find.
The online newlib reentrancy documentation describes the classic struct _reent / _impure_ptr model,
but does not explain that a newlib build configured with --enable-newlib-reent-thread-local instead replaces struct _reent with compiler TLS objects.
It does not explain that the reentrancy functions are obsolete here.
When library integration is provided, it should be complete.
FreeRTOS provides limited support for newlib reentrancy as explained above,
but fails to provide complete library integration.
Most spectacularly, configUSE_NEWLIB_REENTRANT does not make newlib's shared malloc pool thread-safe.
The application must still provide working __malloc_lock(struct _reent *) and __malloc_unlock(struct _reent *) implementations (or the corresponding retargetable-lock implementation when that newlib configuration is used).
The malloc lock must support recursive acquisition because newlib may acquire it recursively.
FreeRTOS Thread Local Storage Pointers documentation should make absolutely clear that this FreeRTOS facility has nothing to do with C/C++ language TLS support, and that FreeRTOS does not itself implement GCC language-level TLS.
arm-none-eabi) TLS supportIt is important to distinguish GCC's language/code-generation capability from whether a particular pre-built bare-metal toolchain was configured for native ELF TLS.
| Arm GNU Toolchain release | GCC version | GCC understands _Thread_local / __thread / thread_local |
Native ELF TLS in Arm arm-none-eabi distribution |
|---|---|---|---|
| 10.3-2021.10 | GCC 10.3.1 | Yes | No; toolchain built with TLS disabled |
| 11.3.Rel1 | GCC 11.3 | Yes | No; toolchain built with TLS disabled |
| 12.2.Rel1 | GCC 12.2 | Yes | No; toolchain built with TLS disabled |
| 12.3.Rel1 | GCC 12.3 | Yes | No; toolchain built with TLS disabled |
| 13.2.Rel1 | GCC 13.2 | Yes | No; toolchain built with TLS disabled |
| 13.3.Rel1 | GCC 13.3 | Yes | No; toolchain built with TLS disabled |
| 14.2.Rel1 | GCC 14.2 | Yes | No; toolchain built with TLS disabled |
| 14.3.Rel1 | GCC 14.3 | Yes | No; toolchain built with TLS disabled |
| 15.2.Rel1 | GCC 15.2 | Yes | Yes; native ELF TLS enabled |
For vendor-supplied GCC toolchains, including a compiler shipped inside an MCU IDE, the GCC version number alone is not sufficient. The vendor can build the same GCC version with different configuration options. Check the actual compiler with:
arm-none-eabi-gcc -v
and inspect its configure options.
Older Arm arm-none-eabi distributions built with --disable-tls may still cause GCC to emit emulated TLS sequences (for example involving __emutls_get_address) for TLS declarations. That is not the same as native ELF TLS and still requires a suitable runtime implementation. It should not be mistaken for a ready-to-use FreeRTOS TLS implementation.
FreeRTOS also has a separate facility whose name contains "thread local storage":
#define configNUM_THREAD_LOCAL_STORAGE_POINTERS n
This adds an array of arbitrary application pointers to each TCB, accessed with FreeRTOS APIs such as vTaskSetThreadLocalStoragePointer() and pvTaskGetThreadLocalStoragePointer().
These pointers are simply per-task storage provided by FreeRTOS. They are not:
struct _reent mechanism,xTLSBlock),In particular, they do not by themselves implement _Thread_local, __thread, or C++ thread_local.
Yes. They represent different state and can coexist peacefully:
FreeRTOS task
+------------------------------------+
| classic newlib struct _reent |
| errno, library state, ... |
+------------------------------------+
| GCC compiler TLS block |
| application _Thread_local data |
| C++ thread_local objects |
| (and newlib TLS objects if |
| newlib was built that way) |
+------------------------------------+
In current FreeRTOS, the built-in configUSE_NEWLIB_REENTRANT adapter uses the single FreeRTOS-specific-TLS xTLSBlock to hold the classic newlib struct _reent. GCC compiler TLS is a separate mechanism.
Consequently, GCC compiler TLS cannot simply be added by independently configuring a second FreeRTOS-specific-TLS block: there is no second block. A GCC TLS implementation must either use other per-task state (potentially including the separate FreeRTOS application TLS pointers), or replace the normal newlib adapter with a custom/composite FreeRTOS-specific TLS implementation that services both requirements.
This can make adding GCC thread support substantially more difficult than merely enabling a compiler option. The integration must create and initialize each task's GCC TLS image (including the required .tdata initial values and .tbss zero-initialization), retain the per-task TLS base, and make GCC's thread-pointer mechanism return the TLS base belonging to the task currently executing. On 32-bit Arm with GCC's soft thread-pointer model this includes providing the behavior expected through __aeabi_read_tp(). Task creation/deletion and, for SMP, simultaneous execution on different processors must all be handled correctly.
For classic newlib on a single-core FreeRTOS system, the existing _impure_ptr integration is straightforward. For SMP, either:
__getreent(), together with the necessary locking for genuinely shared newlib resources, or_Thread_local configuration with a correctly implemented compiler TLS environment.The second approach requires a toolchain whose TLS implementation is actually usable on the target and an RTOS/port integration that establishes the correct TLS block/thread pointer for every running task on every core.
No! FreeRTOS provides mechanisms that could be used as part of a
GCC TLS implementation, but the FreeRTOS kernel does not itself implement the
GCC/Arm ELF TLS ABI for _Thread_local, __thread, or C++ thread_local objects.
In particular, FreeRTOS does not automatically:
.tdata) into that image,.tbss),__aeabi_read_tp() for the currently executing task,thread_local dynamic initialization in each task,thread_local destructors to run when a task terminates.The required TLS image and layout are determined by the linked application and target TLS ABI, not by FreeRTOS. Build the final ELF file with GCC TLS enabled and inspect it with the GNU binutils supplied with the toolchain.
For example:
arm-none-eabi-readelf -SW application.elf
arm-none-eabi-readelf -lW application.elf
arm-none-eabi-readelf -sW application.elf
Look for:
.tdata -- initialized per-thread data,.tbss -- zero-initialized per-thread data,PT_TLS program header describing the complete TLS template.For a PT_TLS segment:
p_filesz is the size of the file-backed TLS initialization image to copy from the ELF/load image,p_memsz is the total TLS memory image size required for each thread,p_filesz through p_memsz - 1 must be zero-initialized,p_align specifies the required alignment.The exact linker symbols naming the start/end of the TLS template are not standardized for a bare-metal linker script. The application linker script should therefore export explicit symbols for the TLS template and size, for example:
__tls_start = ADDR(.tdata);
__tls_tdata_size = SIZEOF(.tdata);
__tls_end = ADDR(.tbss) + SIZEOF(.tbss);
__tls_size = __tls_end - __tls_start;
The actual linker script must also preserve the alignment required by the TLS ABI. Do not assume that .tdata immediately followed by .tbss in the ELF file is sufficient to determine the runtime thread-pointer value; GCC addresses TLS variables relative to the ABI-defined thread pointer, not simply relative to the beginning of an arbitrary allocated buffer.
A FreeRTOS/GCC integration must perform the equivalent of the following when each task is created:
.tbss (including any layout padding as required).On 32-bit Arm when GCC uses the software thread-pointer model (-mtp=soft), generated TLS accesses call:
void *__aeabi_read_tp(void);
The Arm ABI defines this function as returning the current thread pointer. A FreeRTOS implementation must therefore make __aeabi_read_tp() return the ABI-correct thread pointer for the current task.
A simplistic implementation that merely returns the start address of .tdata is not necessarily correct: the relationship between the thread pointer and the application's TLS block is defined by the Arm ELF TLS ABI and by the relocations emitted by GCC/linker. The implementation must use the same layout convention assumed by the toolchain.
For C objects and C++ objects that require only static initialization, constructing a new task's TLS state consists of copying the initialized TLS image and zeroing the remainder.
For example:
_Thread_local int a = 123; /* copied from .tdata */
_Thread_local int b; /* zeroed in .tbss */
After the TLS image has been created correctly, GCC-generated accesses to a and b work once the current task's thread pointer is established.
thread_local objects requiring constructorsC++ thread_local objects can require dynamic initialization:
thread_local MyClass object(arg1, arg2);
The bytes in the TLS template are not sufficient to construct such an object. GCC emits additional initialization code and a per-thread guard so that the constructor is run for each thread before the object's first required use.
Therefore, in general, a FreeRTOS port should not attempt to discover C++ constructors by scanning .tdata and calling them manually. Instead it must provide a correct GCC TLS environment so that GCC's generated TLS wrapper/initialization code can operate normally.
To determine what a particular application actually requires, inspect the final ELF and undefined symbols, for example:
arm-none-eabi-nm -C application.elf
arm-none-eabi-nm -u application.elf
arm-none-eabi-objdump -dr application.elf
Look for TLS wrapper/initialization functions generated by G++, and for runtime dependencies such as __cxa_thread_atexit.
Once the task has a valid TLS block and thread pointer, GCC normally performs dynamic initialization according to the C++ thread_local rules when the variable is first used. The per-thread guard which records whether initialization has occurred is itself thread-local state, so it also depends on the TLS block having been initialized correctly.
thread_local destructorsA non-trivial C++ thread_local object also requires its destructor to run when that thread terminates.
GCC normally registers such destructors through the C++ runtime, commonly via:
__cxa_thread_atexit(...)
FreeRTOS task deletion does not inherently provide C++ thread-exit destructor semantics. Therefore a complete integration must provide whatever C++ runtime support the selected GCC/libstdc++ build expects and must arrange for registered destructors to execute when the corresponding FreeRTOS task exits.
This is particularly awkward when one task deletes another task: destructors for the deleted task must run using that task's TLS context, not the deleting task's TLS context.
Supporting GCC language TLS on FreeRTOS is therefore not just a context-switch modification. A complete port must coordinate:
thread_local initialization,FreeRTOS provides useful storage and task-lifecycle hooks, but it does not provide this complete GCC TLS runtime integration itself.
tasks.c (classic xNewLib_reent implementation)FreeRTOS.h (generic C-runtime TLS abstraction)FreeRTOS.h_Thread_local reentrancy patch discussion-mtp=)