In January I wrote Compressed debug sections. The venerable zlib shows its age and there are replacements which are better in every metric except adoption and a larger memory footprint. The obvious choice was Zstandard, but I was not so confident about adoptinig it and solvin the ecosystem issue. At any rate, I slowly removed some legacy .zdebug support from llvm-project so that a new format could be more easily introduced.
lld 15 ELF changes
llvm-project 15 was just released. I added some lld/ELF notes to https://github.com/llvm/llvm-project/blob/release/15.x/lld/docs/ReleaseNotes.rst. Here I will elaborate on some changes.
-march=, -mcpu=, and -mtune=
In GCC and Clang, there are three major options specifying the architecture and microarchitecture the generated code can run on. The general semantics are described below, but each target machine may assign different semantics.
-march=X: (execution domain) Generate code that can use instructions available in the architecture X-mtune=X: (optimization domain) Optimize for the microarchitecture X, but does not change the ABI or make assumptions about available instructions-mcpu=X: Specify both-march=and-mtune=but can be overridden by the two options. The supported values are generally the same as-mtune=. The architecture name is inferred fromX
glibc and DT_GNU_HASH
tl;dr "Easy Anti-Cheat"'s incompatibility with glibc 2.36 provides shared objects (libc.so.6, ld-linux-x86_64.so.2) is an instance of Hyrum's law.
- On 2022-08-02 glibc 2.36 was released.
- On the same day the x86-64 package was moved to
[core]on Arch Linux. - On 2022-08-03 Jelgnum reported that with the new glibc, "Easy Anti-Cheat" cannot load the anti-cheat module (GLIBC update broke EAC for most games that use it).
- Multiple Arch Linux game users confirmed the problem.
- Frogging101 bisected the problem to the glibc commit Do not use --hash-style=both for building glibc shared objects.
- The problem led to heated discussions, some clickbait news, and claims such as "glibc breaks ABI" and "glibc does not prioritize compatibility with pre-existing applications".
I feel compelled to demystify the accident and wish that people can stop defamation to glibc.
DWARF in reproducible builds
UNDER CONSTRUCTION
Deterministic builds with clang and lld describes several degrees of build determinism. Here is the description of local determinism:
Like incremental basic determinism, but builds are also independent of the name of the build directory. Builds of the same source code on the same machine produce exactly the same output every time, independent of the location of the source checkout directory or the build directory.
RISC-V linker relaxation in lld
On 2022-07-07, I added a RISC-V linker relaxation framework in ld.lld and implemented R_RISCV_ALIGN/R_RISCV_CALL/R_RISCV_CALL_PLT relaxation. The changes will be included in the next llvm-project release 15.0.0. This post describes the implementation.
See The dark side of RISC-V linker relaxation for more information about RISC-V linker relaxation.
Everything I know about glibc
UNDER CONSTRUCTION
- Repository: https://sourceware.org/git/gitweb.cgi?p=glibc.git
- Wiki: https://sourceware.org/glibc/wiki/
- Bugzilla: https://sourceware.org/bugzilla/
- Mailing lists:
{libc-announce,libc-alpha,libc-locale,libc-stable,libc-help}@sourceware.org - Patchwork: https://patchwork.sourceware.org/project/glibc/list/ Patch Review Workflow
- Contribution Checklist: https://sourceware.org/glibc/wiki/Contribution%20checklist
- Build bots: https://builder.sourceware.org/buildbot/#/builders, search "glibc"
- public inbox: https://inbox.sourceware.org/libc-alpha/
glibc is an implementation of the user-space side of standard C/POSIX functions with Linux extensions.
C standard library headers in C++
In recent ISO C++ standards, [depr.c.headers] describes how a C header name.h is transformed to the corresponding C++ cname header. There is a helpful example:
[ Example: The header
assuredly provides its declarations and definitions within the namespace std. It may also provide these names within the global namespace. The header <stdlib.h> assuredly provides the same declarations and definitions within the global namespace, much as in the C Standard. It may also provide these names within the namespace std. — end example ]
"may also" in the wording allows implementations to provide mix-and-match, e.g. std::exit can be used with #include <stdlib.h> and ::exit can be used with #include <cstdlib>.
libstdc++ chooses to enable global namespace declarations with C++ cname header. For example, #include <cstdlib> also includes the corresponding C header stdlib.h and we get declarations in both the global namespace and the namespace std. 1
2. /usr/include/c++/12/cstdlib
.. /usr/include/stdlib.h
The preprocessed output looks like: 1
2
3
4
5
6
7
8
9extern void exit (int __status) noexcept (true) __attribute__ ((__noreturn__));
extern "C++"
{
namespace std __attribute__ ((__visibility__ ("default")))
{
using ::exit;
}
}
The compiler knows that the declarations in std are identical to the ones in the global namespace. The compiler recognizes some library functions and can optimize them. By using the compiler can optimize some C library functions in namespace std (e.g. many std::mem* and std::str* functions).
For some C standard library headers, libstdc++ provides wrappers (libstdc++-v3/include/c_compatibility/) which take precedence over the glibc headers. The configuration of libstdc++ uses --enable-cheaders=c_global by default. if GLIBCXX_C_HEADERS_C_GLOBAL in libstdc++-v3/include/Makefile.am describes that the 6 wrappers (complex.h, fenv.h, tgmath.h, math.h, stdatomic.h, stdlib.h) shadow the C library headers of the same name. For example, #include <stdlib.h> includes the wrapper stdlib.h which includes cstdlib, therefore bringing exit into the namespace std. 1
2
3. /usr/include/c++/12/stdlib.h
.. /usr/include/c++/12/cstdlib
... /usr/include/stdlib.h
Unwinding through a signal handler
This post has some notes about unwinding through a signal handler. You may want to read Stack unwinding first.
1 | // a.c |
(printf and dladdr are not required to be async-signal-safe functions, but here we apparently know using them can't cause problems.)
Tips: we can additionally add the following code block to get memory mappings. 1
2
3
4
5char buf[128];
FILE *f = fopen("/proc/self/maps", "r");
while (fgets(buf, sizeof buf, f))
printf("%s", buf);
fclose(f);
Build the program with either llvm-project libunwind or nongnu libunwind: (Some targets default to 1
2
3
4
5
6# ninja -C /tmp/Debug unwind builtins
clang -g -I llvm-project/libunwind/include a.c -no-pie --unwindlib=libunwind --rtlib=compiler-rt -ldl -Wl,-E,-rpath,/tmp/Debug/lib/x86_64-unknown-linux-gnu -o llvm
# autoreconf -i; mkdir -p out/debug; ../../configure CFLAGS='-O0 -g' CXXFLAGS='-O0 -g'; make -j 20
libunwind=/tmp/p/libunwind
clang -g -I $libunwind/include -I $libunwind/out/debug/include a.c -no-pie $libunwind/out/debug/src/.libs/libunwind.a $libunwind/out/debug/src/.libs/libunwind-x86_64.a -llzma -ldl -Wl,-E -o nongnu-fno-asynchronous-unwind-tables. In the absence of C++ exceptions, we need at least -funwind-tables.)
glibc x86-64
With either implementation, the output looks like the following on Linux glibc x86-64. I annotated the lines with location information. 1
2
3
4
5
6pc=0x0000000000206d2a sp=0x00007fffd366bce0 ./nongnu: # in handler, the instruction after call unw_getcontext
pc=0x00007f5962cb0920 sp=0x00007fffd366c500 /lib/x86_64-linux-gnu/libc.so.6: # __restore_rt
pc=0x00007f5962cb08a1 sp=0x00007fffd366d200 /lib/x86_64-linux-gnu/libc.so.6:gsignal # raise
pc=0x0000000000206cfd sp=0x00007fffd366d320 ./nongnu:main
pc=0x00007f5962c9b7fd sp=0x00007fffd366d340 /lib/x86_64-linux-gnu/libc.so.6:__libc_start_main
pc=0x0000000000206bba sp=0x00007fffd366d410 ./nongnu:_start # from crt1.o
__restore_rt is a signal trampoline defined in glibc sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c: (Newer ports use VDSO.)1
2
3
4
5 nop
.align 16
__restore_rt:
movq $15, %rax # __NR_rt_sigreturn
syscall
glibc's sigaction sets the sa_restorer field of sigaction to __restore_rt, and sets the SA_RESTORER. The kernel sets up the __restore_rt frame with saved process context information (ucontext_t structure) before jumping to the signal handler. See kernel arch/x86/kernel/signal.c:setup_rt_frame. Upon returning from the signal handler, control passes to __restore_rt. See man 2 sigreturn.
__restore_rt is implemented in assembly. It comes with DWARF call frame information in .eh_frame.
1 | % llvm-dwarfdump -eh-frame /lib/x86_64-linux-gnu/libc.so.6 |
The DW_OP_breg7 RSP offsets correspond to the ucontext_t offsets of these registers. 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15% cat sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c
...
do_cfa_expr \
do_expr (8 /* r8 */, oR8) \
do_expr (9 /* r9 */, oR9) \
do_expr (10 /* r10 */, oR10) \
% cat sysdeps/unix/sysv/linux/x86_64/ucontext_i.sym
...
#define ucontext(member) offsetof (ucontext_t, member)
#define mcontext(member) ucontext (uc_mcontext.member)
#define mreg(reg) mcontext (gregs[REG_##reg])
oRBP mreg (RBP)
oRSP mreg (RSP)
oRBX mreg (RBX)
With the information, libunwind can unwind through the trampoline without knowing the ucontext_t structure. Note that all general purpose registers are encoded. libunwind/docs/unw_get_reg.man says
However, for signal frames (see unw_is_signal_frame(3)), it is usually possible to access all registers.
Volatile registers are also saved in the saved process context information. This is different from other frames where volatile registers' information is typically lost.
glibc AArch64
The output looks like: 1
2
3
4
5
6
7pc=0x0000000000214b10 sp=0x0000ffffe81f6050 ./nongnu: # handler
pc=0x0000ffffa55cd5b0 sp=0x0000ffffe81f6c70 linux-vdso.so.1:__kernel_rt_sigreturn
pc=0x0000ffffa5438070 sp=0x0000ffffe81f7ed0 /lib/aarch64-linux-gnu/libc.so.6:gsignal
pc=0x0000000000214bfc sp=0x0000ffffe81f8000 ./nongnu:main
pc=0x0000ffffa5425090 sp=0x0000ffffe81f8010 /lib/aarch64-linux-gnu/libc.so.6:__libc_start_main
pc=0x00000000002149cc sp=0x0000ffffe81f8160 ./nongnu: # _start
pc=0x00000000002149cc sp=0x0000ffffe81f8160 ./nongnu: # _start
As a relatively new port, Linux AArch64 defines the signal trampoline __kernel_rt_sigreturn in the VDSO (see arch/arm64/kernel/vdso/sigreturn.S). This is unlike x86-64 which defines the function in libc. We can use gdb to dump the VDSO.
1 | (gdb) i proc m |
1 | nop |
As of Linux 5.8 (https://git.kernel.org/linus/87676cfca14171fc4c99d96ae2f3e87780488ac4), vdso.so does not have PT_GNU_EH_FRAME. Therefore unwinders (llvm-project libunwind, nongnu libunwind, libgcc_s.so.1) ignore its unwind tables. In gdb, gdb/aarch64-linux-tdep.c recognizes the two instructions and encodes how the kernel sets up the ucontext_t structure.
Previously, vdso.so generated a small set of CFI instructions to encode X29 (FP) and X30 (LR). 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28% llvm-dwarfdump -eh-frame vdso.so
000000c0 0000001c 00000000 CIE
Format: DWARF32
Version: 1
Augmentation: "zRS"
Code alignment factor: 4
Data alignment factor: -8
Return address column: 30
Augmentation data: 1B
DW_CFA_def_cfa: WSP +0
DW_CFA_def_cfa: W29 +0
DW_CFA_offset: W29 0
DW_CFA_offset_extended_sf: W30 8
DW_CFA_nop:
DW_CFA_nop:
DW_CFA_nop:
CFA=W29: W29=[CFA], W30=[CFA+8]
000000e0 00000010 00000024 FDE cie=000000c0 pc=000005b0...000005b8
Format: DWARF32
DW_CFA_nop:
DW_CFA_nop:
DW_CFA_nop:
0x5b0: CFA=W29: W29=[CFA], W30=[CFA+8]
However, there was a serious problem: CFI cannot describe a signal trampoline frame. AArch64 does not define a register number for PC and provides no direct way to encode the PC of the previous frame. Instead, it sets return_address_register to X30 and the unwinder updates the PC to whatever value the saved X30 is. Actually, with unw_get_reg(&cursor, UNW_REG_IP, &pc); unw_get_reg(&cursor, UNW_AARCH64_X30, &x30);, we know pc == x30. This approach works fine when LR forms a chain since we know between two adjacent frames, the sets {PC, X30} differ by one element. However, when unwinding through the signal trampoline, the CFI can describe the previous PC but not the previous X30.
musl x86-64
src/signal/x86_64/restore.s implements a signal trampoline __restore_rt. There is no .eh_frame information.
nongnu libunwind does not know that __restore_rt is a signal trampoline (unw_is_signal_frame always returns 0). On ELF targets, -O1 and above typically imply -fomit-frame-pointer and many functions do not save RBP. Note: some functions may save RBP even with -fomit-frame-pointer.
In the absence of a valid frame chain, combined with the fact that nongnu libunwind does not recognize Linux x86-64's signal trampoline, libunwind cannot unwind through the __restore_rt frame. gdb recognizes the signal trampoline frame and with its FP-based unwinding it can retrieve several frames, but not the ones above raise.
1 | % ld.lld @response.release.txt && ./nongnu |
If musl is built with -fno-omit-frame-pointer, nongnu libunwind will use its FP-based fallback (see src/x86_64/Gstep.c). The output looks like: 1
2
3
4
5pc=0x0000000000206ada sp=0x00007fffd51b1830 0 ./nongnu:
pc=0x00007f0f09352858 sp=0x00007fffd51b2040 0 /home/ray/musl/out/release-fp/lib/libc.so:__setjmp
pc=0x0000000000206aaa sp=0x00007fffd51b2db0 0 ./nongnu:main
pc=0x00007f0f092f88ec sp=0x00007fffd51b2dd0 0 /home/ray/musl/out/release-fp/lib/libc.so:
pc=0x00000000002069d6 sp=0x00007fffd51b2e00 0 ./nongnu:_start
unw_step uses the saved RBP to infer RSP/RBP/RIP in the previous frame. If the signal handler saves RBP and calls unw_step, the saved RBP is essentially the RBP value in the signal trampoline frame. 1
2
3rbp_loc = DWARF_LOC(rbp, 0);
rsp_loc = DWARF_VAL_LOC(c, rbp + 16);
rip_loc = DWARF_LOC (rbp + 8, 0);
Actually, not every source file needs to be built with -fno-omit-frame-pointer. We just need to build the source files that transfer control to the user program, and their callers. For this example, building src/signal/raise.c with -fno-omit-frame-pointer allows us to unwind to main. Additionally rebuilding src/env/__libc_start_main.c allows us to unwind to _start.
musl's Makefile specifies -fno-asynchronous-unwind-tables (see option to enable eh_frame for a 2011 discussion). If CFLAGS -g is specified, libc.so will have .debug_frame. gdb can retrieve the caller of raise: 1
2
3
4
5#0 handler (signo=10) at a.c:9
#1 <signal handler called>
#2 __restore_sigs (set=set@entry=0x7fffffffe240) at ../../arch/x86_64/syscall_arch.h:40
#3 0x00007ffff7fa36e0 in raise (sig=sig@entry=10) at ../../src/signal/raise.c:11
#4 0x00000000002071ff in main () at a.c:33
nongnu libunwind can be built with --enable-debug-frame to support .debug_frame. Unfortunately, since it does not recognize the signal trampoline, it cannot retrieve the main frame for this example.
Unwinders' compatibility with libc implementations
The values represent how the unwinder unwinds through the signal trampoline frame.
| Linux glibc | Linux musl | |
| nongnu libunwind AArch64 | recognizes signal trampoline in VDSO | not tested |
| nongnu libunwind x86-64 | .eh_frame in libc.so.6 | unwindable if FP is enabled |
| gdb AArch64 | recognizes signal trampoline in VDSO | not tested |
| gdb x86-64 | recognizes signal trampoline | recognizes signal trampoline |
Links to signal trampoline frame related code
- gcc
libgcc/config/aarch64/linux-unwind.h:aarch64_fallback_frame_state - gdb
gdb/aarch64-linux-tdep.c:aarch64_linux_rt_sigframe,gdb/amd64-linux-tdep.c:amd64_linux_sigtramp_start - llvm-project libunwind https://reviews.llvm.org/D90898
- Linux kernel
arch/x86/kernel/signal.c:setup_rt_frame
Core dump
The kernel core dumper coredump.c is simple. The glibc __restore_rt page or the VDSO is not prioritized in the presence of a core file limit. If the page is missing in the core file, gdb prog core -ex bt -batch will not be able to unwind past the signal trampoline. A userspace core dumper may be handy.