When can glibc be built with Clang?

In September, I wrote "So, dear glibc, will you be happy with my sending Clang patches?" in Build glibc with LLD 13. We have come to a turning point.

In Linux Plumbers Conference 2021, at the glibc Birds of a Feather session, I asked the Clang buildability question to the glibc stewards. (Interlude: I did not realize that I should attend the conference (it was a great opportunity from an outlier to meet some glibc folks). In Tuesday, Wei Wu (lazyparser) kindly gave me his account: "想去参加LPC么?我会议太多了今天参加不过来". I happily accepted it and typed the question during the glibc session.)

So I got positive responses. "Carlos: Yes, we could be happy with clang buildability." "Joseph: Patches should be split into logical changes." This is really great news! My unnesting patch had sat there for a while and I was unsure about the Clang buildability interest.

Of course, every change needs to be assessed where they belong to, LLVM/Clang or glibc.

My interest in Clang/LLD buildability did not arise suddenly. It has been multiple years. I made my first patch improving LLD's glibc buildability in June 2018. Probably since June 2019, I pushed first LLVM fix mproving glibc buildability [MC][ELF] Don't create relocations with section symbols for STB_LOCAL ifunc. I have sporadic contribution since then. Today I think there is not too much to improve on LLVM/Clang's side, but many need glibc adapatation.

As a rough estimate, we may need 30 patches to fix glibc build on x86-64 for the --disable-werror configuration. Fixing other popular architectures or all tests will take more. Adrian Ratiu mentioned that shebs/clangify exists in A tale of two toolchains and glibc. The branch in the 2018 era has about 30 commits.


Below I will introduce the main challenges (GCC nested functions, asm label after first use, Clang's less permissive warnings) and a selective set of miscellaneous changes.

GCC nested functions

glibc's dynamic loader (often called ld.so) implementation (elf/ plus arch-specific code in sysdeps/*/) made extensive use of a feature called "nested functions". https://gcc.gnu.org/onlinedocs/gcc/Nested-Functions.html says

A nested function is a function defined inside another function. Nested functions are supported as an extension in GNU C, but are not supported by GNU C++.

This is a no-go for Clang which does not support this feature (error: function definition is not allowed here). The dynamic loader usage actually impacted readability and debuggability, so people agreed that the usage should simply be removed.

As a worst example, the dynamic loader had a function like:

1
2
3
4
5
6
7
8
#ifndef RESOLVE_MAP
static
#else
auto
#endif
inline void __attribute__ ((unused, always_inline))
elf_get_dynamic_info (struct link_map *l)
{

Its file could be included twice, one with RESOLVE_MAP defined and one without. It was difficult to understand the code intention.

As of 2021-10-07, the usage in the dynamic loader has been removed by my elf: Avoid nested functions in the loader [BZ #27220]. Adhemerval Zanella fixed some fallout.

The glibc dynamic loader code is extremely tricky. I think not taking inspiration from NetBSD code in 1992 was unfortunate.

Remaining usage include:

asm label after first use

In GCC and Clang, an asm label can change the symbol name for a definition or a non-definition declaration. glibc makes heavy use of this feature even in public headers.

#pragma redefine_extname is a similar feature which can change the symbol name. In Clang, #pragma redefine_extname is converted to an asm label and shares code with the asm label implementation.

Clang before 12.0.0 had an issue that a built-in function ignored the asm label. I fixed it and made some notes on https://maskray.me/blog/2020-10-15-intra-call-and-libc-symbol-renaming (TODO: translate it into English).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
typedef unsigned long size_t;

#ifdef ASM_LABEL
extern void *memcpy(void *, const void *, size_t) asm("__GI_memcpy");
#else
#pragma redefine_extname memcpy __GI_memcpy // before the first use, works
extern void *memcpy(void *, const void *, size_t);
#endif

void *test_memcpy(void *dst, const void *src, size_t n) {
// Clang < 12: callq memcpy
// Clang >= 12: callq __GI_memcpy
return memcpy(dst, src, n);
}

Unfortunately, there is a limitation. If a #pragma redefine_extname is declared after the first use of the symbol, the rename operation will be silently ignored.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// a.c
typedef unsigned long size_t;

extern void *memcpy(void *, const void *, size_t);

void *test_memcpy(void *dst, const void *src, size_t n) {
return memcpy(dst, src, n);
}

#ifdef ASM_LABEL
extern void *memcpy(void *, const void *, size_t) asm("__GI_memcpy");
#else
#pragma redefine_extname memcpy __GI_memcpy // before the first use, works
#endif

If an asm label applies to a redeclaration after the first use, Clang will kindly give an error to notify that it cannot handle the situation:

1
2
3
4
5
6
7
% clang a.c -S -o - | grep callq
callq memcpy@PLT
% clang a.c -S -o - -DASM_LABEL
a.c:18:14: error: cannot apply asm label to function after its first use
extern void *memcpy(void *, const void *, size_t) asm("__GI_memcpy");
^ ~~~~~~~~~~~~~
1 error generated.

glibc makes extensive usage of hidden_proto and libc_hidden_proto, where such problems may occur in many places.

It seems that disabling __USE_EXTERN_INLINES can bypass issues with some symbols.

_Float128

long double and float128 support in GCC is a huge mess.

In its PowerPC port, long double has 3 mangling schemes:

  • -mlong-double-64: e
  • -mlong-double-128 -mabi=ibmlongdouble: g
  • -mlong-double-128 -mabi=ieeelongdouble: u9__ieee128 (gcc <= 8.1: U10__float128)

In its x86 port, two different long double configurations share the same mangling code.

  • -mlong-double-64: e
  • -mlong-double-80: e
  • -mlong-double-128: g

(I fixed some Clang issues in https://reviews.llvm.org/D64276.)

__float128 has the same mangling code with 128-bit long double but is considered a distinct type in both C and C++. So if you have two overloads foo(__float128) and foo(long double), you will get a compiler/linker error due to conflicting symbols.

_Float128 is identical to __float128. _Float128 is available in C mode but not in C++ mode. When a future standard C++ introduces a float128 type, it has to be yet another new type to be different from the 128-bit long double.

1
2
3
4
5
6
7
8
9
// C
#include <stdio.h>

int main() {
// _Float128 = __float128
puts(_Generic((_Float128)0, __float128: "same", default: "different"));
// _Float128 != long double, even in -mlong-double-128 mode.
puts(_Generic((_Float128)0, long double: "same", default: "different"));
}

Anyhow, glibc builds some float128 code on some PowerPC, x86, and perhaps a few other architectures. Clang doesn't support _Float128. Falling back to typedef __float128 _Float128 is probably a good choice.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
--- a/sysdeps/x86/bits/floatn.h
+++ b/sysdeps/x86/bits/floatn.h
@@ -29,5 +29,6 @@
#if (defined __x86_64__ \
? __GNUC_PREREQ (4, 3) \
- : (defined __GNU__ ? __GNUC_PREREQ (4, 5) : __GNUC_PREREQ (4, 4)))
+ : (defined __GNU__ ? __GNUC_PREREQ (4, 5) : __GNUC_PREREQ (4, 4))) \
+ || defined __clang__
# define __HAVE_FLOAT128 1
#else
@@ -88,5 +89,5 @@ typedef __float128 _Float128;

/* __builtin_huge_valf128 doesn't exist before GCC 7.0. */
-# if !__GNUC_PREREQ (7, 0)
+# if !__GNUC_PREREQ (7, 0) && !defined __clang__
# define __builtin_huge_valf128() ((_Float128) __builtin_huge_val ())
# endif
@@ -97,5 +98,5 @@ typedef __float128 _Float128;
attempts to use _Float128 sNaNs will not work properly with older
compilers. */
-# if !__GNUC_PREREQ (7, 0)
+# if !__GNUC_PREREQ (7, 0) && !defined __clang__
# define __builtin_copysignf128 __builtin_copysignq
# define __builtin_fabsf128 __builtin_fabsq
@@ -109,5 +110,5 @@ typedef __float128 _Float128;
been a __builtin_signbitf128 in GCC and the type-generic builtin is
only available since GCC 6. */
-# if !__GNUC_PREREQ (6, 0)
+# if !__GNUC_PREREQ (6, 0) && !defined __clang__
# define __builtin_signbitf128 __signbitf128
# endif

-fheinous-gnu-extensions

Chris Lattner added the funny option in commit cda4d7e19619e136f2b597110b6a52ee9d641862. It has an associated comment for CheckAsmLValue:

1
2
3
GNU C has an extremely ugly extension whereby they silently ignore "noop" casts in places where an lvalue is required by an inline asm.
We emulate this behavior when -fheinous-gnu-extensions is specified, but provide a strong guidance to not use it.
This method checks to see if the argument is an acceptable l-value and returns false if it is a case we can handle.

This does look like wrong usage but also has a niche type checking feature:

1
2
3
4
5
6
7
8
9
10
// GCC
void foo() {
int i;
long l;
__asm ("" : "=r" ((unsigned)i));
__asm ("" : "=r" ((long)l));
__asm ("" : "=r" ((long long)l));
__asm ("" : "=r" ((int)l)); // error: lvalue required in ‘asm’ statement
__asm ("" : "=r" ((long)i)); // error: lvalue required in ‘asm’ statement
}
1
2
3
4
5
6
7
8
9
10
// Clang
void foo() {
int i;
long l;
__asm ("" : "=r" ((unsigned)i)); // error: invalid use of a cast in a inline asm context requiring an lvalue: remove the cast or build with -fheinous-gnu-extensions
__asm ("" : "=r" ((long)l)); // ...
__asm ("" : "=r" ((long long)l)); // ...
__asm ("" : "=r" ((int)l)); // ...
__asm ("" : "=r" ((long)i)); // ...
}

I attempted a fix [PATCH] include/longlong.h: Remove incorrect lvalue to rvalue conversion from asm output constraints. If GCC/glibc folks don't accept the loss of type checking, we probably have to carry around -fheinous-gnu-extensions.

1
2
3
4
5
6
7
8
9
10
11
--- a/stdlib/longlong.h
+++ b/stdlib/longlong.h
@@ -493,6 +493,6 @@ extern UDItype __umulsidi3 (USItype, USItype);
#define add_ssaaaa(sh, sl, ah, al, bh, bl) \
__asm__ ("add{q} {%5,%1|%1,%5}\n\tadc{q} {%3,%0|%0,%3}" \
- : "=r" ((UDItype) (sh)), \
- "=&r" ((UDItype) (sl)) \
+ : "=r" (sh), \
+ "=&r" (sl) \
: "%0" ((UDItype) (ah)), \
"rme" ((UDItype) (bh)), \

LLVM's integrated assembler

On most targets, Clang uses LLVM's integrated assembler. It lacks support for some unpopular features and may be stricter than GNU as in some cases.

1
2
3
4
5
6
7
8
9
% cat a.s
bndmov %bnd0,(%rsp)
bnd
jmp *%r11
% as a.s
% clang -c a.s
a.s:2:1: error: invalid instruction mnemonic 'bnd'
bnd
^~~

I attempted to improve sysdeps/x86_64/configure.ac but was asked to just remove the deprecated Intel MPX support. So I sent [PATCH] elf: Remove Intel MPX support (lazy PLT, ld.so profile, and LD_AUDIT).

As another example, sysdeps/powerpc/powerpc64/le/power10/memmove.S uses the directive .machine power9. For some other x86 projects I noticed that GNU as' .arch supports a very wide range of .arch operands.

In LLVM, *AsmParser files do not do a good job tracking what features are needed to assemble instructions. It is also tricky to recognize all GNU as supported operands. The ISA feature compatibility check, while looks nice, actually provides a lower value.

So I patched LLVM's x86 and PowerPC targets to ignore .arch and .machine operands.

A few sysdeps/i386/fpu/*.S files use .tfloat (80-bit extended precision floating point) which is unsupported by LLVM's integrated assembler.

Inline asm parsing

While GCC does minimal parsing of an inline asm statement (https://gcc.gnu.org/onlinedocs/gcc/Size-of-an-asm.html), Clang parses everything.

glibc uses an inline asm trick to get the constant value of a macro (e.g. a Linux syscall number). @@@ is invalid asm, so Clang rejects it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
% cat a.c
#include <linux/version.h>
#include <sys/syscall.h>
#include <bits/syscall.h>

int main() {
asm ("@@@%0@@@" :: "i"((long int)(X)));
}
% gcc -S -DX=__NR_mmap a.c -o - | grep @@@
@@@$9@@@
% clang -S -DX=__NR_mmap a.c -o /dev/null
a.c:6:8: error: unexpected token at start of statement
asm ("@@@%0@@@" :: "i"((long int)(X)));
^
<inline asm>:1:2: note: instantiated into assembly here
@@@$9@@@
^
1 error generated.

The fix is simple. We can just surround the invalid part with a pair of /* */. I avoid # because it may have weird interpretation on some exotic architecture (https://sourceware.org/binutils/docs/as/Comments.html#Comments).

More rigid semantic analysis and finicky warnings

_Static_assert

The first operand of a _Static_assert declaration must be an integer constant expression. C11 defines operands which are required to make up an integer constant expression.

6 An integer constant expression shall have integer type and shall only have operands that are integer constants, enumeration constants, character constants, sizeof expressions whose results are integer constants, _Alignof expressions, and floating constants that are the immediate operands of casts. Cast operators in an integer constant expression shall only convert arithmetic types to integer types, except as part of an operand to the sizeof or _Alignof operator.

10 An implementation may accept other forms of constant expressions.

An implementation may or may not accept a const object (e.g. const size_t allocation_size = 32768;) as an operand. Clang is simple: a const object cannot be used as an operand in a constant expression. GCC is inconsistent (PR102502):

  • -O2: accepted
  • -O2 -Wpedantic: warning: expression in static assertion is not an integer constant expression [-Wpedantic]
  • -O0: error: expression in static assertion is not constant
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
% cat reduce.i
const int __alloc_dir_allocation_size = 8;
void __alloc_dir() { _Static_assert(__alloc_dir_allocation_size, ""); }

% gcc reduce.i -c -std=c11
reduce.i: In function ‘__alloc_dir’:
reduce.i:2:37: error: expression in static assertion is not constant
2 | void __alloc_dir() { _Static_assert(__alloc_dir_allocation_size, ""); }
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
% gcc reduce.i -c -std=c11 -O1
% gcc reduce.i -c -std=c11 -O2
% gcc reduce.i -c -std=c11 -O2 -Wpedantic
reduce.i: In function ‘__alloc_dir’:
reduce.i:2:37: warning: expression in static assertion is not an integer constant expression [-Wpedantic]
2 | void __alloc_dir() { _Static_assert(__alloc_dir_allocation_size, ""); }
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~

% clang reduce.i -c -std=c11
reduce.i:2:37: error: static_assert expression is not an integral constant expression
void __alloc_dir() { _Static_assert(__alloc_dir_allocation_size, ""); }
^~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

sysdeps/unix/sysv/linux/opendir.c has such an issue, which has been fixed by linux: Fix a possibly non-constant expression in _Static_assert.