This article describes linker notes about Portable Executable (PE)
and Common Object File Format (COFF) used on Windows and UEFI
environments.
In ELF, an object file can be a relocatable file, an executable file,
or a shared object file. On Windows, the term "object file" usually
refers to relocatable files like ELF. Such files use the Common Object
File Format (COFF) while image files (e.g. executables and DLLs) use the
Portable Executable (PE) format.
Input files
The input files to the linker can be object files, archive files, and
import libraries. GNU ld and lld-link allow linking against DLL files
without an import library.
If a DSO has an undefined STB_GLOBAL symbol that is
defined in a relocatable object file but not exported, should the
--no-allow-shlib-undefined feature report an error? You may
want to check out Dependency
related linker options for a discussion of this option and the symbol
exporting rule.
AddressSanitizer (ASan) is a compiler technology that detects
addressability-related memory errors with some additional checks. It
consists of two components: compiler instrumentation and a runtime
library. To put it simply,
The compiler instruments global variables, stack frames, and heap
allocations to monitor shadow memory.
The compiler also instruments memory access instructions to verify
shadow memory.
In case of an error, the inserted code invokes a callback
(implemented in the runtime library) to report the error along with a
stack trace. Typically, the program will terminate after displaying the
error message.
This article describes global variable instrumentation.
Global variable
instrumentation
AddressSanitizer instruments certain defined global variables of LLVM
external or internal linkage. To be instrumented, the variable must
satisfy a bunch of conditions.
It is not thread-local.
It has a smaller alignment.
It is not synthesized by LLVM.
It does not have the no_sanitize_address attribute in
LLVM IR. Variables receive this attribute when annotated as
__attribute__((no_sanitize("address"))) or
__attribute__((disable_sanitizer_instrumentation)) in
C/C++.
1 2
int g0; constlong g1 = 42;
Each instrumented global variable is padded with a right redzone to
detect out-of-bounds accesses.
On ELF platforms, by default (since Clang 17.0) each instrumented
global variable receives an associated __asan_global_$name
variable, which is located within the asan_globals section.
Additionally, there are several related variables, including some
unnamed ones (@0 and @1), as well as
__odr_asan_gen_g0 and __odr_asan_gen_g1, along
with metadata nodes (!0 and !1), which we will
discuss in more detail later."
The module constructor asan.module_ctor processes
garbage-collectable asan_globals input sections. This
constructor invokes a runtime callback to register the instrumented
global variables, which involves poisoning the redzone and conducting
ODR violation checks. I will discuss ODR violation checking later.
void __asan_register_globals(__asan_global *globals, uptr n) { if (!flags()->report_globals) return; ... for (uptr i = 0; i < n; i++) RegisterGlobal(&globals[i]);
// Poison the metadata. It should not be accessible to user code. PoisonShadow(reinterpret_cast<uptr>(globals), n * sizeof(__asan_global), kAsanGlobalRedzoneMagic); }
staticvoidRegisterGlobal(const Global *g){ ... if (CanPoisonMemory()) PoisonRedZones(*g); }
Every full granule in the shadow of the redzone is filled with 0xf9
(kAsanGlobalRedzoneMagic) while a partial granule is filled
in a manner similar to partially-addressable stack memory.
If an access occurs within a redzone byte poisoned by 0xf9 or within
a partial redzone preceding 0xf9, the runtime will report a
global-buffer-overflow error. Here is an example:
The global variable poisoning mechanism offers a straightforward
means to detect differences in variable definitions between two
components, such as between the main executable and a shared object, or
between two shared objects. This can be considered a category of ODR
violations.
==1299789==HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_odr_violation=0 SUMMARY: AddressSanitizer: odr-violation: global 'var' at a.cc in /tmp/c/a ==1299789==ABORTING
The default mode, detect_odr_violation=2, also prohibits
symbol interposition on variables. If you change long to
int in b.cc, you will still encounter an
odr-violation error. In contrast, with
detect_odr_violation=1, errors are suppressed if the
registered variables are of the same size.
For a variable named $var, a one-byte variable,
__odr_asan_gen_$var, is created with the original linkage
(essentially must be external).
If $var is defined in two instrumented modules, their
__odr_asan_gen_$var symbols reference to the same copy due
to symbol interposition. When registering $var, the runtime
checks whether __odr_asan_gen_$var is already 1, and if
yes, the program has an ODR violation; otherwise
__odr_asan_gen_$var is set to 1.
If a.supp contains the following text, running the
program with the environment variable
ASAN_OPTIONS=suppressions=a.supp suppresses errors due to
the variable name var.
1
odr_violation:^var$
An ODR violation is reported for two different linked units, say,
exe and b.so. With static linking, the issue
can be suppressed due to archive member extraction semantics if the
b.a member is not extracted.
ODR indicator
The previous example uses
-fsanitize-address-use-odr-indicator.
Prior to Clang 16,
-fno-sanitize-address-use-odr-indicator was the default for
non-Windows platforms. The runtime checks checks whether a variable has
been registered by verifying whether its redzone has been poisoned, and
reports an ODR violation when the redzone has been poisoned.
This mode eliminates the need for an additional variable like
__odr_asan_gen_$var, but it can lead to interaction issues
when mixing instrumented and uninstrumented components. In the case of a
shared object, if the reference to $var in
__asan_global_$var is interposed with an uninstrumented
variable due to symbol interposition, it may result in a spurious error
stating, "The following global variable is not properly aligned."
For Clang 16, I introduced the use of
-fsanitize-address-use-odr-indicator by default for
non-Windows targets (see https://reviews.llvm.org/D137227).
The definition of f in foo.cc is
instrumented, resulting in the creation of __asan_global_f.
However, the executable actually accesses the copy created by the linker
due to copy relocation.
When -asan-use-private-alias=1 is in effect (the default
since Clang 16), the __asan_global_f variable references
the unused copy inside the shared object. The executable accesses the
copy-relocated variable, whose redzone is not poisoned, resulting in no
error.
Conversely, when -asan-use-private-alias=0 is in effect,
the __asan_global_f variable references the copy-relocated
variable and poisons the redzone within the executable. Consequently,
accessing f[5] leads to the expected error.
Garbage collection
Since Clang 17, asan.module_ctor is, by default, placed
in a COMDAT group. When multiple instrumented relocatable object files
are linked together, only one asan.module_ctor is
retained.
__asan_global_g0 is positioned in a section that links
to the section defining g0 using the
SHF_LINK_ORDER flag. During linking, if the linker discards
the section defining g0, the asan_globals
section containing __asan_global_g0 will also be discarded.
For more detail on SHF_LINK_ORDER, you can refer to Metadata
sections, COMDAT and SHF_LINK_ORDER.
Before Clang 17, the default behavior was to use
-fno-sanitize-address-globals-dead-stripping. In this mode,
the instrumentation places pointers to instrumented global variables in
a metadata array and calls __asan_register_globals.
__asan_register_globals then iterates over the array and
registers each global variable.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
@g0= dso_local global { i32, [28xi8] } zeroinitializer,align32 @g1= dso_local global { i64, [24xi8] } zeroinitializer,align32
asan.module_ctor references the metadata array
@0, which, in turn, references @1 and
@2. @1 and @2 reference the
global variables g0 and g1, respectively. This
unfortunately indicates that g0 and g1 cannot
be discarded by section-based garbage collection.
It's important to note that this version of
asan.module_ctor is not placed within a COMDAT group. In
another compile unit, a separate asan.module_ctor
references a different metadata array. As a result, these
asan.module_ctor functions cannot share the same
implementation.
In a linked component, both __asan_init and
__asan_version_mismatch_check_v8 will be called multiple
times, incurring a small overhead.
Regrettably, the default setting of
-fsanitize-address-globals-dead-stripping in Clang 17 had a
bug. Specifically, when there are no global variables, and the unique
module ID is non-empty, a COMDAT asan.module_ctor is
created without any __asan_register_elf_globals calls. If
this COMDAT is selected as the prevailing copy by the linker, the
linkage unit will lack a __asan_register_elf_globals call,
resulting in an unpoisoned redzone and a non-functional ODR violation
checker.
I have fixed this in the main branch (#67745) but
LLVM 17.0.2 does not contain the fix.
Global variable metadata
Before Clang 15, Clang's instrumentation included
llvm.asan.globals, and the AddressSanitizer runtime
required its object file feature for symbolization.
AddressSanitizer provides a check to detect whether a dynamic
initializer for one global variable accesses dynamically initialized
global variables defined in another compile unit, which helps identify
certain initialization order issues. This catches certain initialization
order fiasco issues.
Here is an example:
1 2 3 4 5 6 7 8 9 10 11 12 13
cat > a0.cc <<'eof' #include <stdio.h> extern int a1; static int fa0() { return 1; } int a0 = fa0(); int main() { printf("%d %d\n", a0, a1); } eof cat > a1.cc <<'eof' extern int a0; static int fa1() { return a0+1; } int a1 = fa1(); eof clang++ -fsanitize=address a0.cc a1.cc -o a
% ASAN_OPTIONS=strict_init_order=1 ./a ================================================================= ==124921==ERROR: AddressSanitizer: initialization-order-fiasco on address 0x5577b1cd6b00 at pc 0x5577b12fbbca bp 0x7ffe75a0a280 sp 0x7ffe75a0a260 READ of size 4 at 0x5577b1cd6b00 thread T0 #0 0x5577b12fbbc9 in fa1() /tmp/t/d/a1.cc:2:27 #1 0x5577b12fbbec in __cxx_global_var_init /tmp/t/d/a1.cc:3:10 #2 0x5577b12fbc64 in _GLOBAL__sub_I_a1.cc /tmp/t/d/a1.cc #3 0x7ff44e0107f5 in call_init csu/../csu/libc-start.c:145:3 #4 0x7ff44e0107f5 in __libc_start_main csu/../csu/libc-start.c:347:5 #5 0x5577b11b46d0 in _start (/tmp/t/d/a+0x766d0)
0x5577b1cd6b00 is located 0 bytes inside of global variable 'a0' defined in '/tmp/t/d/a0.cc:4' (0x5577b1cd6b00) of size 4 registered at: #0 0x5577b11d1da4 in __asan_register_globals /usr/local/google/home/maskray/llvm/compiler-rt/lib/asan/asan_globals.cpp:363:3 #1 0x5577b11d2181 in __asan_register_elf_globals /usr/local/google/home/maskray/llvm/compiler-rt/lib/asan/asan_globals.cpp:346:3 #2 0x5577b12fbb57 in asan.module_ctor a0.cc #3 0x7ff44e0107f5 in call_init csu/../csu/libc-start.c:145:3 #4 0x7ff44e0107f5 in __libc_start_main csu/../csu/libc-start.c:347:5
SUMMARY: AddressSanitizer: initialization-order-fiasco /tmp/t/d/a1.cc:2:27 in fa1() ...
When check_initialization_order is enabled, while
strict_init_order is disabled, AddressSanitizer performs a
weak check allowing a compile unit that is about to be initialized to
access global variables in an already initialized compile unit. In this
scenario, the previous example does not result in an error:
For the following case, the weak check can still catch the
initialization order fiasco:
1 2 3 4 5 6 7 8 9 10 11 12 13
cat > a0.cc <<'eof' #include <stdio.h> extern int a1; int a0 = []() { return a1-1; }(); int main() { printf("%d %d\n", a0, a1); } eof cat > a1.cc <<'eof' extern int a0; static int fa1() { return 2; } int a1 = fa1(); eof clang++ -g -fsanitize=address a0.cc a1.cc -o a ASAN_OPTIONS=check_initialization_order=1:strict_init_order=0 ./a
Clang translates C++ dynamic initialization into a global
initialization function within the llvm.global_ctors list.
AddressSanitizer augments this global initialization function with
__asan_before_dynamic_init and
__asan_after_dynamic_init. These two functions work
together to check for initialization order issues when
check_initialization_order is enabled.
For instrumented global variables with initializers, the
has_dynamic_init variable in the __asan_global
metadata is set to true. These variables are collected into the
dynamic_init_globals array.
__asan_before_dynamic_init is called for each compile
unit. This function iterates over dynamic_init_globals and
poisons those whose DynInitGlobal::initialized value is
false. Subsequently, the global initialization function is executed. If
it accesses the poisoned memory, it triggers a report for an
initialization order issue. Following this,
__asan_after_dynamic_init processes these global variables,
unpoisoning them.
Clang is a C/C++ compiler that generates LLVM IR and utilitizes LLVM
to generate relocatable object files. Using the classic three-stage
compiler structure, the stages can be described as follows:
1
C/C++ =(front end)=> LLVM IR =(middle end)=> LLVM IR (optimized) =(back end)=> relocatable object file
If we follow the internal representations of instructions, a more
detailed diagram looks like this:
1
C/C++ =(front end)=> LLVM IR =(middle end)=> LLVM IR (optimized) =(instruction selector)=> MachineInstr =(AsmPrinter)=> MCInst =(assembler)=> relocatable object file
LLVM and Clang are designed as a collection of libraries. This post
describes how different libraries work together to create the final
relocatable object file. I will focus on how a function goes through the
multiple compilation stages.
Since 2012, LLVM has relied on its self-hosted Phabricator instance
on Google Cloud Platform for code review, but now it's making a
transition to GitHub pull requests. In this post, I'll share my
perspective on this switch, highlighting GitHub offers significant
benefits in some areas while having major drawbacks in the review
process.
I may update this article as the process stabilizes further.
Transition to GitHub pull
requests
The move to GitHub pull requests has been a topic of discussion over
the past few years. Several lengthy threads on the subject have
emerged:
This article describes some notes about MIPS with a focus on the ELF
object file format, GCC, binutils, and LLVM/Clang.
In the llvm-project project, I sometimes find myself assigned as a
reviewer for MIPS patches. I want to be transparent that I have no
interest in MIPS, but my concern lies with the specific components that
are impacted (Clang driver, ld.lld, MC, compiler-rt, etc.). Therefore,
regrettably, I have to spend some time studying MIPS.
Using copper as a mirror, one can straighten their attire; using the
past as a mirror, one can understand rise and fall; using people as a
mirror, one can discern gains and losses. -- 贞观政要
clangDriver is the library implementing the compiler driver for
Clang. It utilitizes LLVMOption to process command line options. As
options are processed when required, as opposed to use a large
switch, Clang gets the ability to detect unused options
straightforwardly.