This article describes the dependency related linker options
--warn-backrefs. Deploying them in a build system can
improve build health.
-z defs (alias
--no-undefined) tells the
linker to report an error for an unresolved undefined non-weak symbol
from a relocatable object file. Executable links (
-pie) default to
-z defs while shared
object links (
-shared) default to
"Unresolved" means that (a) no other relocatable object file linked into the component provides a definition, and (b) no shared object linked into the component provides a definition.
When an undefined symbol in X is provided by a link-time shared
object Y, we can say that X depends on Y. The linker will record this
fact by adding a
DT_NEEDED dynamic tag. If Y has a
DT_NEEDED value is the SONAME;
DT_NEEDED value is the path of Y (either
absolute or relative).
I can think of several reasons that the loose default for
-shared links was chosen.
ELF supports interposition. Say, a shared object has an undefined symbol. The symbol may be provided by an arbitrary shared object or by the executable at runtime. By not requiring the dependencies to be fully specified, runtime can have flexibility.
There may be mutual references between two shared objects X and Y. We cannot break the tie with a regular approach if fully dependencies need to be specified. A variant is that the executable has X as its dependency while X also references symbols from the executable.
For (a), such unbounded flexibility does not quite fit into a build system. With a modular design, the libraries should have well-defined roles and dependencies. We do not substitute an arbitrary shared object for a link-time shared object. When we need such flexibility, we can define an interface and make several shared objects implement the interface.
Having fully specified dependencies makes the shipped shared object X convenient to use. It is likely that an executable link does not use X's dependency Y. It would feel awkward if the executable link needs to additionally link against Y when linking against X.
If you have read my article about ELF interposition, you should know that having the dependency information makes direct bindings possible, which can improve symbol lookup time for the dynamic loader. No ELF system other than Solaris has implemented direct bindings, though.
- indicates a bad layering of libraries. X and Y are no longer
isolated components. A change in X may affect Y and vice versa. The unit
testing for X needs to involve Y. With archives, you will need
--start-group X.a Y.a --end-groupwith GNU ld and gold. Actually merging X and Y is often a better strategy.
If we don't merge X and Y, http://blog.darlinghq.org/2018/07/mach-o-linking-and-loading-tricks.html mentions that the Mach-O approach for such circular dependencies is usually to link the libraries twice.
ld -o libfoo.dylib foo.o -flat_namespace -undefined suppress
The ELF counterpart is:
ld -shared foo.o -o foo.so
ld -shared bar.o -o bar.so
ld -shared -z defs foo.o bar.so -o foo.so
ld -shared -z defs bar.o foo.so -o bar.so
A build system may support archives as well as shared objects. An
archive is a collection of regular object files, with special archive
member selection semantics in the linker. (We will discuss the archive
member selection later.) If A.a needs definitions from B.a and you do
not supply B.a when linking a dependent executable
ld ... A.a instead of
ld ... A.a b.a), you
know that you may be welcomed by
undefined reference to
(GNU ld) or
undefined symbol: (ld.lld). In practice a build
system needs to track dependencies for archives.
-z now tells the linker to set the
dynamic tag. ld.so will resolve
eagerly and report an error if there is an unresolved symbol.
-z now can be seen as a relaxed
The full depencency requirement is moved from link-time to runtime.
--no-allow-shlib-undefined tells the linker to report an
error for an unresolved undefined symbol from a shared object.
Executable links (
-pie) default to
-z defs while shared object links (
ld -shared a.o b.so -o a.so, if
b.so has an undefined symbol not defined by
b.so's dependencies, the linker will
report an error.
gold and ld.lld do not recursively load
Instead, they report an error only when
DT_NEEDED list is all on the linker command line. Say,
b.so depends on
ld.lld -shared a.o b.so c.so -o a.so will not error for an
unresolved undefined symbol in
d.so is not on the linker
ld.lld -shared a.so b.so c.so d.so -o a.so
If a build system is
-z defs clean, it will also be
--no-allow-shlib-undefined clean. If a build system cannot
catch some propagated problems.
// a.cc => a.o => a.so (not linked with -z defs)
ld b.o a.so will report an error for the undefined
If we use Bazel layering check features as an analogy,
-z defs (link what you use) is like
layering_check (include what you use) while
--no-allow-shlib-undefined is a bit like
hdrs_check. See Layering check with
--no-allow-shlib-undefined checks shared objects
GNU ld checks shared objects recursively while gold and ld.lld don't. There are several reasons that ld.lld doesn't.
First, complexity. In the GNU ld manpage,
-rpath-link=dir lists 12 rules, which are just
overwhelming. Several points don't play well with cross linking.
Eliminating difference between native link and a cross link is an
important design choice of ld.lld. In addition, I know
--no-copy-dt-needed-entries can affect the behavior subtly,
which isn't even documented.
Second, performance. Recursive loading needs to parse additional shared objects which may slow down the link a bit.
--allow-shlib-undefined is an
unfortunate default for
-shared. Changing it may be
disruptive today. Mach-O and PE/COFF have many problems but this may be
a place where they got right. I have several paragraphs describing it in
Recursive loading of --allow-shlib-undefined gives me a feeling of
working around the wrong default.
Gentoo Linux has a quality assurance project to enable
This concept is required to understand
Please check out Symbol
VMS (now OpenVMS), Mach-O ld64, Windows link.exe, and ld.lld use a different design.
ld.lld ... definition.a reference.o, the archive
definition.a lists the defined symbols. When
definition.a, ld.lld uses "lazy symbols" to
represent the lazy definitions. Each lazy symbol has an associated
archive(member) name. When processing
undefined symbol can cause the lazy symbol to be fetched, i.e. a
definition.a member will be extracted.
This archive processing strategy is nice because in the absence of
ld.lld ... definition.a reference.o
ld.lld ... reference.o definition.a cause no ordering
--start-group is a no-op in ld.lld. The
traditional approach may iterate over the archive members more than
--start-group can exacerbate the problem. The
ld.lld approach turns out to be easier to implement and can improve
archive processing performance.
Let's talk about the case with 4 libraries: a, b, c, and d. The
dependency edges are: a->b, a->c, b->d, c->d. There is an
unspecified edge: b->c. Here the lazy symbol representation (without
--warn-backrefs) loses one major advantage of the
traditional approach: loose layering check.
Let's consider two link orders.
ld ... a.a c.a b.a d.amay lead to an error, because some members of
c.amay be dropped if they do not resolve a previously undefined symbol.
ld ... a.a b.a c.a d.ais fine, because after
b.ais processed, we should have seen all the symbol requirements on
a, b, c, dis a topological order of the full dependency list.
The layering check is loose because it only checks one particular topological order. Nevertheless, a build system using this option can catch many missing dependency edges.
Working at the binary level, the option can catch some problems not
detected by modules based layering check
clang -fmodule-name=X -fmodules-strict-decluse
error: module X does not depend on a module exporting 'string.h'),
e.g. using a declaration without including a header.
--warn-backrefs was added to check archive
processing compatibility problems with GNU ld. In ld.lld 11.0.0, I
--warn-backrefs to make the
compatibility checking reliable.
Because the archive name is remembered, the diagnostic (e.g.
warning: backward reference detected: foo in a1.o refers to a2.o)
is better than GNU ld's (no filename).
Linking sandwich problem
ld def1.a ref.o def2.a when both
def2.a define a symbol which is
ref.o. I call this a linking sandwich.
The traditional approach succeeds: the linker simply ignores
def1.a and then extracts a member of
ld.lld extracts a member of
def1.a when the undefined
ref.o is scanned. The
may or may not be extracted.
If extracted, the
def1.a member and
member have a duplicate symbol. If both definitions are
STB_GLOBAL, ld.lld will report a duplicate symbol
If not extracted, then GNU ld and ld.lld extract different archive
members. The most common incarnation is that
def2.a have the same path, e.g.
liblldCommon.a liblldCOFF.a liblldCommon.a. This case is
benign. I submitted https://reviews.llvm.org/D77522 to suppress the
--warn-backrefs diagnostic. If
def2.a have different paths, this is like an
one-definition-rule violation. I think if we ever report a warning, we
may need a new option.