cquery最近改动与libclang.so一字节补丁

cquery最近改动

cquery介绍参看使用cquery：C++ language server。

最近cquery改动比较多(终于要熬过这个圣诞了)：

  • 可执行文件从build/app变到build/release/bin/cquery了，支持release/debug/asan等多种waf variants，使用RPATH
  • ./waf configure --bundled=5.0.1可以用上最新出炉的clang+llvm 5.0.1
  • Riatre把Windows构建修好了 #154
  • FreeBSD可以使用了 #155及third party库改动，感谢ngkaho1234把sparsepp FreeBSD kvm搞定
  • 不需要在initializationOptions里指定resourceDir了，感谢jiegec的#137
  • 各种模板改进和function template/class template内函数引用的支持。支持了CXCursor_OverloadedDeclRef函数调用#174，但template call template clang-c接口没有暴露相应信息，可能无解。
  • textDocument/hover信息把函数名插入到函数类型里。用了一些heuristics处理_Atomic decltype() throw() __attribute(()) typeof()，碰到-> int(*)()这种还是没救的，数组括号也不好，但大多数情况都显示得不错的。
  • 加入了实验性的--enable-comments，检索注释。#183 #188 #191 注释和原来的声明信息一起显示，带来了UI的挑战。
  • VSCode使用浮动窗口，显示多行textDocument/hover不成问题。但Emacs lsp-mode和LanguageClient-neovim就遇到一些困难https://github.com/autozimu/LanguageClient-neovim/issues/224 https://github.com/emacs-lsp/lsp-ui/issues/17
  • workspace/symbol模糊匹配 #182
  • danielmartin自己的repo加了实验性的textDocument/formatting，格式化。这必须用clang C++ API，作者有一些顾虑。

用了一个O(n^2) sequence alignment算法，根据编辑距离、camelCase等启发因素排序候选符号(func,type,path,...)。以foo bar为模式会返回fooBar foobar foozbar等，而fooBar排在前面。Emacs xref-find-apropos会自作聪明地把模式用空格分割后当作正规表达式转义，需要覆盖掉。

libclang handleReference null pointer dereference

下面介绍重点，libclang一字节补丁。

This is a longstanding issue bothering us, with all the 3 bundled clang+llvm versions: --bundled-clang={4.0.0,5.0.0,5.0.1}.

topisani raised the topic in https://gitter.im/cquery-project/Lobby and I fianlly made up my mind to diagnose it.

https://github.com/jacobdufault/cquery/issues/192

1
2
3
4
5
6
% cd /tmp
% git clone https://github.com/nlohmann/json
% cd json
% (mkdir build; cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..)
% ln -s build/compile_commands.json
% nvim test/src/unit-constructor1.cpp  # or other editor

By appending --log-file /tmp/cq.log to the cquery command line, we can see the error message in /tmp/cq.log: 

1
2
indexer.cc:1892  WARN| Indexing /tmp/json/test/src/unit-iterators2.cpp failed with errno=1
libclang: crash detected during indexing TU

errno=1 indicates CXError_Failure，libclang uses a sigaction+setjmp based crash recovery mechanism to recover from SIGSEGV and returns CXError_Failure. Sadly cquery does not report enough diagnostics for this. For these files, there is no hover, definition, or references information.

https://github.com/jacobdufault/cquery/issues/219 HighCommander4 narrowed it down to a very simple reproduce： 

1
2
3
4
5
template <typename>
struct actor;

template <template <typename> class Actor = actor>
struct terminal;

default template argument = actor causes the null pointer dereference.

In https://github.com/llvm-mirror/clang/blob/master/tools/libclang/CXIndexDataConsumer.cpp#L203

1
2
3
handleReference(ND, Loc, Cursor,
                dyn_cast_or_null<NamedDecl>(ASTNode.Parent),
                ASTNode.ContainerDC, ASTNode.OrigE, Kind);

dyn_cast_or_null<NamedDecl>(ASTNode.Parent) may return NULL. In some code executed later on https://github.com/llvm-mirror/clang/blob/master/tools/libclang/CXIndexDataConsumer.cpp#L935:

1
2
ContainerInfo Container;
getContainerInfo(DC, Container);

getContainerInfo tries to cast DC, which uses a field in DC and causes a null pointer dereference.

I have sent the patch to clang upstream for review, but clang+llvm 5.0.1 was just released. The holiday season is approaching and it is unrealistic to get this submitted and get a new release in the near future. For Linux users who do not want to build clang+llvm from source, my makeshift is ./waf configure --bundled-clang=5.0.1 with

1
2
3
4
5
# --bundled-clang=5.0.1
% printf '\x4d' | dd of=build/release/lib/clang+llvm-5.0.1-x86_64-linux-gnu-ubuntu-14.04/lib/libclang.so.5.0 obs=1 seek=$[0x47aece] conv=notrunc

# --bundled-clang=4.0.0
% printf '\x4d' | dd of=build/release/lib/clang+llvm-4.0.0-x86_64-linux-gnu-ubuntu-14.04/lib/libclang.so.4.0 obs=1 seek=$[0x4172b5] conv=notrunc

What?

Description

Let's check the problematic function in libclang:

1
2
3
4
5
6
7
8
9
10
11
12
13
bool CXIndexDataConsumer::handleReference(const NamedDecl *D, SourceLocation Loc,
                                      CXCursor Cursor,
                                      const NamedDecl *Parent,
                                      const DeclContext *DC,
                                      const Expr *E,
                                      CXIdxEntityRefKind Kind) {
  if (!CB.indexEntityReference)  // redundant check since it must be non NULL
    return false;

  if (!D)
    return false;
  if (Loc.isInvalid())
    return false;

According to System V x86-64 ABI, the arguments of this function are passed in the following way:

1
2
3
4
5
6
7
8
this: rdi
D: rsi
Loc: rdx
Cursor: stack
Parent: rcx
DC: r8
E: r9
Kind: stack

cquery sets CB.indexEntityReference to OnIndexReference so this field is guaranteed to be non NULL.

Scroll down the assembly listing a little bit and search for the if (!CB.indexEntityReference) condition:

1
2
3
4
5
6
7
8
9
10
11
% objdump -M intel -Cd build/release/lib/clang+llvm-5.0.1-x86_64-linux-gnu-ubuntu-14.04/lib/libclang.so.5 --start-address 0x47ae90 --stop-address 0x47b190
......
  47aeb0:	45 31 ed             	xor    r13d,r13d
  47aeb3:	45 85 ff             	test   r15d,r15d
  47aeb6:	0f 84 19 03 00 00    	je     47b1d5
  47aebc:	48 85 ed             	test   rbp,rbp
  47aebf:	0f 84 10 03 00 00    	je     47b1d5
  47aec5:	49 8b 44 24 18       	mov    rax,QWORD PTR [r12+0x18]  # load `IndexerCallbacks &CB`, which is actually a pointer
  47aeca:	48 8b 40 38          	mov    rax,QWORD PTR [rax+0x38]  # load CB.indexEntityReference
  47aece:	48 85 c0             	test   rax,rax     # a redundant check to see if it is null; we replace it with check of `DC`
  47aed1:	0f 84 fe 02 00 00    	je     47b1d5 <clang::cxindex::CXIndexDataConsumer::handleReference(clang::NamedDecl const*, clang::SourceLocation, CXCursor, clang::NamedDecl const*, clang::DeclContext const*, clang::Expr const*, CXIdxEntityRefKind)+0x345>

This redundant if statement produces 0x47aeca test rax,rax. If we replace it with if (!DC) (because DC may be NULL and we should avoid null pointer dereference), since DC is passed in the register R8, we may use test r8,r8: 

1
2
48 85 c0  test rax,rax
4d 85 c0  test r8,r8

It is now clear that we only need to patch one byte, i.e. the aforementioned printf+dd hack. For radare2 users, r2 -nwqc 'wx 4d @ 0x47aece' build/release/lib/clang+llvm-5.0.1-x86_64-linux-gnu-ubuntu-14.04/lib/libclang.so.5.0.

Sadly radare2 dropped the ball when it was in need and I had to resort to printf+dd... Its assembling of test r8,r8 was incorrect https://github.com/radare/radare2/issues/9071😢