2025-01-12

Understanding and improving Clang -ftime-report

Clang provides a few options to generate timing report. Among them, -ftime-report and -ftime-trace can be used to analyze the performance of Clang's internal passes.

-fproc-stat-report records time and memory on spawned processes (ld, and gas if -fno-integrated-as).
-ftime-trace, introduced in 2019, generates Clang timing information in the Chrome Trace Event format (JSON). The format supports nested events, providing a rich view of the front end.
-ftime-report: The option name is borrowed from GCC.

This post focuses on the traditional -ftime-report, which uses a line-based textual format.

Understanding `-ftime-report` output

The output consists of information about multiple timer groups. The last group spans the largest interval and encompasses timing data from other groups.

Up to Clang 19, the last group is called "Clang front-end time report". You would see something like the following.

2024-12-31

2024年总结

一如既往，主要在工具链领域耕耘。

Blogging

I have been busy creating posts, authoring a total of 31 blog posts (including this one). 7 posts resonated on Hacker News, garnering over 50 points. (https://news.ycombinator.com/from?site=maskray.me).

I have also revised many posts initially written between 2020 and 2024.

Mastodon: https://hachyderm.io/@meowray

2024-12-30

Skipping boring functions in debuggers

In debuggers, stepping into a function with arguments that involve function calls may step into the nested function calls, even if they are simple and uninteresting, such as those found in the C++ STL.

GDB

Consider the following example:

#include <cstdio>
#include <memory>
#include <vector>
using namespace std;

void foo(int i, int j) {
  printf("%d %d\n", i, j);
}

int main() {
  auto i = make_unique<int>(3);
  vector v{1,2};
  foo(*i, v.back()); // step into
}

When GDB stops at the foo call, the step (s) command will step into std::vector::back and std::unique_ptr::operator*. While you can execute finish (fin) and then execute s again, it's time-consuming and distracting, especially when dealing with complex argument expressions.

2024-12-25

Exporting Tweets

On https://x.com/settings/, click More -> Settings and privacy -> Download an archive of your data. Wait for a message from x.com: "@XXX your X data is ready" Download the archive.

1	cp data/tweets.js tweets.ts

Change the first line from window.YTD.tweets.part0 = [ to let part0 = [, and append

import { unescape } from "@std/html/entities";

let out = part0.map(tw => [new Date(tw.tweet.created_at), tw.tweet.full_text])
out.sort((a,b) => a[0] - b[0])

let yy0 = 0, mm0 = 0, str = ''
for (let i=0, j=0; i<=out.length; i++) {
  let d = i<out.length ? out[i][0] : new Date('9999-12-31')
  let yy = d.getYear()+1900, mm = d.getMonth()+1
  if (yy0 != yy) {
    if (str.length) {
      try {
        Deno.mkdirSync(String(yy0))
      } catch (e) {
      }
      Deno.writeTextFileSync(`${yy0}/index.md`, str)
    }
    yy0 = yy
    mm0 = 0
    str = `# ${yy0}\n`
    if (i == out.length) break
  }
  if (mm0 != mm) {
    str += `\n## ${yy}-${String(mm).padStart(2,'0')}\n`
    mm0 = mm
  }
  str += `\n${unescape(out[i][1]).replace(/(http(s)?:[-/.\w]+)/, "<$1>")}\n`
}

Then run deno run --allow-write=. tweets.ts

% cat 2022/index.md
# 2022

## 2022-01

tweet0

tweet1

## 2022-02

...

tweet0

tweet1

2024-12-22

Simplifying disassembly with LLVM tools

Both compiler developers and security researchers have built disassemblers. They often prioritize different aspects. Compiler toolchains, benefiting from direct contributions from CPU vendors, tend to offer more accurate and robust decoding. Security-focused tools, on the other hand, often excel in user interface design.

For quick disassembly tasks, rizin provides a convenient command-line interface.

2024-12-01

clang-format and single-line statements

The Google C++ Style is widely adopted by projects. It contains a brace omission guideline in Looping and branching statements:

For historical reasons, we allow one exception to the above rules: the curly braces for the controlled statement or the line breaks inside the curly braces may be omitted if as a result the entire statement appears on either a single line (in which case there is a space between the closing parenthesis and the controlled statement) or on two lines (in which case there is a line break after the closing parenthesis and there are no braces).

2024-11-17

Removing global state from LLD

LLD, the LLVM linker, is a mature and fast linker supporting multiple binary formats (ELF, Mach-O, PE/COFF, WebAssembly). Designed as a standalone program, the code base relies heavily on global state, making it less than ideal for library integration. As outlined in RFC: Revisiting LLD-as-a-library design, two main hurdles exist:

Fatal errors: they exit the process without returning control to the caller. This was actually addressed for most scenarios in 2020 by utilizing llvm::sys::Process::Exit(val, /*NoCleanup=*/true) and CrashRecoveryContext (longjmp under the hood).
Global variable conflicts: shared global variables do not allow two concurrent invocation.

I understand that calling a linker API could be convenient, especially when you want to avoid shipping another executable (which can be large when you link against LLVM statically). However, I believe that invoking LLD as a separate process remains the recommended approach. There are several advantages:

Build system control: Build systems gain greater control over scheduling and resource allocation for LLD. In an edit-compile-link cycle, the link could need more resources and threading is more useful.
Better parallelism management
Global state isolation: LLVM's global state (primarily cl::opt and ManagedStatic) is isolated.

2024-11-10

Keeping pace with LLVM: compatibility strategies

LLVM's C++ API doesn't offer a stability guarantee. This means function signatures can change or be removed between versions, forcing projects to adapt.

On the other hand, LLVM has an extensive API surface. When a library like llvm/lib/Y relies functionality from another library, the API is often exported in header files under llvm/include/llvm/X/, even if it is not intended to be user-facing.

To be compatible with multiple LLVM versions, many projects rely on #if directives based on the LLVM_VERSION_MAJOR macro. This post explores the specific techniques used by ccls to ensure compatibility with LLVM versions 7 to 19. For the latest release (ccls 0.20241108), support for LLVM versions 7 to 9 has been discontinued.

Given the tight coupling between LLVM and Clang, the LLVM_VERSION_MAJOR macro can be used for both version detection. There's no need to check CLANG_VERSION_MAJOR.

2024-11-02

Tinkering with Neovim

After migrating from Vim to Emacs as my primary C++ editor in 2015, I switched from Vim to Neovim for miscellaneous non-C++ tasks as it is more convenient in a terminal. Customizing the editor with a language you are comfortable with is important. I found myself increasingly drawn to Neovim's terminal-based simplicity for various tasks. Recently, I've refined my Neovim setup to the point where I can confidently migrate my entire C++ workflow away from Emacs.

This post explores the key improvements I've made to achieve this transition. My focus is on code navigation.

2024-10-20

ccls and LSP Semantic Tokens

I've spent countless hours writing and reading C++ code. For many years, Emacs has been my primary editor, and I leverage ccls' (my C++ language server) rainbow semantic highlighting feature.

The feature relies on two custom notification messages $ccls/publishSemanticHighlight and $ccls/publishSkippedRanges. $ccls/publishSemanticHighlight provides a list of symbols, each with kind information (function, type, or variable) of itself and its semantic parent (e.g. a member function's parent is a class), storage duration, and a list of ranges.

struct CclsSemanticHighlightSymbol {
  int id = 0;
  SymbolKind parentKind;
  SymbolKind kind;
  uint8_t storage;
  std::vector<std::pair<int, int>> ranges;

  std::vector<lsRange> lsRanges; // Only used by vscode-ccls
};

struct CclsSemanticHighlight {
  DocumentUri uri;
  std::vector<CclsSemanticHighlightSymbol> symbols;
};

An editor can use consistent colors to highlight different occurrences of a symbol. Different colors can be assigned to different symbols.

Tobias Pisani created emacs-cquery (the predecessor to emacs-ccls) in Nov 2017. Despite not being a fan of Emacs Lisp, I added the rainbow semantic highlighting feature for my own use in early 2018. My setup also relied heavily on these two settings:

Bolding and underlining variables of static duration storage
Italicizing member functions and variables

1 2	(setq ccls-sem-highlight-method 'font-lock) (ccls-use-default-rainbow-sem-highlight)

Key symbol properties (member, static) were visually prominent in my Emacs environment.

My Emacs hacking days are a distant memory – beyond basic configuration tweaks, I haven't touched elisp code since 2018. As my Elisp skills faded, I increasingly turned to Neovim for various editing tasks. Naturally, I wanted to migrate my C++ development workflow to Neovim as well. However, a major hurdle emerged: Neovim lacked the beloved rainbow highlighting I enjoyed in Emacs.

Thankfully, Neovim supports "semantic tokens" from LSP 3.16, a standardized approach adopted by many editors.

I've made changes to ccls (available on a branch; PR) to support semantic tokens. This involves adapting the $ccls/publishSemanticHighlight code to additionally support textDocument/semanticTokens/full and textDocument/semanticTokens/range.

I utilize a few token modifiers (static, classScope, functionScope, namespaceScope) for highlighting:

vim.cmd([[
hi @lsp.mod.classScope.cpp gui=italic
hi @lsp.mod.static.cpp gui=bold
hi @lsp.typemod.variable.namespaceScope.cpp gui=bold,underline
]])

While this approach is a significant improvement over relying solely on nvim-treesitter, I'm still eager to implement rainbow semantic tokens. Although LSP semantic tokens don't directly distinguish symbols, we can create custom modifiers to achieve similar results.

tokenModifiers: {
  "declaration", "definition", "static", ...

  "id0", "id1", ... "id9",
}

In the user-provided initialization options, I set highlight.rainbow to 10.

ccls assigns the same modifier ID to tokens belonging to the same symbol, aiming for unique IDs for different symbols. While we only have a few predefined IDs (each linked to a specific color), there's a slight possibility of collisions. However, this is uncommon and generally acceptable.

For a token with type variable, Neovim's built-in LSP plugin assigns a highlight group @lsp.typemod.variable.id$i.cpp where $i is an integer between 0 and 9. This allows us to customize a unique foreground color for each modifier ID.

local func_colors = {
  '#e5b124', '#927754', '#eb992c', '#e2bf8f', '#d67c17',
  '#88651e', '#e4b953', '#a36526', '#b28927', '#d69855',
}
local type_colors = {
  '#e1afc3', '#d533bb', '#9b677f', '#e350b6', '#a04360',
  '#dd82bc', '#de3864', '#ad3f87', '#dd7a90', '#e0438a',
}
local param_colors = {
  '#e5b124', '#927754', '#eb992c', '#e2bf8f', '#d67c17',
  '#88651e', '#e4b953', '#a36526', '#b28927', '#d69855',
}
local var_colors = {
  '#429921', '#58c1a4', '#5ec648', '#36815b', '#83c65d',
  '#419b2f', '#43cc71', '#7eb769', '#58bf89', '#3e9f4a',
}
local all_colors = {
  class = type_colors,
  constructor = func_colors,
  enum = type_colors,
  enumMember = var_colors,
  field = var_colors,
  ['function'] = func_colors,
  method = func_colors,
  parameter = param_colors,
  struct = type_colors,
  typeAlias = type_colors,
  typeParameter = type_colors,
  variable = var_colors
}
for type, colors in pairs(all_colors) do
  for i = 1,#colors do
    for _, lang in pairs({'c', 'cpp'}) do
      vim.api.nvim_set_hl(0, string.format('@lsp.typemod.%s.id%s.%s', type, i-1, lang), {fg=colors[i]})
    end
  end
end

vim.cmd([[
hi @lsp.mod.classScope.cpp gui=italic
hi @lsp.mod.static.cpp gui=bold
hi @lsp.typemod.variable.namespaceScope.cpp gui=bold,underline
]])

Now, let's analyze the C++ code above using this configuration.

While the results are visually pleasing, I need help implementing code lens functionality.

Inactive code highlighting

Inactive code regions (skipped ranges in Clang) are typically displayed in grey. While this can be helpful for identifying unused code, it can sometimes hinder understanding the details. I simply disabled the inactive code feature.

#ifdef X
... // colorful
#else
... // normal instead of grey
#endif

Refresh

When opening a large project, the initial indexing or cache loading process can be time-consuming, often leading to empty lists of semantic tokens for the initially opened files. While ccls prioritizes indexing these files, it's unclear how to notify the client to refresh the files. The existing workspace/semanticTokens/refresh request, unfortunately, doesn't accept text document parameters.

In contrast, with $ccls/publishSemanticHighlight, ccls proactively sends the notification after an index update (see main_OnIndexed).

void main_OnIndexed(DB *db, WorkingFiles *wfiles, IndexUpdate *update) {
  ...

  db->applyIndexUpdate(update);

  // Update indexed content, skipped ranges, and semantic highlighting.
  if (update->files_def_update) {
    auto &def_u = *update->files_def_update;
    if (WorkingFile *wfile = wfiles->getFile(def_u.first.path)) {
      wfile->setIndexContent(g_config->index.onChange ? wfile->buffer_content
                                                      : def_u.second);
      QueryFile &file = db->files[update->file_id];
      // Publish notifications to the file.
      emitSkippedRanges(wfile, file);
      emitSemanticHighlight(db, wfile, file);
      // But how do we send a workspace/semanticTokens/refresh request?????
    }
  }
}

While the semantic token request supports partial results in the specification, Neovim lacks this implementation. Even if it were, I believe a notification message with a text document parameter would be a more efficient and direct approach.

export interface SemanticTokensParams extends WorkDoneProgressParams,
	PartialResultParams {
	/**
	 * The text document.
	 */
	textDocument: TextDocumentIdentifier;
}

Other clients

emacs-ccls

Once this feature branch is merged, Emacs users can simply remove the following lines:

1 2	(setq ccls-sem-highlight-method 'font-lock) (ccls-use-default-rainbow-sem-highlight)

How to change lsp-semantic-token-modifier-faces to support rainbow semantic tokens in lsp-mode and emacs-ccls?

The general approach is similar to the following, but we need a feature from lsp-mode (https://github.com/emacs-lsp/lsp-mode/issues/4590).

(setq lsp-semantic-tokens-enable t)
(defface lsp-face-semhl-namespace-scope
         '((t :weight bold)) "highlight for namespace scope symbols" :group 'lsp-semantic-tokens)
(cl-loop for color in '("#429921" "#58c1a4" "#5ec648" "#36815b" "#83c65d"
                        "#417b2f" "#43cc71" "#7eb769" "#58bf89" "#3e9f4a")
       for i = 0 then (1+ i)
       do (custom-declare-face (intern (format "lsp-face-semhl-id%d" i))
                               `((t :foreground ,color))
                               "" :group 'lsp-semantic-tokens))
(setq lsp-semantic-token-modifier-faces
      `(("declaration" . lsp-face-semhl-interface)
        ("definition" . lsp-face-semhl-definition)
        ("implementation" . lsp-face-semhl-implementation)
        ("readonly" . lsp-face-semhl-constant)
        ("static" . lsp-face-semhl-static)
        ("deprecated" . lsp-face-semhl-deprecated)
        ("abstract" . lsp-face-semhl-keyword)
        ("async" . lsp-face-semhl-macro)
        ("modification" . lsp-face-semhl-operator)
        ("documentation" . lsp-face-semhl-comment)
        ("defaultLibrary" . lsp-face-semhl-default-library)
        ("classScope" . lsp-face-semhl-member)
        ("namespaceScope" . lsp-face-semhl-namespace-scope)
        ,@(cl-loop for i from 0 to 10
                   collect (cons (format "id%d" i)
                                 (intern (format "lsp-face-semhl-id%d" i))))
        ))

vscode-ccls

We require assistance to eliminate the $ccls/publishSemanticHighlight feature and adopt built-in semantic tokens support. Due to the lack of active maintenance for vscode-ccls, I'm unable to maintain this plugin for an editor I don't frequently use.

Misc

I use a trick to switch ccls builds without changing editor configurations.

#!/bin/zsh
#export CCLS_TRACEME=s
export LD_PRELOAD=/usr/lib/libmimalloc.so

type=
[[ -f /tmp/ccls-build ]] && type=$(</tmp/ccls-build)

case $type in
  strace)
    exec strace -s999 -e read,write -o /tmp/strace.log -f ~/ccls/out/debug/ccls --log-file=/tmp/cc.log -v=1 "$@";;
  debug)
    exec ~/ccls/out/debug/ccls --log-file=/tmp/cc.log -v=2 "$@";;
  release)
    exec ~/ccls/out/release/ccls --log-file=/tmp/cc.log -v=1 "$@";;
  *)
    exec /usr/bin/ccls --log-file=/tmp/cc.log -v=1 "$@";;
esac

Usage:

1 2	echo debug > /tmp/ccls-build nvim # out/debug/ccls is now used

Understanding -ftime-report output

Blogging

GDB

Inactive code highlighting

Refresh

Other clients

Misc

Understanding `-ftime-report` output