Update project to require Clang 16 or newer. (#3649)

This should also unblock our switch to C++20 and other improvements.

There are three core parts of the change --

1) Updating our infrastructure to fetch and find Clang-16.
2) Updating our documentation to reflect this and help folks with any
   system issues they encounter.

The infrastructure change is unfortunately tricky. We can't get Clang 16
easily on GitHub's runner images, and in the past we've had persistent
problems with flakiness when our actions download this much during their
runs. Due to the flakiness, we've previously removed all downloading of
dependencies outside of Bazel itself, and added retry loops around Bazel
specifically to overcome flaky downloads.

This change tries to address these problems by populating the Clang and
LLVM toolchain in a place that we can then cache using the built-in
GitHub action caching infrastructure. This seems like by far the least
likely to flake way of downloading extra things into our runs. And since
these are relatively slow moving dependencies, we should populate this
cache very, very rarely.

For Linux, this downloads the binary release artifact from GitHub,
prunes out large parts of it that we don't need, and then caches this as
a local toolchain. This proves both small and fast.

For macOS, this uses a trick to cache the destination of Homebrew
installs. It unfortunately caches the *entire* Homebrew installation
though, and so it also goes to some lengths to prune and minimize how
much is installed from Homebrew. The result is "only" a 2gb cache image.
Because of the size and slower download and filesystem, the macOS runs
see a 1 - 2 minute slowdown.

We might extend the Linux infrastructure here usefully if we want to
test multiple LLVM versions. We might also extend the macOS version to
get a cheaper way to prune parts of the system and free up disk space,
or to cache other Homebrew installed tools if needed.

Last but not least, this brought to the forefront an issue with our C++
toolchain integration which relied on a specific CMake build option
being set in the LLVM toolchain install. This option isn't used in the
official release artifacts. Instead, switch to a more robust approach to
linking libc++abi statically that shouldn't have these problems.

Beyond the infrastructure changes, this also updates the documentation
to reflect requiring Clang 16 or newer, and adds some extra tips for
folks that are missing this.

The documentation is also updated to address a problem with getting the
right libc++abi files installed to support the more robust linking
strategy. This may reduce the problems we've seen in the past around
libc++abi and linking on other Linux distros as well.

---------

Co-authored-by: josh11b <josh11b@users.noreply.github.com>

.github/workflows/tests.yaml

@@ -6,7 +6,7 @@ name: test
 
 on:
   push:
-    branches: [trunk]
+    branches: [trunk, action-test]
   pull_request:
   merge_group:
 
@@ -76,12 +76,58 @@ jobs:
           # Match the min version listed in docs/project/contribution_tools.md
           python-version: '3.9'
 
-      # Use LLVM following:
+      # Install and cache LLVM 16 from Homebrew.
+      # TODO: We can potentially remove this and simplify things when the
+      # Homebrew version of LLVM updates to 16 here:
       # https://github.com/actions/runner-images/blob/main/images/macos/macos-12-Readme.md
+      - name: Cache Homebrew (macOS)
+        if: steps.filter.outputs.has_code == 'true' && matrix.os == 'macos-12'
+        id: cache-homebrew-macos
+        uses: actions/cache@v3
+        env:
+          cache-name: cache-homebrew
+        with:
+          # Cover all the critical parts of Homebrew here.
+          path: |
+            /usr/local/Homebrew
+            /usr/local/Cellar
+            /usr/local/Frameworks
+            /usr/local/bin
+            /usr/local/opt
+          # Note the key needs to include all the packages we're adding.
+          key: Homebrew-Cache-${{ runner.os }}-${{ runner.arch }}
+
+      - name: Install LLVM and Clang with Homebrew (macOS)
+        if:
+          steps.filter.outputs.has_code == 'true' && matrix.os == 'macos-12' &&
+          steps.cache-homebrew-macos.outputs.cache-hit != 'true'
+        run: |
+          echo '*** Prune brew leaves'
+          # We prune all the leaf packages to have a minimal environment. This
+          # both minimizes the install space and avoids accidental dependencies
+          # on installed packages.
+          brew leaves
+          LEAVES=$(brew leaves | egrep -v '^(bazelisk|gh|git|git-lfs|gnu-tar|go@.*|jq|pipx|node@.*|openssl@.*|wget|yq|zlib)$')
+          brew uninstall -f --ignore-dependencies $LEAVES
+          echo '*** Installing LLVM deps'
+          brew install --force-bottle --only-dependencies llvm@16
+          echo '*** Installing LLVM itself'
+          brew install --force-bottle --force --verbose llvm@16
+          echo '*** brew info llvm@16'
+          brew info llvm@16
+          echo '*** brew autoremove'
+          brew autoremove
+          echo '*** brew info'
+          brew info
+          echo '*** brew leaves'
+          brew leaves
+          echo '*** brew config'
+          brew config
+
       - name: Setup LLVM and Clang (macOS)
         if: steps.filter.outputs.has_code == 'true' && matrix.os == 'macos-12'
         run: |
-          LLVM_PATH="$(brew --prefix llvm@15)"
+          LLVM_PATH="$(brew --prefix llvm@16)"
           echo "Using ${LLVM_PATH}"
           echo "${LLVM_PATH}/bin" >> $GITHUB_PATH
           echo '*** ls "${LLVM_PATH}"'
@@ -89,18 +135,49 @@ jobs:
           echo '*** ls "${LLVM_PATH}/bin"'
           ls "${LLVM_PATH}/bin"
 
-      # Use LLVM following:
-      # https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2204-Readme.md
-      - name: Setup LLVM and Clang (Ubuntu)
+      # Cache and install a recent version of LLVM. This uses the GitHub action
+      # cache to avoid directly downloading on each iteration and improve
+      # reliability.
+      - name: Cache LLVM and Clang installation (Ubuntu)
+        if:
+          steps.filter.outputs.has_code == 'true' && matrix.os == 'ubuntu-22.04'
+        id: cache-llvm-ubuntu
+        uses: actions/cache@v3
+        env:
+          cache-name: cache-llvm
+        with:
+          path: ~/llvm
+          key: LLVM-16-Cache-${{ runner.os }}-${{ runner.arch }}
+
+      - name: Download LLVM and Clang installation (Ubuntu)
+        if:
+          steps.filter.outputs.has_code == 'true' && matrix.os == 'ubuntu-22.04'
+          && steps.cache-llvm-ubuntu.outputs.cache-hit != 'true'
+        run: |
+          cd ~
+          LLVM_RELEASE=clang+llvm-16.0.4-x86_64-linux-gnu-ubuntu-22.04
+          echo "*** Downloading $LLVM_RELEASE"
+          wget --show-progress=off "https://github.com/llvm/llvm-project/releases/download/llvmorg-16.0.4/$LLVM_RELEASE.tar.xz"
+          echo "*** Extracting $LLVM_RELEASE"
+          tar -xJf "$LLVM_RELEASE.tar.xz"
+          echo "*** Moving to 'llvm'"
+          mv "$LLVM_RELEASE" llvm
+          echo "*** Testing `clang++ --version`"
+          ~/llvm/bin/clang++ --version
+
+          # The installation contains *huge* parts of LLVM we don't need for the
+          # toolchain. Prune them here to keep our cache small.
+          echo "*** Cleaning the 'llvm' directory"
+          rm llvm/lib/{*.a,*.so,*.so.*,*.bc}
+          rm llvm/bin/{flang-*,mlir-*,clang-{scan-deps,check,repl},*-test,llvm-{lto*,reduce,bolt*,exegesis,jitlink},bugpoint,opt,llc}
+          echo "*** Size of the 'llvm' directory"
+          du -hs llvm
+
+      - name: Setup LLVM and Clang paths (Ubuntu)
         if:
           steps.filter.outputs.has_code == 'true' && matrix.os == 'ubuntu-22.04'
         run: |
-          # TODO: Re-enable once llvm-15 is working.
-          # https://github.com/actions/runner-images/issues/8253
-          # LLVM_PATH="/usr/lib/llvm-15"
-          # if [[ ! -e "${LLVM_PATH}" ]]; then
-          LLVM_PATH="/usr/lib/llvm-14"
-          # fi
+          LLVM_PATH=~/llvm
           echo "Using ${LLVM_PATH}"
           echo "${LLVM_PATH}/bin" >> $GITHUB_PATH
           echo '*** ls "${LLVM_PATH}"'

MODULE.bazel.lock

@@ -1658,7 +1658,7 @@
     },
     "//bazel/cc_toolchains:clang_configuration.bzl%clang_toolchain_extension": {
       "general": {
-        "bzlTransitiveDigest": "Kc8IiGDwk4jswZNBnPRlrEMAB8XGoLRpc9IHkKq/DQI=",
+        "bzlTransitiveDigest": "K2JE5G8tvZ+UBmAPF5s/YSUNg53pTvAZi2ause87buQ=",
         "accumulatedFileDigests": {},
         "envVariables": {},
         "generatedRepoSpecs": {

bazel/cc_toolchains/clang_cc_toolchain_config.bzl

@@ -541,14 +541,11 @@ def _impl(ctx):
         )],
     )
 
-    # With clang 14 and lower, we expect it to be built with libc++ debug
-    # support. In later LLVM versions, we expect the assertions define to work.
-    # clang 17 deprecates LIBCPP_ENABLE_ASSERTIONS in favor of HARDENED_MODE.
-    if clang_version and clang_version <= 14:
-        libcpp_debug_flags = ["-D_LIBCPP_DEBUG=1"]
-    elif clang_version and clang_version <= 16:
+    if clang_version and clang_version <= 16:
         libcpp_debug_flags = ["-D_LIBCPP_ENABLE_ASSERTIONS=1"]
     else:
+        # Clang 17 deprecates LIBCPP_ENABLE_ASSERTIONS in favor of
+        # HARDENED_MODE.
         libcpp_debug_flags = ["-D_LIBCPP_ENABLE_HARDENED_MODE=1"]
 
     linux_flags_feature = feature(
@@ -566,14 +563,8 @@ def _impl(ctx):
                             # Force the C++ standard library and runtime
                             # libraries to be statically linked. This works even
                             # with libc++ and libunwind despite the names,
-                            # provided libc++ is built with two CMake options:
+                            # provided libc++ is built with the CMake option:
                             # - `-DCMAKE_POSITION_INDEPENDENT_CODE=ON`
-                            # - `-DLIBCXX_STATICALLY_LINK_ABI_IN_STATIC_LIBRARY`
-                            # These are both required because of PR43604
-                            # (impacting at least Debian packages of libc++) and
-                            # PR46321 (impacting most other packages).
-                            # We recommend using Homebrew's LLVM install on
-                            # Linux.
                             "-static-libstdc++",
                             "-static-libgcc",
                             # Link with Clang's runtime library. This is always
@@ -586,6 +577,11 @@ def _impl(ctx):
                             "-L" + llvm_bindir + "/../lib",
                             # Link with pthread.
                             "-lpthread",
+                            # Force linking the static libc++abi archive here.
+                            # This *should* be linked automatically, but not
+                            # every release of LLVM correctly sets the CMake
+                            # flags to do so.
+                            "-l:libc++abi.a",
                         ],
                     ),
                 ]),

bazel/cc_toolchains/clang_configuration.bzl

@@ -20,7 +20,7 @@ def _run(repository_ctx, cmd):
 def _clang_version(version_output):
     """Returns version information, or a (None, "unknown") tuple if not found.
 
-    Returns both the major version number (14) and the full version number for
+    Returns both the major version number (16) and the full version number for
     caching.
     """
     clang_version = None
@@ -39,7 +39,7 @@ def _clang_version(version_output):
         return (clang_version, clang_version_for_cache)
     clang_version_for_cache = version_output[version_start:version_newline]
 
-    # Find a dot to indicate something like 'clang version 14.0.6', and grab the
+    # Find a dot to indicate something like 'clang version 16.0.1', and grab the
     # major version.
     version_dot = version_output.find(".", version_start)
     if version_dot != -1 and version_dot < version_newline:

docs/project/contribution_tools.md

@@ -42,12 +42,17 @@ These commands should help set up a development environment on your machine.
 # Update apt.
 sudo apt update
 
+# Check that the `clang` version is at least 16, our minimum version. That needs
+# the number of the `:` in the output to be over 16. For example, `1:16.0-57`.
+apt-cache show clang | grep 'Version:'
+
 # Install tools.
 sudo apt install \
   bazel \
   clang \
   gh \
   libc++-dev \
+  libc++abi-dev \
   lld \
   python3 \
   pipx
@@ -65,7 +70,24 @@ pre-commit install
 bazel test //...:all
 ```
 
-> NOTE: Most LLVM 14+ installs should build Carbon. If you're having issues, see
+If the version of `clang` is earlier than 16, you may still have version 16
+available. You can use the following install instead:
+
+```shell
+# Install explicitly versioned Clang tools.
+sudo apt install \
+  clang-16 \
+  libc++-16-dev \
+  libc++abi-16-dev \
+  lld-16
+
+# In your Carbon checkout, tell Bazel where to find `clang`. You can also
+# export this path as the `CC` environment variable, or add it directly to
+# your `PATH`.
+echo "build --repo_env=CC=$(readlink -f $(which clang-16))" >> user.bazelrc
+```
+
+> NOTE: Most LLVM 16+ installs should build Carbon. If you're having issues, see
 > [troubleshooting build issues](#troubleshooting-build-issues).
 
 > NOTE: If you don't have a `bazel` package, see
@@ -217,8 +239,8 @@ work reliably include:
 
 Many build issues result from the particular options `clang` and `llvm` have
 been built with, particularly when it comes to system-installed versions. If you
-run `clang --version`, you should see at least version 14. If you see an older
-version, please update.
+run `clang --version`, you should see at least version 16. If you see an older
+version, please update, or use the special `clang-16` instructions above.
 
 System installs of macOS typically won't work, for example being an old LLVM
 version or missing llvm-ar; [setup commands](#setup-commands) includes LLVM from
@@ -236,6 +258,7 @@ providing the output of the following diagnostic commands:
 ```shell
 echo $CC
 which clang
+which clang-16
 clang --version
 grep llvm_bindir $(bazel info workspace)/bazel-execroot/external/bazel_cc_toolchain/clang_detected_variables.bzl