Remove the LLVM bootstrap and use Hombrew installed LLVM-12. (#551)

Now that LLVM 12 has been released we no longer have any need to
bootstrap LLVM to get the desired featureset. LLVM 12 is available
widely, including in Homebrew across multiple platforms and in the
GitHub action runners.

Sadly, the Linux distribution builds of LLVM-12 are largely broken and
not as useful for us. The Homebrew Linux install was also broken
originally, but I've worked extensively with the Homebrew folks to get
the Linux install into a really good shape. It should now work reliably.

There are two primary bugs in Linux LLVM packages that need to be fixed
before we can just use them:

- https://bugs.llvm.org/show_bug.cgi?id=43604
- https://bugs.llvm.org/show_bug.cgi?id=46321

Once those are addressed and point releases with the fixes widely
available we can further simplify things.

Even with the need to use Homebrew installs, using the released LLVM has
the extra advantage of making it easy to properly support Darwin ARM and
I've added that configuration so that I can test things there.

Last but not least, this will significantly shrink our build outputs
which should allow building much more in continuous integration on
GitHub actions without exceeding the action cache size limits. I've even
added several tweaks and adjustments to the compile and build flags to
improve the build performance and reduce the build output size.

Once this is landed and stable, we can consider adding the refactoring
tooling back to our CI.

One of the biggest downsides of this path is that our CI has to download
and install the LLVM toolchain from Homebrew on each run. This is pretty
slow (takes a couple of minutes). But it is a fixed overhead -- it won't
get worse over time. Eventually, we can either look at a much fancier
action configuration to avoid this or hopefully the Debian packages will
get updated and we can move back to those.

The bootstrapping has served us long enough at this point. We can
resurrect it if we ever find a compelling reason for breaking off of the
latest LLVM release as our host toolchain.

Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>

.bazelrc

@@ -16,13 +16,18 @@ build --host_crosstool_top=@bazel_cc_toolchain
 # https://github.com/bazelbuild/bazel/issues/13315
 build --incompatible_dont_enable_host_nonhost_crosstool_features=false
 
+# Default dynamic linking to off. While this can help build performance in some
+# edge cases with very large linked executables and a slow linker, between using
+# fast linkers on all platforms (LLD and the Apple linker), as well as having
+# relatively few such executables, shared objects simply waste too much space in
+# our builds.
+build --dynamic_mode=off
+
 # Completely disable Bazel's automatic stripping of debug information. Removing
 # that information causes unhelpful backtraces from unittest failures and other
 # crashes. Optimized builds already avoid using debug information by default.
 build --strip=never
 
-build:force_local_bootstrap --repo_env=CARBON_FORCE_LOCAL_BOOTSTRAP_BUILD=1
-
 # Configuration for enabling Address Sanitizer. Note that this is enabled by
 # default for fastbuild. The config is provided to enable ASan even in
 # optimized or other build configurations.

.github/workflows/tests.yaml

@@ -58,11 +58,47 @@ jobs:
           go get github.com/bazelbuild/bazelisk
           echo "$(go env GOPATH)/bin" >> $GITHUB_PATH
 
-      # Print the various tool versions to help in debugging.
-      - name: Print tool versions
+      # Setup to the latest LLVM and Clang release on Linux runners.
+      #
+      # Ideally we would use the pre-installed versions in the image, but the
+      # debian packages for LLVM-12 are broken due to several bugs:
+      # https://bugs.llvm.org/show_bug.cgi?id=43604
+      # https://bugs.llvm.org/show_bug.cgi?id=46321
+      #
+      # For now, we rely on Homebrew to manage installing a correctly built
+      # toolchain. We also take some care to be as resilient as possible to
+      # issues fetching and installing the toolchain.
+      - name: Setup LLVM and Clang on Ubuntu
+        if: matrix.os == 'ubuntu-latest'
+        run: |
+          brew update
+          brew install --force-bottle --only-dependencies llvm
+          brew install --force-bottle --force --verbose llvm
+          brew info llvm
+          brew config
+          echo "$(brew --prefix llvm)/bin" >> $GITHUB_PATH
+
+      # Just add the Homebrew installed LLVM to the path on macOS, the image has
+      # LLVM-12 pre-installed.
+      - name: Setup LLVM and Clang on macOS
+        if: matrix.os == 'macos-latest'
+        run: |
+          echo "$(brew --prefix llvm)/bin" >> $GITHUB_PATH
+
+      # Print the various tool paths and versions to help in debugging.
+      - name: Print tool debugging info
         run: |
+          echo $PATH
+          which bazelisk
           bazelisk --version
+          which python
           python --version
+          which clang
+          clang --version
+          which clang++
+          clang++ --version
+          which clang-format
+          clang-format --version
 
       # Make the date available to subsequent steps.
       - name: Get date

WORKSPACE

@@ -21,18 +21,10 @@ pip_install(
     requirements = "//github_tools:requirements.txt",
 )
 
-# Bootstrap a Clang and LLVM toolchain.
-load("//bazel/cc_toolchains:clang_bootstrap.bzl", "bootstrap_clang_toolchain")
-
-bootstrap_clang_toolchain(name = "bootstrap_clang_toolchain")
-
 # Configure the bootstrapped Clang and LLVM toolchain for Bazel.
 load("//bazel/cc_toolchains:clang_configuration.bzl", "configure_clang_toolchain")
 
-configure_clang_toolchain(
-    name = "bazel_cc_toolchain",
-    clang = "@bootstrap_clang_toolchain//:bin/clang",
-)
+configure_clang_toolchain(name = "bazel_cc_toolchain")
 
 local_repository(
     name = "llvm_bazel",

bazel-clang-toolchain

@@ -1 +0,0 @@
-bazel-out/../../../external/bootstrap_clang_toolchain

bazel/cc_toolchains/clang_bootstrap.bzl

@@ -1,280 +0,0 @@
-# Part of the Carbon Language project, under the Apache License v2.0 with LLVM
-# Exceptions. See /LICENSE for license information.
-# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-
-"""Starlark rules to bootstrap Clang (and LLVM).
-
-These rules are loaded as part of the `WORKSPACE`, and used by
-`clang_configuration.bzl`. The llvm-project submodule is used for the build.
-"""
-
-FORCE_LOCAL_BOOTSTRAP_ENV = "CARBON_FORCE_LOCAL_BOOTSTRAP_BUILD"
-
-def _run(
-        repository_ctx,
-        cmd,
-        timeout = 10,
-        environment = {},
-        quiet = True):
-    """Runs the provided `cmd`, checks for failure, and returns the result."""
-    exec_result = repository_ctx.execute(
-        cmd,
-        timeout = timeout,
-        environment = environment,
-        quiet = quiet,
-    )
-    if exec_result.return_code != 0:
-        fail("Unable to run command successfully: %s" % str(cmd))
-
-    return exec_result
-
-def _detect_system_clang(repository_ctx):
-    """Detects whether a system-provided clang can be used.
-
-    Returns a tuple of (is_clang, environment).
-    """
-
-    # If the user provides an explicit `CC` environment variable, use that as
-    # the compiler.
-    cc = repository_ctx.os.environ.get("CC")
-    cxx = repository_ctx.os.environ.get("CXX")
-    if cc or cxx:
-        version_output = _run(repository_ctx, [cc, "--version"]).stdout
-        return "clang" in version_output, {}
-
-    # If we can build our Clang toolchain using a system-installed Clang, try
-    # to do so.
-    system_clang = repository_ctx.which("clang")
-    if system_clang:
-        return True, {
-            "CC": str(system_clang),
-            "CXX": str(system_clang) + "++",
-        }
-    return False, {}
-
-def _get_cmake_defines(repository_ctx, is_clang):
-    """Returns a long list of cmake defines for the bootstrap."""
-    modules_setting = "OFF"
-    if is_clang:
-        modules_setting = "ON"
-
-    static_link_cxx = "ON"
-    unstable_libcxx_abi = "ON"
-    if repository_ctx.os.name.lower().startswith("mac os"):
-        # macOS doesn't support the static C++ standard library linking. Turn
-        # it off here, and disable the unstable libc++ ABI as we will also be
-        # unable to use it later on.
-        static_link_cxx = "OFF"
-        unstable_libcxx_abi = "OFF"
-
-    return [
-        "-DLLVM_ENABLE_PROJECTS=clang;clang-tools-extra;lld;libcxx;libcxxabi;compiler-rt;libunwind",
-        "-DCMAKE_BUILD_TYPE=Release",
-        "-DLLVM_ENABLE_ASSERTIONS=OFF",
-        "-DLLVM_ENABLE_MODULES=" + modules_setting,
-        "-DLLVM_STATIC_LINK_CXX_STDLIB=" + static_link_cxx,
-        "-DLLVM_TARGETS_TO_BUILD=AArch64;X86",
-        "-DLIBCXX_ABI_UNSTABLE=" + unstable_libcxx_abi,
-        "-DLIBCXX_ENABLE_ASSERTIONS=OFF",
-        "-DLIBCXXABI_ENABLE_ASSERTIONS=OFF",
-
-        # Disable components of the build that we don't use while building Carbon.
-        "-DCLANG_ENABLE_ARCMT=OFF",
-        "-DCLANG_INCLUDE_TESTS=OFF",
-        "-DCLANG_TOOL_APINOTES_TEST_BUILD=OFF",
-        "-DCLANG_TOOL_ARCMT_TEST_BUILD=OFF",
-        "-DCLANG_TOOL_CLANG_CHECK_BUILD=OFF",
-        "-DCLANG_TOOL_CLANG_DIFF_BUILD=OFF",
-        "-DCLANG_TOOL_CLANG_EXTDEF_MAPPING_BUILD=OFF",
-        "-DCLANG_TOOL_CLANG_FUZZER_BUILD=OFF",
-        "-DCLANG_TOOL_CLANG_IMPORT_TEST_BUILD=OFF",
-        "-DCLANG_TOOL_CLANG_OFFLOAD_BUNDLER_BUILD=OFF",
-        "-DCLANG_TOOL_CLANG_OFFLOAD_WRAPPER_BUILD=OFF",
-        "-DCLANG_TOOL_CLANG_SCAN_DEPS_BUILD=OFF",
-        "-DCLANG_TOOL_CLANG_SHLIB_BUILD=OFF",
-        "-DCLANG_TOOL_C_ARCMT_TEST_BUILD=OFF",
-        "-DCLANG_TOOL_C_INDEX_TEST_BUILD=OFF",
-        "-DCLANG_TOOL_DIAGTOOL_BUILD=OFF",
-        "-DCLANG_TOOL_LIBCLANG_BUILD=OFF",
-        "-DCLANG_TOOL_SCAN_BUILD_BUILD=OFF",
-        "-DCLANG_TOOL_SCAN_VIEW_BUILD=OFF",
-        "-DLLVM_BUILD_UTILS=OFF",
-        "-DLLVM_ENABLE_BINDINGS=OFF",
-        "-DLLVM_ENABLE_LIBXML2=OFF",
-        "-DLLVM_ENABLE_OCAMLDOC=OFF",
-        "-DLLVM_INCLUDE_BENCHMARKS=OFF",
-        "-DLLVM_INCLUDE_DOCS=OFF",
-        "-DLLVM_INCLUDE_EXAMPLES=OFF",
-        "-DLLVM_INCLUDE_GO_TESTS=OFF",
-        "-DLLVM_INCLUDE_TESTS=OFF",
-        "-DLLVM_INCLUDE_UTILS=OFF",
-        "-DLLVM_TOOL_BUGPOINT_BUILD=OFF",
-        "-DLLVM_TOOL_BUGPOINT_PASSES_BUILD=OFF",
-        "-DLLVM_TOOL_DSYMUTIL_BUILD=OFF",
-        "-DLLVM_TOOL_GOLD_BUILD=OFF",
-        "-DLLVM_TOOL_LLC_BUILD=OFF",
-        "-DLLVM_TOOL_LLI_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_AS_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_BCANALYZER_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_CAT_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_CFI_VERIFY_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_CONFIG_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_CVTRES_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_CXXDUMP_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_CXXFILT_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_CXXMAP_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_C_TEST_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_DIFF_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_DWARFDUMP_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_ELFABI_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_EXEGESIS_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_EXTRACT_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_GO_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_GSYMUTIL_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_IFS_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_ISEL_FUZZER_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_ITANIUM_DEMANGLE_FUZZER_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_JITLINK_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_JITLISTENER_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_LIBTOOL_DARWIN_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_LINK_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_LIPO_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_LTO2_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_LTO_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_MCA_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_MC_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_ML_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_MICROSOFT_DEMANGLE_FUZZER_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_MT_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_OPT_FUZZER_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_PDBUTIL_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_PROFDATA_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_PROFGEN_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_RC_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_READOBJ_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_REDUCE_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_RTDYLD_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_SHLIB_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_SIZE_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_SPECIAL_CASE_LIST_FUZZER_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_SPLIT_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_STRESS_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_STRINGS_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_XRAY_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_YAML_NUMERIC_PARSER_FUZZER_BUILD=OFF",
-        "-DLLVM_TOOL_LLVM_YAML_PARSER_FUZZER_BUILD=OFF",
-        "-DLLVM_TOOL_LTO_BUILD=OFF",
-        "-DLLVM_TOOL_OBJ2YAML_BUILD=OFF",
-        "-DLLVM_TOOL_OPT_BUILD=OFF",
-        "-DLLVM_TOOL_OPT_VIEWER_BUILD=OFF",
-        "-DLLVM_TOOL_REMARKS_SHLIB_BUILD=OFF",
-        "-DLLVM_TOOL_SPLIT_FILE_BUILD=OFF",
-        "-DLLVM_TOOL_VERIFY_USELISTORDER_BUILD=OFF",
-        "-DLLVM_TOOL_YAML2OBJ_BUILD=OFF",
-    ]
-
-def _local_cmake_build_clang_toolchain(repository_ctx):
-    """Locally build the LLVM toolchain with CMake and Ninja.
-
-    This is used as a fallback for when we can't download a prebuilt set of
-    binaries and libraries for a particular platform.
-    """
-    repository_ctx.report_progress("Configuring Clang toolchain bootstrap...")
-    is_clang, environment = _detect_system_clang(repository_ctx)
-
-    cmake = repository_ctx.which("cmake")
-    if not cmake:
-        fail("`cmake` not found: is it installed?")
-    ninja = repository_ctx.which("ninja")
-    if not ninja:
-        fail("`ninja` not found: is it installed?")
-
-    workspace_dir = repository_ctx.path(repository_ctx.attr._workspace).dirname
-    llvm_dir = repository_ctx.path("%s/third_party/llvm-project/llvm" %
-                                   workspace_dir)
-
-    repository_ctx.report_progress(
-        "Running CMake for the Clang toolchain build...",
-    )
-    cmake_args = [cmake, "-G", "Ninja", str(llvm_dir)]
-    cmake_args += _get_cmake_defines(repository_ctx, is_clang)
-    _run(
-        repository_ctx,
-        cmake_args,
-        timeout = 600,
-        environment = environment,
-        # This is very slow, so print output as a form of progress.
-        quiet = False,
-    )
-
-    # Run ninja for the final build.
-    repository_ctx.report_progress("Building the Clang toolchain...")
-    _run(
-        repository_ctx,
-        [ninja],
-        timeout = 10800,
-        # This is very slow, so print output as a form of progress.
-        quiet = False,
-    )
-
-def _download_prebuilt_toolchain(repository_ctx):
-    """Downloads and extracts an LLVM build for the current platform.
-
-    Returns `True` when a toolchain can be successfully downloaded.
-    """
-    repository_ctx.report_progress("Checking for a downloadable toolchain...")
-    os = repository_ctx.os.name
-    if os == "linux":
-        url = "https://github.com/mmdriley/llvm-builds/releases/download/r32/llvm-linux.tar.xz"
-        sha256 = "db9f2698aa84935efca3402bdebada127de16f6746adbe54d4cdb7e3b8fec5f3"
-    elif os == "mac os x":
-        url = "https://github.com/mmdriley/llvm-builds/releases/download/r32/llvm-macos.tar.xz"
-        sha256 = "937b81c235977ed2b265baf656f30b7a03c33b6299090d91beb72c2b41846673"
-    elif os.startswith("windows"):
-        url = "https://github.com/mmdriley/llvm-builds/releases/download/r32/llvm-windows.tar.xz"
-        sha256 = "b6b015f9f2fcfb79381004e6a3ae925df4fb827cf7e07f3d5b0b66210fddd172"
-    else:
-        print(("No prebuilt LLVM toolcahin to download for {}, falling back " +
-               "to a local build. This may be very slow!").format(os))
-        return False
-
-    repository_ctx.report_progress("Downloading and extracting a toolchain...")
-    repository_ctx.download_and_extract(url, sha256 = sha256)
-    return True
-
-def _bootstrap_clang_toolchain_impl(repository_ctx):
-    """Bootstrap a fresh Clang and LLVM toolchain for use.
-
-    This will first try to download a pre-built archive of the LLVM toolchain
-    if one is available.
-
-    Otherwise will locally build the toolchain using CMake out of the LLVM
-    submodule.
-    """
-    force_local_build = False
-    if FORCE_LOCAL_BOOTSTRAP_ENV in repository_ctx.os.environ:
-        print("Forcing a local bootstrap build. This may be very slow!")
-        force_local_build = True
-
-    if force_local_build or not _download_prebuilt_toolchain(repository_ctx):
-        # Fallback to a local build.
-        _local_cmake_build_clang_toolchain(repository_ctx)
-
-    # Create a BUILD file to mark the package. The files are usually used
-    # without Bazel labels, but the export allows dependencies; for example, to
-    # use clang-tidy.
-    repository_ctx.file("BUILD", 'exports_files(glob(["bin/*"]))')
-
-bootstrap_clang_toolchain = repository_rule(
-    implementation = _bootstrap_clang_toolchain_impl,
-    configure = True,
-    attrs = {
-        # We use a label pointing at the workspace file to compute the
-        # workspace directory.
-        "_workspace": attr.label(
-            default = Label("//:WORKSPACE"),
-            allow_single_file = True,
-        ),
-    },
-    environ = ["CC", "CXX", FORCE_LOCAL_BOOTSTRAP_ENV],
-)

bazel/cc_toolchains/clang_cc_toolchain_config.bzl

@@ -272,6 +272,8 @@ def _impl(ctx):
             actions = codegen_compile_actions,
             flag_groups = [flag_group(flags = [
                 "-O1",
+                "-mllvm",
+                "-fast-isel",
             ])],
         )],
     )
@@ -444,10 +446,14 @@ def _impl(ctx):
                 # We don't need the recovery behavior of UBSan as we expect
                 # builds to be clean. Not recoverying is a bit cheaper.
                 "-fno-sanitize-recover=undefined",
+                # Don't embed the full path name for files. This limits the size
+                # and combined with line numbers is unlikely to result in many
+                # ambiguities.
+                "-fsanitize-undefined-strip-path-components=-1",
                 # Force some expensive UBSan checks to the cheaper trap mode.
                 # The dedicated debugging message is unlikely to be critical for
                 # these.
-                "-fsanitize-trap=alignment,null,return,unreachable",
+                "-fsanitize-trap=alignment,bool,null,return,unreachable",
                 # Needed due to clang AST issues, such as in
                 # clang/AST/Redeclarable.h line 199.
                 "-fno-sanitize=vptr",
@@ -484,17 +490,24 @@ def _impl(ctx):
                     flag_group(
                         flags = [
                             "-fuse-ld=lld",
-                            "-Wl,-no-as-needed",
-                            # Force the C++ standard library to be statically
-                            # linked. This works even with libc++ despite the
-                            # name, however we have to manually link the ABI
-                            # library and libunwind.
+                            "-stdlib=libc++",
+                            "-unwindlib=libunwind",
+                            # Force the C++ standard library and runtime
+                            # libraries to be statically linked. This works even
+                            # with libc++ and libunwind despite the names,
+                            # provided libc++ is built with two CMake options:
+                            # - `-DCMAKE_POSITION_INDEPENDENT_CODE=ON`
+                            # - `-DLIBCXX_STATICALLY_LINK_ABI_IN_STATIC_LIBRARY`
+                            # These are both required because of PR43604
+                            # (impacting at least Debian packages of libc++) and
+                            # PR46321 (impacting most other packages).
+                            # We recommend using Homebrew's LLVM install on
+                            # Linux.
                             "-static-libstdc++",
-                            # Force static linking with libc++abi as well.
-                            "-l:libc++abi.a",
+                            "-static-libgcc",
                             # Link with Clang's runtime library. This is always
                             # linked statically.
-                            #"-rtlib=compiler-rt",
+                            "-rtlib=compiler-rt",
                             # Explicitly add LLVM libs to the search path to
                             # preempt the detected GCC installation's library
                             # paths. Those might have a system installed libc++
@@ -734,10 +747,10 @@ def _impl(ctx):
 
     # Next, add the features based on the target platform. Here too the
     # features are order sensitive. We also setup the sysroot here.
-    if (ctx.attr.target_cpu == "k8"):
+    if ctx.attr.target_cpu == "k8":
         features += [linux_flags_feature]
         sysroot = None
-    elif (ctx.attr.target_cpu == "darwin"):
+    elif ctx.attr.target_cpu in ["darwin", "darwin_arm64"]:
         sysroot = sysroot_dir
     else:
         fail("Unsupported target platform!")

bazel/cc_toolchains/clang_configuration.bzl

@@ -17,6 +17,30 @@ def _run(repository_ctx, cmd):
 
     return exec_result
 
+def _detect_system_clang(repository_ctx):
+    """Detects whether the system-provided clang can be used.
+
+    Returns a tuple of (is_clang, environment).
+    """
+
+    # If the user provides an explicit `CC` environment variable, use that as
+    # the compiler. This should point at the `clang` executable to use.
+    cc = repository_ctx.os.environ.get("CC")
+    if cc:
+        version_output = _run(repository_ctx, [cc, "--version"]).stdout
+        if not "clang" in version_output:
+            fail("The `CC` environment variable is not a Clang compiler.")
+        return repository_ctx.path(cc)
+
+    # Try looking on the path. We check for the specific versioned name and then the normal name.
+    system_clang = repository_ctx.which("clang-12")
+    if not system_clang:
+        system_clang = repository_ctx.which("clang")
+        if not system_clang:
+            fail("Unable to find a `clang` executable on the system path.")
+
+    return system_clang
+
 def _compute_clang_resource_dir(repository_ctx, clang):
     """Runs the `clang` binary to get its resource dir."""
     output = _run(
@@ -88,15 +112,14 @@ def _configure_clang_toolchain_impl(repository_ctx):
         "cc_toolchain_config.bzl",
     )
 
-    # Run the bootstrapped clang to detect relevant features for the toolchain.
-    clang = repository_ctx.path(repository_ctx.attr.clang)
-    if clang.basename != "clang":
-        fail("The provided Clang binary must be `clang`, but is `%s` (%s)" %
-             (clang.basename, str(clang)))
+    # Find a Clang C++ compiler, and where it lives. We need to walk symlinks
+    # here as the other LLVM tools may not be symlinked into the PATH even if
+    # `clang` is. We also insist on finding the basename of `clang++` as that is
+    # important for C vs. C++ compiles.
+    clang = _detect_system_clang(repository_ctx)
+    clang = clang.realpath.dirname.get_child("clang++")
 
-    # Adjust this to the "clang++" binary to ensure we get the correct behavior
-    # when configuring it.
-    clang = repository_ctx.path(str(clang) + "++")
+    # Compute the various directories used by Clang.
     resource_dir = _compute_clang_resource_dir(repository_ctx, clang)
     sysroot_dir = None
     if repository_ctx.os.name.lower().startswith("mac os"):
@@ -124,6 +147,7 @@ def _configure_clang_toolchain_impl(repository_ctx):
 configure_clang_toolchain = repository_rule(
     implementation = _configure_clang_toolchain_impl,
     configure = True,
+    local = True,
     attrs = {
         "_clang_toolchain_build": attr.label(
             default = Label("//bazel/cc_toolchains:clang_toolchain.BUILD"),
@@ -141,11 +165,6 @@ configure_clang_toolchain = repository_rule(
             ),
             allow_single_file = True,
         ),
-        # This must point at the `clang` binary inside a full LLVM toolchain
-        # installation.
-        "clang": attr.label(
-            allow_single_file = True,
-            mandatory = True,
-        ),
     },
+    environ = ["CC"],
 )

bazel/cc_toolchains/clang_toolchain.BUILD

@@ -21,6 +21,7 @@ cc_toolchain_suite(
     name = "bazel_cc_toolchain",
     toolchains = {
         "darwin": ":cc-compiler-darwin",
+        "darwin_arm64": ":cc-compiler-darwin-arm64",
         "k8": ":cc-compiler-k8",
     },
 )
@@ -64,3 +65,23 @@ cc_toolchain_config(
     name = "local-darwin",
     target_cpu = "darwin",
 )
+
+cc_toolchain(
+    name = "cc-compiler-darwin-arm64",
+    all_files = ":empty",
+    ar_files = ":empty",
+    as_files = ":empty",
+    compiler_files = ":empty",
+    dwp_files = ":empty",
+    linker_files = ":empty",
+    objcopy_files = ":empty",
+    strip_files = ":empty",
+    supports_param_files = 1,
+    toolchain_config = ":local-darwin-arm64",
+    toolchain_identifier = "local-darwin-arm64",
+)
+
+cc_toolchain_config(
+    name = "local-darwin-arm64",
+    target_cpu = "darwin_arm64",
+)

compile_flags.txt

@@ -76,11 +76,7 @@ bazel-execroot/external/llvm-project/llvm/utils/unittest/googletest/include
 -isystem
 bazel-bin/external/llvm-project/llvm/utils/unittest/googletest/include
 -std=c++17
--nostdinc++
--isystem
-bazel-clang-toolchain/include/c++/v1
--isystem
-bazel-clang-toolchain/lib/clang/12.0.0/include
+-stdlib=libc++
 -no-canonical-prefixes
 -Wno-builtin-macro-redefined
 -D__DATE__="redacted"

docs/project/contribution_tools.md

@@ -25,7 +25,7 @@ contributions.
     -   [Bazel and Bazelisk](#bazel-and-bazelisk)
     -   [buildifier](#buildifier)
     -   [Clang and LLVM](#clang-and-llvm)
-    -   [Ninja](#ninja)
+        -   [Manual installations (not recommended)](#manual-installations-not-recommended)
     -   [pre-commit](#pre-commit)
 -   [Optional tools](#optional-tools)
     -   [Carbon-maintained](#carbon-maintained)
@@ -188,23 +188,52 @@ Our recommended way of installing is:
 ### Clang and LLVM
 
 [Clang](https://clang.llvm.org/) and [LLVM](https://llvm.org/) are used to
-compile and link Carbon, and are provided through git submodules. A complete
-toolchain will be built and cached as part of standard `bazel` execution. This
-can be very slow on less powerful computers or laptops (30 minutes to an hour).
-However, it should only happen when either Bazel or LLVM's submodule is updated,
-which we try to minimize. If you need to force a rebuild of the toolchain, you
-can use `bazel sync --configure`.
+compile and link Carbon as part of its build. Their source code are also
+provided through git submodules for incorporation into Carbon or Carbon tools as
+libraries. While the source submodule tracks upstream LLVM, the project expects
+the LLVM 12 release (or newer) to be installed with Clang and other tools in
+your `PATH` for use in building Carbon itself.
 
-### Ninja
+Our recommended way of installing is:
 
-[Ninja](https://ninja-build.org/) is used to build Clang and LLVM.
+```bash
+brew install llvm
+```
 
-Our recommended way of installing is:
+`llvm` is keg-only; bear in mind this requires updating `PATH` for it because
+it's not part of the standard Homebrew path. Read the output of `brew install`
+for the necessary path changes, or add something to your `PATH` like:
 
 ```bash
-brew install ninja
+export PATH="$(brew --prefix llvm)/bin:${PATH}"
+```
+
+Carbon expects the `PATH` to include the installed tooling. If set, `CC` should
+also point at `clang`. Our build environment will detect the `clang` binary
+using `CC` then `PATH`, and will expect the rest of the LLVM toolchain to be
+available in the same directory as `clang`. However, various scripts and tools
+assume that the LLVM toolchain will be in `PATH`, particularly for tools like
+`clang-format` and `clang-tidy`.
+
+> TODO: We'd like to use `apt`, but standard LLVM Debian packages are not
+> configured correctly for our needs. We are currently aware of two libc++
+> issues, [43604](https://bugs.llvm.org/show_bug.cgi?id=43604) and
+> [46321](https://bugs.llvm.org/show_bug.cgi?id=46321).
+
+#### Manual installations (not recommended)
+
+You can also build and install `LLVM` yourself if you prefer. The essential
+CMake options to pass in order for this to work reliably include:
+
+```
+-DLLVM_ENABLE_PROJECTS=clang;clang-tools-extra;lld
+-DLLVM_ENABLE_RUNTIMES=compiler-rt;libcxx;libcxxabi;libunwind
+-DRUNTIMES_CMAKE_ARGS=-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF;-DCMAKE_POSITION_INDEPENDENT_CODE=ON;-DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON;-DLIBCXX_STATICALLY_LINK_ABI_IN_SHARED_LIBRARY=OFF;-DLIBCXX_STATICALLY_LINK_ABI_IN_STATIC_LIBRARY=ON;-DLIBCXX_USE_COMPILER_RT=ON;-DLIBCXXABI_USE_COMPILER_RT=ON;-DLIBCXXABI_USE_LLVM_UNWINDER=ON
 ```
 
+However, we primarily test against the Homebrew installation, so if building
+LLVM and Clang yourself you may hit some issues.
+
 ### pre-commit
 
 We use [pre-commit](https://pre-commit.com) to run

migrate_cpp/BUILD

@@ -10,7 +10,6 @@ py_binary(
     data = [
         ":clang_tidy.yaml",
         "//migrate_cpp/cpp_refactoring",
-        "@bootstrap_clang_toolchain//:bin/clang-tidy",
     ],
     python_version = "PY3",
 )

migrate_cpp/migrate_cpp.py

@@ -12,7 +12,6 @@ import os
 import subprocess
 import sys
 
-_CLANG_TIDY = "../external/bootstrap_clang_toolchain/bin/clang-tidy"
 _CPP_REFACTORING = "./cpp_refactoring/cpp_refactoring"
 _H_EXTS = {".h", ".hpp"}
 _CPP_EXTS = {".c", ".cc", ".cpp", ".cxx"}
@@ -73,11 +72,10 @@ class _Workflow(object):
     def _clang_tidy(self):
         """Runs clang-tidy to fix C++ files in a directory."""
         self._print_header("Running clang-tidy...")
-        clang_tidy = self._data_file(_CLANG_TIDY)
         with open(self._data_file("clang_tidy.yaml")) as f:
             config = f.read()
         subprocess.run(
-            [clang_tidy, "--fix", "--config", config] + self._cpp_files
+            ["clang-tidy", "--fix", "--config", config] + self._cpp_files
         )
 
     def _cpp_refactoring(self):

scripts/create_compdb.py

@@ -57,7 +57,7 @@ except FileNotFoundError:
 
 # Prepend the `clang` executable path to the arguments that looks into our
 # downloaded Clang toolchain.
-arguments = [str(Path("bazel-clang-toolchain/bin/clang"))] + arguments
+arguments = ["clang++"] + arguments
 
 print("Building compilation database...")