Modify clang detection to build clang (#261)

Per chandlerc's comment:

First, this builds inside the Bazel tree rather than in the source
repository. This ensures we start in a clean directory each time the
repository rule is run again. We don't need to detect an existing build
with this, and it will only be rebuilt when the workspace file or the
repository rule implementation is changed. This can be forced by using:
```
bazel sync --configure
```

Second, we use an implicit dependency on the `WORKSPACE` file to locate
the workspace directory automatically, and the `HEAD` file from the
`llvm-project` submodule to trigger a rebuild if the submodule is
updated.

Third, teach the CMake script to try to use a system-installed `clang`
if installed and not overridden by the `CC` environment variable. This
is very different from the prior logic -- this is only used with the
CMake build, and so should work with any system C++ compiler that can
build Clang and LLVM.

Lastly, this tweaks the CMake options to tune this build given that we
now fully control it and it will only be used in this context. This
still results in a 1.5gb build for me. =/ But its as small as I can make
it really. It's a frustrating long list, but I couldn't find a more
brief way of representing this.

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>

.bazelversion

@@ -1 +1 @@
-last_green
+latest

WORKSPACE

@@ -26,12 +26,17 @@ load("@rules_foreign_cc//:workspace_definitions.bzl", "rules_foreign_cc_dependen
 
 rules_foreign_cc_dependencies()
 
-# Detect and configure a Clang and LLVM based toolchain.
-load("//bazel/cc_toolchains:clang_detection.bzl", "detect_clang_toolchain")
+# Bootstrap a Clang and LLVM toolchain.
+load("//bazel/cc_toolchains:clang_bootstrap.bzl", "bootstrap_clang_toolchain")
 
-detect_clang_toolchain(
+bootstrap_clang_toolchain(name = "bootstrap_clang_toolchain")
+
+# Configure the bootstrapped Clang and LLVM toolchain for Bazel.
+load("//bazel/cc_toolchains:clang_configuration.bzl", "configure_clang_toolchain")
+
+configure_clang_toolchain(
     name = "bazel_cc_toolchain",
-    workspace_dir = __workspace_dir__,
+    clang = "@bootstrap_clang_toolchain//:bin/clang",
 )
 
 local_repository(

bazel/cc_toolchains/clang_bootstrap.bzl

@@ -0,0 +1,234 @@
+# Part of the Carbon Language project, under the Apache License v2.0 with LLVM
+# Exceptions. See /LICENSE for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Starlark rules to bootstrap Clang (and LLVM).
+
+These rules are loaded as part of the `WORKSPACE`, and used by
+`clang_configuration.bzl`. The llvm-project submodule is used for the build.
+"""
+
+def _run(
+        repository_ctx,
+        cmd,
+        timeout = 10,
+        environment = {},
+        quiet = True):
+    """Runs the provided `cmd`, checks for failure, and returns the result."""
+    exec_result = repository_ctx.execute(
+        cmd,
+        timeout = timeout,
+        environment = environment,
+        quiet = quiet,
+    )
+    if exec_result.return_code != 0:
+        fail("Unable to run command successfully: %s" % str(cmd))
+
+    return exec_result
+
+def _detect_system_clang(repository_ctx):
+    """Detects whether a system-provided clang can be used.
+
+    Returns a tuple of (is_clang, environment).
+    """
+
+    # If the user provides an explicit `CC` environment variable, use that as
+    # the compiler.
+    cc = repository_ctx.os.environ.get("CC")
+    cxx = repository_ctx.os.environ.get("CXX")
+    if cc or cxx:
+        version_output = _run(repository_ctx, [cc, "--version"]).stdout
+        return "clang" in version_output, {}
+
+    # If we can build our Clang toolchain using a system-installed Clang, try
+    # to do so.
+    system_clang = repository_ctx.which("clang")
+    if system_clang:
+        return True, {
+            "CC": str(system_clang),
+            "CXX": str(system_clang) + "++",
+        }
+    return False, {}
+
+def _get_cmake_defines(repository_ctx, is_clang):
+    """Returns a long list of cmake defines for the bootstrap."""
+    modules_setting = "OFF"
+    if is_clang:
+        modules_setting = "ON"
+
+    static_link_cxx = "ON"
+    unstable_libcxx_abi = "ON"
+    if repository_ctx.os.name.lower().startswith("mac os"):
+        # macOS doesn't support the static C++ standard library linking. Turn
+        # it off here, and disable the unstable libc++ ABI as we will also be
+        # unable to use it later on.
+        static_link_cxx = "OFF"
+        unstable_libcxx_abi = "OFF"
+
+    return [
+        "-DLLVM_ENABLE_PROJECTS=clang;clang-tools-extra;lld;libcxx;libcxxabi;compiler-rt;libunwind",
+        "-DCMAKE_BUILD_TYPE=Release",
+        "-DLLVM_ENABLE_ASSERTIONS=OFF",
+        "-DLLVM_ENABLE_MODULES=" + modules_setting,
+        "-DLLVM_STATIC_LINK_CXX_STDLIB=" + static_link_cxx,
+        "-DLLVM_TARGETS_TO_BUILD=AArch64;X86",
+        "-DLIBCXX_ABI_UNSTABLE=" + unstable_libcxx_abi,
+        "-DLIBCXX_ENABLE_ASSERTIONS=OFF",
+        "-DLIBCXXABI_ENABLE_ASSERTIONS=OFF",
+
+        # Disable components of the build that we don't use while building Carbon.
+        "-DCLANG_ENABLE_ARCMT=OFF",
+        "-DCLANG_INCLUDE_TESTS=OFF",
+        "-DCLANG_TOOL_APINOTES_TEST_BUILD=OFF",
+        "-DCLANG_TOOL_ARCMT_TEST_BUILD=OFF",
+        "-DCLANG_TOOL_CLANG_CHECK_BUILD=OFF",
+        "-DCLANG_TOOL_CLANG_DIFF_BUILD=OFF",
+        "-DCLANG_TOOL_CLANG_EXTDEF_MAPPING_BUILD=OFF",
+        "-DCLANG_TOOL_CLANG_FUZZER_BUILD=OFF",
+        "-DCLANG_TOOL_CLANG_IMPORT_TEST_BUILD=OFF",
+        "-DCLANG_TOOL_CLANG_OFFLOAD_BUNDLER_BUILD=OFF",
+        "-DCLANG_TOOL_CLANG_OFFLOAD_WRAPPER_BUILD=OFF",
+        "-DCLANG_TOOL_CLANG_SCAN_DEPS_BUILD=OFF",
+        "-DCLANG_TOOL_CLANG_SHLIB_BUILD=OFF",
+        "-DCLANG_TOOL_C_ARCMT_TEST_BUILD=OFF",
+        "-DCLANG_TOOL_C_INDEX_TEST_BUILD=OFF",
+        "-DCLANG_TOOL_DIAGTOOL_BUILD=OFF",
+        "-DCLANG_TOOL_LIBCLANG_BUILD=OFF",
+        "-DCLANG_TOOL_SCAN_BUILD_BUILD=OFF",
+        "-DCLANG_TOOL_SCAN_VIEW_BUILD=OFF",
+        "-DLLVM_BUILD_UTILS=OFF",
+        "-DLLVM_ENABLE_BINDINGS=OFF",
+        "-DLLVM_ENABLE_LIBXML2=OFF",
+        "-DLLVM_ENABLE_OCAMLDOC=OFF",
+        "-DLLVM_INCLUDE_BENCHMARKS=OFF",
+        "-DLLVM_INCLUDE_DOCS=OFF",
+        "-DLLVM_INCLUDE_EXAMPLES=OFF",
+        "-DLLVM_INCLUDE_GO_TESTS=OFF",
+        "-DLLVM_INCLUDE_TESTS=OFF",
+        "-DLLVM_INCLUDE_UTILS=OFF",
+        "-DLLVM_TOOL_BUGPOINT_BUILD=OFF",
+        "-DLLVM_TOOL_BUGPOINT_PASSES_BUILD=OFF",
+        "-DLLVM_TOOL_DSYMUTIL_BUILD=OFF",
+        "-DLLVM_TOOL_GOLD_BUILD=OFF",
+        "-DLLVM_TOOL_LLC_BUILD=OFF",
+        "-DLLVM_TOOL_LLI_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_AS_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_BCANALYZER_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_CAT_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_CFI_VERIFY_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_CONFIG_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_CVTRES_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_CXXDUMP_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_CXXFILT_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_CXXMAP_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_C_TEST_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_DIFF_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_DWARFDUMP_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_ELFABI_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_EXEGESIS_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_EXTRACT_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_GO_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_GSYMUTIL_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_IFS_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_ISEL_FUZZER_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_ITANIUM_DEMANGLE_FUZZER_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_JITLINK_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_JITLISTENER_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_LIBTOOL_DARWIN_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_LINK_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_LIPO_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_LTO2_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_LTO_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_MCA_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_MC_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_ML_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_MICROSOFT_DEMANGLE_FUZZER_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_MT_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_OPT_FUZZER_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_PDBUTIL_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_PROFDATA_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_PROFGEN_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_RC_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_READOBJ_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_REDUCE_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_RTDYLD_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_SHLIB_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_SIZE_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_SPECIAL_CASE_LIST_FUZZER_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_SPLIT_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_STRESS_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_STRINGS_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_XRAY_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_YAML_NUMERIC_PARSER_FUZZER_BUILD=OFF",
+        "-DLLVM_TOOL_LLVM_YAML_PARSER_FUZZER_BUILD=OFF",
+        "-DLLVM_TOOL_LTO_BUILD=OFF",
+        "-DLLVM_TOOL_OBJ2YAML_BUILD=OFF",
+        "-DLLVM_TOOL_OPT_BUILD=OFF",
+        "-DLLVM_TOOL_OPT_VIEWER_BUILD=OFF",
+        "-DLLVM_TOOL_REMARKS_SHLIB_BUILD=OFF",
+        "-DLLVM_TOOL_SPLIT_FILE_BUILD=OFF",
+        "-DLLVM_TOOL_VERIFY_USELISTORDER_BUILD=OFF",
+        "-DLLVM_TOOL_YAML2OBJ_BUILD=OFF",
+    ]
+
+def _bootstrap_clang_toolchain_impl(repository_ctx):
+    """Returns the path a bootstrapped Clang executable.
+
+    This bootstraps Clang and the rest of the LLVM toolchain from the LLVM
+    submodule.
+    """
+    repository_ctx.report_progress("Configuring Clang toolchain bootstrap...")
+    is_clang, environment = _detect_system_clang(repository_ctx)
+
+    cmake = repository_ctx.which("cmake")
+    if not cmake:
+        fail("`cmake` not found: is it installed?")
+    ninja = repository_ctx.which("ninja")
+    if not ninja:
+        fail("`ninja` not found: is it installed?")
+
+    workspace_dir = repository_ctx.path(repository_ctx.attr._workspace).dirname
+    llvm_dir = repository_ctx.path("%s/third_party/llvm-project/llvm" %
+                                   workspace_dir)
+
+    repository_ctx.report_progress(
+        "Running CMake for the Clang toolchain build...",
+    )
+    cmake_args = [cmake, "-G", "Ninja", str(llvm_dir)]
+    cmake_args += _get_cmake_defines(repository_ctx, is_clang)
+    _run(
+        repository_ctx,
+        cmake_args,
+        timeout = 600,
+        environment = environment,
+        # This is very slow, so print output as a form of progress.
+        quiet = False,
+    )
+
+    # Run ninja for the final build.
+    repository_ctx.report_progress("Building the Clang toolchain...")
+    _run(
+        repository_ctx,
+        [ninja],
+        timeout = 10800,
+        # This is very slow, so print output as a form of progress.
+        quiet = False,
+    )
+
+    # Create an empty BUILD file to mark the package. The files are used without
+    # Bazel labels directly pointing at them.
+    repository_ctx.file("BUILD", content = "")
+
+bootstrap_clang_toolchain = repository_rule(
+    implementation = _bootstrap_clang_toolchain_impl,
+    configure = True,
+    attrs = {
+        # We use a label pointing at the workspace file to compute the
+        # workspace directory.
+        "_workspace": attr.label(
+            default = Label("//:WORKSPACE"),
+            allow_single_file = True,
+        ),
+    },
+    environ = ["CC", "CXX"],
+)

bazel/cc_toolchains/clang_cc_toolchain_config.bzl

@@ -22,6 +22,7 @@ load(
     "clang_include_dirs_list",
     "clang_resource_dir",
     "llvm_bindir",
+    "sysroot_dir",
 )
 
 all_compile_actions = [
@@ -72,7 +73,7 @@ def _impl(ctx):
         tool_path(name = "ar", path = llvm_bindir + "/llvm-ar"),
         tool_path(name = "ld", path = llvm_bindir + "/ld.lld"),
         tool_path(name = "cpp", path = llvm_bindir + "/clang-cpp"),
-        tool_path(name = "gcc", path = llvm_bindir + "/clang"),
+        tool_path(name = "gcc", path = llvm_bindir + "/clang++"),
         tool_path(name = "dwp", path = llvm_bindir + "/llvm-dwp"),
         tool_path(name = "gcov", path = llvm_bindir + "/llvm-cov"),
         tool_path(name = "nm", path = llvm_bindir + "/llvm-nm"),
@@ -702,18 +703,20 @@ def _impl(ctx):
         use_module_maps,
     ]
 
-    # Select the features based on the target platform. Currently, this is
-    # configured with the "cpu" attribute for legacy reasons. Further, for
-    # legacy reasons the default is a Linux OS target and the x88-64 CPU name
-    # is "k8".
+    # Select the features and builtin include directories based on the target
+    # platform. Currently, this is configured with the "cpu" attribute for
+    # legacy reasons. Further, for legacy reasons the default is a Linux OS
+    # target and the x88-64 CPU name is "k8".
     if (ctx.attr.target_cpu == "k8"):
         features = common_features + [
             linux_link_flags_feature,
             feature(name = "supports_start_end_lib", enabled = True),
             feature(name = "supports_dynamic_linker", enabled = True),
         ]
+        sysroot = None
     elif (ctx.attr.target_cpu == "darwin"):
         features = common_features
+        sysroot = sysroot_dir
     else:
         fail("Unsupported target platform!")
 
@@ -722,9 +725,11 @@ def _impl(ctx):
         features = features,
         action_configs = action_configs,
         cxx_builtin_include_directories = clang_include_dirs_list + [
-            # Append the share directory for sanitizer data files.
+            # Add Clang's resource directory to the end of the builtin include
+            # directories to cover the use of sanitizer resource files by the driver.
             clang_resource_dir + "/share",
         ],
+        builtin_sysroot = sysroot,
 
         # This configuration only supports local non-cross builds so derive
         # everything from the target CPU selected.

bazel/cc_toolchains/clang_detection.bzl → bazel/cc_toolchains/clang_configuration.bzl

@@ -2,79 +2,21 @@
 # Exceptions. See /LICENSE for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
-"""Starlark repository rules to detect and configure Clang (and LLVM) toolchain.
+"""Starlark repository rules to configure Clang (and LLVM) toolchain for Bazel.
 
 These rules should be run from the `WORKSPACE` file to substitute appropriate
 configured values into a `clang_detected_variables.bzl` file that can be used
 by the actual toolchain configuration.
 """
 
-# Tools that we verify are present as part of the detected Clang & LLVM toolchain.
-_CLANG_LLVM_TOOLS = [
-    "llvm-ar",
-    "ld.lld",
-    "clang-cpp",
-    "clang",
-    "clang++",
-    "llvm-dwp",
-    "llvm-cov",
-    "llvm-nm",
-    "llvm-objcopy",
-    "llvm-strip",
-]
-
 def _run(repository_ctx, cmd):
     """Runs the provided `cmd`, checks for failure, and returns the result."""
-    exec_result = repository_ctx.execute(cmd, timeout = 10)
+    exec_result = repository_ctx.execute(cmd)
     if exec_result.return_code != 0:
         fail("Unable to run command successfully: %s" % str(cmd))
 
     return exec_result
 
-def _find_clang(repository_ctx):
-    """Returns the path to a Clang executable if it can find one.
-
-    This assumes the `CC` environment variable points to a Clang binary or
-    looks for one on the path.
-    """
-    clang = repository_ctx.path("%s/third_party/llvm-project/build/bin/clang" %
-                                repository_ctx.attr.workspace_dir)
-    if not clang.exists:
-        cc_env = repository_ctx.os.environ.get("CC")
-        if not cc_env:
-            # Without a specified `CC` name, simply look for `clang`.
-            clang = repository_ctx.which("clang")
-        elif "/" not in cc_env:
-            # Lookup relative `CC` names according to the system `PATH`.
-            clang = repository_ctx.which(cc_env)
-        else:
-            # An absolute `CC` path is simply be used directly.
-            clang = repository_ctx.path(cc_env)
-            if not clang.exists:
-                fail(("The `CC` environment variable is set to a path (`%s`) " +
-                      "that doesn't exist.") % cc_env)
-
-    # Check if either of the `which` invocations fail.
-    if not clang:
-        missing = "`clang`"
-        if cc_env:
-            missing = "`%s` (from the `CC` environment variable)" % cc_env
-        fail("Unable to find the %s executable on the PATH." % missing)
-
-    version_output = _run(repository_ctx, [clang, "--version"]).stdout
-    if "clang" not in version_output:
-        fail(("Selected Clang executable (`%s`) does not appear to actually " +
-              "be Clang.") % clang)
-
-    # Make sure this is part of a complete Clang and LLVM toolchain.
-    for tool in _CLANG_LLVM_TOOLS:
-        if not clang.dirname.get_child(tool).exists:
-            fail(("Couldn't find executable `%s` that is expected to be part " +
-                  "of the Clang and LLVM toolchain detected with `%s`.") %
-                 (tool, clang))
-
-    return clang
-
 def _compute_clang_resource_dir(repository_ctx, clang):
     """Runs the `clang` binary to get its resource dir."""
     output = _run(
@@ -85,7 +27,15 @@ def _compute_clang_resource_dir(repository_ctx, clang):
     # The only line printed is this path.
     return output.splitlines()[0]
 
-def _compute_clang_cpp_include_search_paths(repository_ctx, clang):
+def _compute_mac_os_sysroot(repository_ctx):
+    """Runs `xcrun` to extract the correct sysroot."""
+    xcrun = repository_ctx.which("xcrun")
+    if not xcrun:
+        fail("`xcrun` not found: is Xcode installed?")
+    output = _run(repository_ctx, [xcrun, "--show-sdk-path"]).stdout
+    return output.splitlines()[0]
+
+def _compute_clang_cpp_include_search_paths(repository_ctx, clang, sysroot):
     """Runs the `clang` binary and extracts the include search paths.
 
     Returns the resulting paths as a list of strings.
@@ -101,15 +51,21 @@ def _compute_clang_cpp_include_search_paths(repository_ctx, clang):
         "-v",
         # Just parse the input, don't generate outputs.
         "-fsyntax-only",
-        # Use libc++ rather than any other standard library.
-        "-stdlib=libc++",
         # Force the language to be C++.
         "-x",
         "c++",
         # Read in an empty input file.
         "/dev/null",
+        # Always use libc++.
+        "-stdlib=libc++",
     ]
 
+    # We need to use a sysroot to correctly represent a run on macOS.
+    if repository_ctx.os.name.lower().startswith("mac os"):
+        if not sysroot:
+            fail("Must provide a sysroot on macOS!")
+        cmd.append("--sysroot=" + sysroot)
+
     # Note that verbose output is on stderr, not stdout!
     output = _run(repository_ctx, cmd).stderr.splitlines()
 
@@ -124,7 +80,7 @@ def _compute_clang_cpp_include_search_paths(repository_ctx, clang):
         for s in output[include_begin:include_end]
     ]
 
-def _detect_clang_toolchain_impl(repository_ctx):
+def _configure_clang_toolchain_impl(repository_ctx):
     # First just symlink in the untemplated parts of the toolchain repo.
     repository_ctx.symlink(repository_ctx.attr._clang_toolchain_build, "BUILD")
     repository_ctx.symlink(
@@ -132,11 +88,23 @@ def _detect_clang_toolchain_impl(repository_ctx):
         "cc_toolchain_config.bzl",
     )
 
-    clang = _find_clang(repository_ctx)
+    # Run the bootstrapped clang to detect relevant features for the toolchain.
+    clang = repository_ctx.path(repository_ctx.attr.clang)
+    if clang.basename != "clang":
+        fail("The provided Clang binary must be `clang`, but is `%s` (%s)" %
+             (clang.basename, str(clang)))
+
+    # Adjust this to the "clang++" binary to ensure we get the correct behavior
+    # when configuring it.
+    clang = repository_ctx.path(str(clang) + "++")
     resource_dir = _compute_clang_resource_dir(repository_ctx, clang)
+    sysroot_dir = None
+    if repository_ctx.os.name.lower().startswith("mac os"):
+        sysroot_dir = _compute_mac_os_sysroot(repository_ctx)
     include_dirs = _compute_clang_cpp_include_search_paths(
         repository_ctx,
         clang,
+        sysroot_dir,
     )
 
     repository_ctx.template(
@@ -145,29 +113,39 @@ def _detect_clang_toolchain_impl(repository_ctx):
         substitutions = {
             "{LLVM_BINDIR}": str(clang.dirname),
             "{CLANG_RESOURCE_DIR}": resource_dir,
-            "{CLANG_INCLUDE_DIRS_LIST}": str([str(path) for path in include_dirs]),
+            "{CLANG_INCLUDE_DIRS_LIST}": str(
+                [str(path) for path in include_dirs],
+            ),
+            "{SYSROOT}": str(sysroot_dir),
         },
         executable = False,
     )
 
-detect_clang_toolchain = repository_rule(
-    implementation = _detect_clang_toolchain_impl,
+configure_clang_toolchain = repository_rule(
+    implementation = _configure_clang_toolchain_impl,
     configure = True,
-    local = True,
     attrs = {
-        "workspace_dir": attr.string(mandatory = True),
         "_clang_toolchain_build": attr.label(
             default = Label("//bazel/cc_toolchains:clang_toolchain.BUILD"),
             allow_single_file = True,
         ),
         "_clang_cc_toolchain_config": attr.label(
-            default = Label("//bazel/cc_toolchains:clang_cc_toolchain_config.bzl"),
+            default = Label(
+                "//bazel/cc_toolchains:clang_cc_toolchain_config.bzl",
+            ),
             allow_single_file = True,
         ),
         "_clang_detected_variables_template": attr.label(
-            default = Label("//bazel/cc_toolchains:clang_detected_variables.tpl.bzl"),
+            default = Label(
+                "//bazel/cc_toolchains:clang_detected_variables.tpl.bzl",
+            ),
+            allow_single_file = True,
+        ),
+        # This must point at the `clang` binary inside a full LLVM toolchain
+        # installation.
+        "clang": attr.label(
             allow_single_file = True,
+            mandatory = True,
         ),
     },
-    environ = ["CC"],
 )

bazel/cc_toolchains/clang_detected_variables.tpl.bzl

@@ -11,3 +11,4 @@ This file gets processed by a repository rule, substituting the
 llvm_bindir = "{LLVM_BINDIR}"
 clang_resource_dir = "{CLANG_RESOURCE_DIR}"
 clang_include_dirs_list = {CLANG_INCLUDE_DIRS_LIST}
+sysroot_dir = "{SYSROOT}"

docs/project/contribution_tools.md

@@ -64,8 +64,6 @@ typical tool setup flow is:
     -   `git config core.fsmonitor rs-git-fsmonitor` to set up
         [rs-git-fsmonitor](#rs-git-fsmonitor-and-watchman) in the clone.
     -   `pre-commit install` to set up [pre-commit](#pre-commit) in the clone.
-4.  If building [Clang and LLVM](#clang-and-llvm), do so in the cloned
-    repository.
 
 ## Package managers
 
@@ -176,40 +174,12 @@ Our recommended way of installing is:
 ### Clang and LLVM
 
 [Clang](https://clang.llvm.org/) and [LLVM](https://llvm.org/) are used to
-compile and link Carbon.
-
-If the `CC` environment variable is set to `clang`, or if `clang` is in the
-path, it can be used without building LLVM from head. However, it's currently
-recommended to build LLVM from head in order to avoid compatibility issues in
-many distributions. Once `third_party/llvm-project/build/bin/clang` exists, it
-will automatically be used instead of `CC`.
-
-In order to build `third_party/llvm-project/build/bin/clang`, please refer to
-[LLVM's instructions](https://llvm.org/docs/GettingStarted.html#getting-the-source-code-and-building-llvm).
-Tips:
-
--   When `carbon-lang` is checked out with submodules, the LLVM submodule is at
-    `third_party/llvm-project`.
-
-    -   In other words, create your `build` directory at
-        `third_party/llvm-project/build`.
-
--   Carbon uses the Clang, LLD, and libc++ subprojects. A corresponding
-    `LLVM_ENABLE_PROJECTS` build looks like:
-
-    ```bash
-    cmake -G Ninja ../llvm \
-        -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;libcxx;libcxxabi;compiler-rt;libunwind" \
-        -DCMAKE_BUILD_TYPE=Release \
-        -DLIBCXX_ABI_UNSTABLE=ON \
-        -DLLVM_ENABLE_ASSERTIONS=OFF \
-        -DLIBCXX_ENABLE_ASSERTIONS=OFF \
-        -DLIBCXXABI_ENABLE_ASSERTIONS=OFF
-    ninja
-    ```
-
-    Note these instructions use [ninja](#ninja), which should be installed
-    first.
+compile and link Carbon, and are provided through git submodules. A complete
+toolchain will be built and cached as part of standard `bazel` execution. This
+can be very slow on less powerful computers or laptops (30 minutes to an hour).
+However, it should only happen when either Bazel or LLVM's submodule is updated,
+which we try to minimize. If you need to force a rebuild of the toolchain, you
+can use `bazel sync --configure`.
 
 ### Ninja