Add a native Bazel build for libpfm and use it in benchmarks. (#3601)

The `libpfm` in the Bazel central repository uses `make` to build it,
which is difficult to integrate with our toolchain. Rather than try to
fix all the issues there, it's easy to just add a native Bazel build for
the library. I don't know that any of the relevant upstream folks are
interested in this kind of build, but it seems easy for us to maintain
as a Carbon project build configuration. I've also not tried to port all
of the different configurations as a consequence, and only 64-bit x86
and Arm as that seems the only likely architectures we'll care about in
the near term.

I've kept this using the `bzlmod` stuff as best I can, and I *think* I'm
holding all of those pieces correctly, but if not, happy for suggestions
on adjustments.

The `google_benchmark` package also has an awkward way of enabling
`libpfm` support using a top-level `bazel` command line flag. I think
this is because of how brittle the Bazel build of `libpfm` is, but I'm
not sure. With the new build, it seems easy to patch `google_benchmark`
to detect the same conditions as we build `libpfm` under, and enable it
there. So I've done this to avoid folks having to pass a command line
flag on platforms where it is supported.

The result is that we now get really nice CPU counter support in our
benchmarks out-of-the-box on Linux x86-64 and AArch64. For example on my
Fedora Asahi install on a Mac Mini I get:

```console
$ bazel run -c opt --copt=-gmlt //common:hashing_benchmark --run_under="taskset -c 4" -- --benchmark_counters_tabular=true --benchmark_perf_counters=CYCLES,INSTRUCTIONS
INFO: Invocation ID: 4aaeb9e9-7df5-4f1f-b56b-c03411790268
INFO: Analyzed target //common:hashing_benchmark (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //common:hashing_benchmark up-to-date:
  bazel-bin/common/hashing_benchmark
INFO: Elapsed time: 0.360s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: /bin/bash -c 'taskset -c 4 bazel-bin/common/hashing_benchmark '\''--benchmark_counters_tabular=true'\'' '\''--benchmark_perf_counters=CYCLES,INSTRUCTIONS'\'''
2024-01-15T00:10:50-08:00
Running /home/chandlerc/.cache/bazel/_bazel_chandlerc/b686aa8910e0845b88c21d715819b076/execroot/_main/bazel-out/aarch64-opt/bin/common/hashing_benchmark
Run on (8 X 2064 MHz CPU s)
CPU Caches:
  L1 Data 64 KiB (x8)
  L1 Instruction 128 KiB (x8)
  L2 Unified 4096 KiB (x2)
Load Average: 0.01, 0.08, 0.08
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Benchmark                                                                           Time             CPU   Iterations     CYCLES INSTRUCTIONS bytes_per_second
--------------------------------------------------------------------------------------------------------------------------------------------------------------
BM_LatencyHash<RandValues<uint8_t>, CarbonHashBench>                             4.11 ns         4.11 ns    170200064    13.1321      9.00587      232.116Mi/s
BM_LatencyHash<RandValues<uint8_t>, AbseilHashBench>                             4.82 ns         4.82 ns    145643520    15.3657      12.0059      197.946Mi/s
BM_LatencyHash<RandValues<uint8_t>, LLVMHashBench>                               7.96 ns         7.95 ns     87956480    25.3737      17.0068      119.991Mi/s
BM_LatencyHash<RandValues<uint16_t>, CarbonHashBench>                            4.11 ns         4.11 ns    170365952    13.1247      9.00587      464.573Mi/s
BM_LatencyHash<RandValues<uint16_t>, AbseilHashBench>                            5.51 ns         5.51 ns    127568896    17.5578      14.0059      346.225Mi/s
BM_LatencyHash<RandValues<uint16_t>, LLVMHashBench>                              8.00 ns         7.99 ns     87085056     25.377      17.0068      238.834Mi/s
BM_LatencyHash<RandValues<std::pair<uint8_t, uint8_t>>, CarbonHashBench>         4.91 ns         4.90 ns    136013824    15.6456      14.0059      389.006Mi/s
BM_LatencyHash<RandValues<std::pair<uint8_t, uint8_t>>, AbseilHashBench>         6.85 ns         6.85 ns    102630400    21.8041      18.0059      278.637Mi/s
BM_LatencyHash<RandValues<std::pair<uint8_t, uint8_t>>, LLVMHashBench>           7.57 ns         7.56 ns     92798976    24.1437      20.0068      252.151Mi/s
BM_LatencyHash<RandValues<uint32_t>, CarbonHashBench>                            4.12 ns         4.12 ns    170229760    13.1272      9.00587      926.444Mi/s
BM_LatencyHash<RandValues<uint32_t>, AbseilHashBench>                            4.93 ns         4.92 ns    145304576    15.3738      12.0059      775.224Mi/s
BM_LatencyHash<RandValues<uint32_t>, LLVMHashBench>                              8.11 ns         8.10 ns     87127040     25.373      17.0068       470.98Mi/s
```

.pre-commit-config.yaml

@@ -233,6 +233,8 @@ exclude: |
   (?x)^(
       MODULE.bazel.lock|
       bazel/bazel_clang_tidy/.*\.patch|
+      bazel/google_benchmark/.*\.patch|
+      bazel/libpfm/.*\.patch|
       bazel/llvm_project/.*\.patch|
       third_party/examples/.*/carbon/.*|
   )$

MODULE.bazel

@@ -37,11 +37,41 @@ bazel_dep(
     version = "1.14.0.bcr.1",
     repo_name = "com_google_googletest",
 )
+
+google_benchmark_version = "1.8.3"
+
 bazel_dep(
     name = "google_benchmark",
-    version = "1.8.3",
+    version = google_benchmark_version,
     repo_name = "com_github_google_benchmark",
 )
+archive_override(
+    module_name = "google_benchmark",
+    integrity = "sha256-a8GApX0j1NlRVRn5KwyD1hsFtbqxiJYfNqx7BrDZ6c4=",
+    patch_strip = 1,
+    patches = ["@//bazel/google_benchmark:0001-Use-libpfm-by-default-on-supported-platforms.patch"],
+    strip_prefix = "benchmark-{0}".format(google_benchmark_version),
+    urls = ["https://github.com/google/benchmark/archive/refs/tags/v{0}.tar.gz".format(google_benchmark_version)],
+)
+
+bazel_dep(
+    name = "libpfm",
+    # The registry only has an old version. We use that here to avoid a miss but
+    # override it with a newer version.
+    version = "4.11.0",
+)
+
+libpfm_version = "4.13.0"
+
+archive_override(
+    module_name = "libpfm",
+    integrity = "sha256-0YuXdkx1VSjBBR03bjNUXQ62DG6/hWgENoE/pbBMw9E=",
+    patch_strip = 1,
+    patches = ["@//bazel/libpfm:0001-Introduce-a-simple-native-Bazel-build.patch"],
+    strip_prefix = "libpfm-{0}".format(libpfm_version),
+    urls = ["https://sourceforge.net/projects/perfmon2/files/libpfm4/libpfm-{0}.tar.gz".format(libpfm_version)],
+)
+
 bazel_dep(name = "rules_bison", version = "0.2.2")
 bazel_dep(name = "rules_flex", version = "0.2.1")
 bazel_dep(name = "rules_m4", version = "0.2.3")

MODULE.bazel.lock

@@ -1,6 +1,6 @@
 {
   "lockFileVersion": 3,
-  "moduleFileHash": "5704ebd75cf3cb5f4ab56ba2939bb74d98752cd7eed0ea9357a9ed39ca41640e",
+  "moduleFileHash": "ddc3243fab92897b86a0cf8a5d3f048e44bf17ca432eb5ec454fa91710337db6",
   "flags": {
     "cmdRegistries": [
       "https://bcr.bazel.build/"
@@ -56,7 +56,7 @@
               "devDependency": false,
               "location": {
                 "file": "@@//:MODULE.bazel",
-                "line": 58,
+                "line": 88,
                 "column": 13
               }
             },
@@ -79,7 +79,7 @@
               "devDependency": false,
               "location": {
                 "file": "@@//:MODULE.bazel",
-                "line": 68,
+                "line": 98,
                 "column": 13
               }
             },
@@ -105,7 +105,7 @@
               "devDependency": false,
               "location": {
                 "file": "@@//:MODULE.bazel",
-                "line": 108,
+                "line": 138,
                 "column": 13
               }
             }
@@ -119,7 +119,7 @@
           "usingModule": "<root>",
           "location": {
             "file": "@@//:MODULE.bazel",
-            "line": 77,
+            "line": 107,
             "column": 35
           },
           "imports": {
@@ -136,7 +136,7 @@
           "usingModule": "<root>",
           "location": {
             "file": "@@//:MODULE.bazel",
-            "line": 123,
+            "line": 153,
             "column": 29
           },
           "imports": {
@@ -153,7 +153,7 @@
           "usingModule": "<root>",
           "location": {
             "file": "@@//:MODULE.bazel",
-            "line": 135,
+            "line": 165,
             "column": 23
           },
           "imports": {
@@ -169,7 +169,7 @@
               "devDependency": false,
               "location": {
                 "file": "@@//:MODULE.bazel",
-                "line": 136,
+                "line": 166,
                 "column": 17
               }
             }
@@ -183,7 +183,7 @@
           "usingModule": "<root>",
           "location": {
             "file": "@@//:MODULE.bazel",
-            "line": 142,
+            "line": 172,
             "column": 20
           },
           "imports": {
@@ -201,7 +201,7 @@
               "devDependency": false,
               "location": {
                 "file": "@@//:MODULE.bazel",
-                "line": 143,
+                "line": 173,
                 "column": 10
               }
             }
@@ -215,7 +215,8 @@
         "com_google_absl": "abseil-cpp@20230802.0",
         "com_googlesource_code_re2": "re2@2023-11-01",
         "com_google_googletest": "googletest@1.14.0.bcr.1",
-        "com_github_google_benchmark": "google_benchmark@1.8.3",
+        "com_github_google_benchmark": "google_benchmark@_",
+        "libpfm": "libpfm@_",
         "rules_bison": "rules_bison@0.2.2",
         "rules_flex": "rules_flex@0.2.1",
         "rules_m4": "rules_m4@0.2.3",
@@ -275,7 +276,7 @@
         "platforms": "platforms@0.0.8",
         "bazel_skylib": "bazel_skylib@1.5.0",
         "com_google_googletest": "googletest@1.14.0.bcr.1",
-        "com_github_google_benchmark": "google_benchmark@1.8.3",
+        "com_github_google_benchmark": "google_benchmark@_",
         "bazel_tools": "bazel_tools@_",
         "local_config_platform": "local_config_platform@_"
       },
@@ -393,10 +394,10 @@
         }
       }
     },
-    "google_benchmark@1.8.3": {
+    "google_benchmark@_": {
       "name": "google_benchmark",
       "version": "1.8.3",
-      "key": "google_benchmark@1.8.3",
+      "key": "google_benchmark@_",
       "repoName": "google_benchmark",
       "executionPlatformsToRegister": [],
       "toolchainsToRegister": [],
@@ -406,23 +407,24 @@
         "platforms": "platforms@0.0.8",
         "rules_foreign_cc": "rules_foreign_cc@0.9.0",
         "rules_cc": "rules_cc@0.0.9",
-        "libpfm": "libpfm@4.11.0",
+        "libpfm": "libpfm@_",
+        "bazel_tools": "bazel_tools@_",
+        "local_config_platform": "local_config_platform@_"
+      }
+    },
+    "libpfm@_": {
+      "name": "libpfm",
+      "version": "",
+      "key": "libpfm@_",
+      "repoName": "libpfm",
+      "executionPlatformsToRegister": [],
+      "toolchainsToRegister": [],
+      "extensionUsages": [],
+      "deps": {
+        "rules_cc": "rules_cc@0.0.9",
+        "platforms": "platforms@0.0.8",
         "bazel_tools": "bazel_tools@_",
         "local_config_platform": "local_config_platform@_"
-      },
-      "repoSpec": {
-        "bzlFile": "@bazel_tools//tools/build_defs/repo:http.bzl",
-        "ruleClassName": "http_archive",
-        "attributes": {
-          "name": "google_benchmark~1.8.3",
-          "urls": [
-            "https://github.com/google/benchmark/archive/refs/tags/v1.8.3.tar.gz"
-          ],
-          "integrity": "sha256-a8GApX0j1NlRVRn5KwyD1hsFtbqxiJYfNqx7BrDZ6c4=",
-          "strip_prefix": "benchmark-1.8.3",
-          "remote_patches": {},
-          "remote_patch_strip": 0
-        }
       }
     },
     "rules_bison@0.2.2": {
@@ -1214,38 +1216,6 @@
         }
       }
     },
-    "libpfm@4.11.0": {
-      "name": "libpfm",
-      "version": "4.11.0",
-      "key": "libpfm@4.11.0",
-      "repoName": "libpfm",
-      "executionPlatformsToRegister": [],
-      "toolchainsToRegister": [],
-      "extensionUsages": [],
-      "deps": {
-        "platforms": "platforms@0.0.8",
-        "rules_foreign_cc": "rules_foreign_cc@0.9.0",
-        "bazel_tools": "bazel_tools@_",
-        "local_config_platform": "local_config_platform@_"
-      },
-      "repoSpec": {
-        "bzlFile": "@bazel_tools//tools/build_defs/repo:http.bzl",
-        "ruleClassName": "http_archive",
-        "attributes": {
-          "name": "libpfm~4.11.0",
-          "urls": [
-            "https://sourceforge.net/projects/perfmon2/files/libpfm4/libpfm-4.11.0.tar.gz"
-          ],
-          "integrity": "sha256-XaX4hyveFLNjTJaI2YD2i9ootRAmhyPMEpc+7bq5/sw=",
-          "strip_prefix": "libpfm-4.11.0",
-          "remote_patches": {
-            "https://bcr.bazel.build/modules/libpfm/4.11.0/patches/module_dot_bazel.patch": "sha256-G0wQJ2mVEoW/L5LGzmbNfuZaxI2+9NDuWJtqvCpM1pc=",
-            "https://bcr.bazel.build/modules/libpfm/4.11.0/patches/add_build_file.patch": "sha256-E61d/qQgmeOcUliWaveHPp1EZoOjkvZJsqhGhHofqUg="
-          },
-          "remote_patch_strip": 0
-        }
-      }
-    },
     "rules_license@0.0.7": {
       "name": "rules_license",
       "version": "0.0.7",

bazel/google_benchmark/0001-Use-libpfm-by-default-on-supported-platforms.patch

@@ -0,0 +1,62 @@
+From 4110acfa9a131a5a93ae4caef26bd34dd5509155 Mon Sep 17 00:00:00 2001
+From: Chandler Carruth <chandlerc@gmail.com>
+Date: Sat, 13 Jan 2024 02:58:47 -0800
+Subject: [PATCH] Use libpfm by default on supported platforms.
+
+---
+ BUILD.bazel | 26 ++++++++++++++++++--------
+ 1 file changed, 18 insertions(+), 8 deletions(-)
+
+diff --git a/BUILD.bazel b/BUILD.bazel
+index 60d31d2..c313c8a 100644
+--- a/BUILD.bazel
++++ b/BUILD.bazel
+@@ -1,3 +1,5 @@
++load("@bazel_skylib//lib:selects.bzl", "selects")
++
+ licenses(["notice"])
+
+ config_setting(
+@@ -24,12 +26,20 @@ config_setting(
+     visibility = ["//visibility:public"],
+ )
+
+-config_setting(
+-    name = "perfcounters",
+-    define_values = {
+-        "pfm": "1",
+-    },
+-    visibility = [":__subpackages__"],
++selects.config_setting_group(
++    name = "aarch64_or_x86_64",
++    match_any = [
++        "@platforms//cpu:aarch64",
++        "@platforms//cpu:x86_64",
++    ],
++)
++
++selects.config_setting_group(
++    name = "linux_aarch64_or_x86_64",
++    match_all = [
++        "@platforms//os:linux",
++        ":aarch64_or_x86_64",
++    ],
+ )
+
+ cc_library(
+@@ -62,11 +72,11 @@ cc_library(
+     defines = [
+         "BENCHMARK_STATIC_DEFINE",
+     ] + select({
+-        ":perfcounters": ["HAVE_LIBPFM"],
++        ":linux_aarch64_or_x86_64": ["HAVE_LIBPFM"],
+         "//conditions:default": [],
+     }),
+     deps = select({
+-        ":perfcounters": ["@libpfm//:libpfm"],
++        ":linux_aarch64_or_x86_64": ["@libpfm//:libpfm"],
+         "//conditions:default": [],
+     }),
+ )
+--
+2.43.0

bazel/google_benchmark/BUILD

@@ -0,0 +1,9 @@
+# Part of the Carbon Language project, under the Apache License v2.0 with LLVM
+# Exceptions. See /LICENSE for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+package(default_visibility = ["//visibility:public"])
+
+exports_files(glob([
+    "*.patch",
+]))

bazel/libpfm/0001-Introduce-a-simple-native-Bazel-build.patch

@@ -0,0 +1,133 @@
+From 04fb28b5673d29a8c38519845c87f4c00c76e9cf Mon Sep 17 00:00:00 2001
+From: Chandler Carruth <chandlerc@gmail.com>
+Date: Sat, 13 Jan 2024 02:15:19 -0800
+Subject: [PATCH] Introduce a simple native Bazel build.
+
+---
+ BUILD.bazel     | 84 +++++++++++++++++++++++++++++++++++++++++++++++++
+ MODULE.bazel    | 10 ++++++
+ WORKSPACE.bazel |  5 +++
+ 3 files changed, 99 insertions(+)
+ create mode 100644 BUILD.bazel
+ create mode 100644 MODULE.bazel
+ create mode 100644 WORKSPACE.bazel
+
+diff --git a/BUILD.bazel b/BUILD.bazel
+new file mode 100644
+index 0000000..427c854
+--- /dev/null
++++ b/BUILD.bazel
+@@ -0,0 +1,84 @@
++# Part of the Carbon Language project, under the Apache License v2.0 with LLVM
++# Exceptions. See /LICENSE for license information.
++# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
++
++load("@rules_cc//cc:defs.bzl", "cc_library")
++
++package(default_visibility = ["//visibility:public"])
++
++aarch64_srcs = [
++    "lib/pfmlib_arm_perf_event.c",
++    "lib/pfmlib_arm.c",
++    "lib/pfmlib_arm_armv8.c",
++    "lib/pfmlib_arm_armv9.c",
++    "lib/pfmlib_tx2_unc_perf_event.c",
++    "lib/pfmlib_kunpeng_unc_perf_event.c",
++    "lib/pfmlib_arm_priv.h",
++    "lib/events/arm_cortex_a57_events.h",
++    "lib/events/arm_cortex_a53_events.h",
++    "lib/events/arm_xgene_events.h",
++    "lib/events/arm_cavium_tx2_events.h",
++    "lib/events/arm_marvell_tx2_unc_events.h",
++    "lib/events/arm_fujitsu_a64fx_events.h",
++    "lib/events/arm_neoverse_n1_events.h",
++    "lib/events/arm_neoverse_n2_events.h",
++    "lib/events/arm_neoverse_v1_events.h",
++    "lib/events/arm_neoverse_v2_events.h",
++    "lib/events/arm_hisilicon_kunpeng_events.h",
++    "lib/events/arm_hisilicon_kunpeng_unc_events.h",
++]
++
++x86_64_srcs = [
++    "lib/pfmlib_amd64_priv.h",
++] + glob(
++    [
++        "lib/pfmlib_amd64*.c",
++        "lib/pfmlib_intel*.c",
++        "lib/pfmlib_intel*_priv.h",
++        "lib/events/amd64_events_*.h",
++        "lib/events/intel_*_events.h",
++    ],
++    exclude = [
++        # 32-bit CPUs
++        "lib/pfmlib_intel_coreduo.c",
++        "lib/pfmlib_intel_p6.c",
++    ],
++)
++
++cc_library(
++    name = "libpfm",
++    srcs = [
++        "lib/events/perf_events.h",
++        "lib/pfmlib_common.c",
++        "lib/pfmlib_perf_event.c",
++        "lib/pfmlib_perf_event_pmu.c",
++        "lib/pfmlib_perf_event_priv.h",
++        "lib/pfmlib_perf_event_raw.c",
++        "lib/pfmlib_priv.h",
++    ] + select({
++        "@platforms//cpu:aarch64": aarch64_srcs,
++        "@platforms//cpu:x86_64": x86_64_srcs,
++    }),
++    hdrs = glob(["include/perfmon/*.h"]),
++    copts = [
++        "-DHAS_OPENAT",
++        "-D_REENTRANT",
++        "-I.",
++        "-fvisibility=hidden",
++    ] + select({
++        "@platforms//cpu:x86_64": [
++            "-DCONFIG_PFMLIB_ARCH_X86",
++            "-DCONFIG_PFMLIB_ARCH_X86_64",
++        ],
++        "//conditions:default": [],
++    }),
++    strip_include_prefix = "include",
++    target_compatible_with = select({
++        # This library only makes sense on Linux, and we only include support
++        # for building on AArch64 and x86-64. Other CPUs can be added to this
++        # list if build support is added for them.
++        "@platforms//cpu:aarch64": ["@platforms//os:linux"],
++        "@platforms//cpu:x86_64": ["@platforms//os:linux"],
++        "//conditions:default": ["@platforms//:incompatible"],
++    }),
++)
+diff --git a/MODULE.bazel b/MODULE.bazel
+new file mode 100644
+index 0000000..c901cbe
+--- /dev/null
++++ b/MODULE.bazel
+@@ -0,0 +1,10 @@
++# Part of the Carbon Language project, under the Apache License v2.0 with LLVM
++# Exceptions. See /LICENSE for license information.
++# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
++
++"""Bazel modules."""
++
++module(name = "libpfm")
++
++bazel_dep(name = "rules_cc", version = "0.0.9")
++bazel_dep(name = "platforms", version = "0.0.8")
+diff --git a/WORKSPACE.bazel b/WORKSPACE.bazel
+new file mode 100644
+index 0000000..9aad57c
+--- /dev/null
++++ b/WORKSPACE.bazel
+@@ -0,0 +1,5 @@
++# Part of the Carbon Language project, under the Apache License v2.0 with LLVM
++# Exceptions. See /LICENSE for license information.
++# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
++
++# See `MODULE.bazel` for details.
+--
+2.43.0

bazel/libpfm/BUILD

@@ -0,0 +1,9 @@
+# Part of the Carbon Language project, under the Apache License v2.0 with LLVM
+# Exceptions. See /LICENSE for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+package(default_visibility = ["//visibility:public"])
+
+exports_files(glob([
+    "*.patch",
+]))