Safety milestones and a 2025 roadmap (#4880)

We propose updating our milestones to accelerate design and
implementation of
memory safety in Carbon, and a roadmap for 2025 reflecting this change.
We also
provide a retrospective for 2024's progress.

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>

README.md

@@ -153,8 +153,9 @@ and the language:
     Carbon (and eventually C++ code as well) into standard executable code. This
     is where most of our current implementation efforts are directed.
 
-If you're interested in contributing, we're currently focused on
-[developing the Carbon toolchain until it can support Carbon ↔ C++ interop](/docs/project/roadmap.md#objective-for-2024-a-working-toolchain-that-supports-c-interop).
+If you're interested in contributing, we're currently focused on developing the
+Carbon toolchain until it can
+[support Carbon ↔ C++ interop](/docs/project/roadmap.md#access-most-non-template-c-apis-in-carbon).
 Beyond that, we plan to continue developing the design and toolchain until we
 can ship the
 [0.1 language](/docs/project/milestones.md#milestone-01-a-minimum-viable-product-mvp-for-evaluation)

docs/project/milestones.md

@@ -83,6 +83,9 @@ around what we expect evaluations to be able to include:
     the critical path and get representative performance results.
     -   This can in turn be a still smaller subset of all aspects of C++
         interoperability based around what impacts interesting benchmarks.
+-   Both strategy and design for memory safety allow evaluators to be confident
+    in safe Carbon having strong memory safety protections, and being
+    incrementally adoptable starting from existing C++ codebases.
 
 ### Language features
 
@@ -226,6 +229,17 @@ to be completed as part of 0.1 beyond _language_ features:
         those platforms.
 -   Build system integration for CMake, and documentation for integrating with
     Make or similar build systems.
+-   Detailed safety strategy for Carbon
+    -   Will include specific expectations for how unsafe C++ code and unsafe
+        Carbon code will interact with safe Carbon code.
+    -   Also includes any tradeoffs or prioritization across different kinds or
+        levels of safety.
+-   Detailed and concrete design for safe Carbon
+    -   Must at least include ways in which most modern C++ is safe: type and
+        initialization safety.
+    -   Must also include spatial, temporal, and mutation safety.
+    -   Will include an analysis of how this impacts _safe_ Rust interop.
+    -   Does _not_ include having a complete implementation in 0.1.
 -   Basic documentation for evaluators from getting started to FAQs.
 
 ## Milestone 0.2: feature complete product for evaluation

docs/project/roadmap.md

@@ -10,130 +10,113 @@ SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
 ## Table of contents
 
--   [Objective for 2024: a working toolchain that supports C++ interop](#objective-for-2024-a-working-toolchain-that-supports-c-interop)
--   [Key results in 2024](#key-results-in-2024)
-    -   [Carbon's toolchain implements enough of the language to build realistic code](#carbons-toolchain-implements-enough-of-the-language-to-build-realistic-code)
-    -   [Carbon's toolchain can build C++ code](#carbons-toolchain-can-build-c-code)
-    -   [Carbon's toolchain works with existing, simple C++ build systems](#carbons-toolchain-works-with-existing-simple-c-build-systems)
-    -   [Carbon has a design and toolchain implementation of basic C++ interop](#carbon-has-a-design-and-toolchain-implementation-of-basic-c-interop)
-    -   [Give talks at 2-3 conferences covering 3-4 different Carbon topics](#give-talks-at-2-3-conferences-covering-3-4-different-carbon-topics)
-    -   [Start building our initial tutorial and introductory material](#start-building-our-initial-tutorial-and-introductory-material)
--   [Beyond 2024](#beyond-2024)
-    -   [Potential 2025 goals: ship a working 0.1 language for evaluation](#potential-2025-goals-ship-a-working-01-language-for-evaluation)
-    -   [Potential 2026-2027 goals: finish 0.2 language, stop experimenting](#potential-2026-2027-goals-finish-02-language-stop-experimenting)
-    -   [Potential goals _beyond_ 2027: ship 1.0 language & organization](#potential-goals-beyond-2027-ship-10-language--organization)
+-   [Objectives for 2025: demo of C++ interop and design of memory safety](#objectives-for-2025-demo-of-c-interop-and-design-of-memory-safety)
+-   [Key results in 2025](#key-results-in-2025)
+    -   [Access most non-template C++ APIs in Carbon](#access-most-non-template-c-apis-in-carbon)
+    -   [Access non-generic Carbon APIs in C++](#access-non-generic-carbon-apis-in-c)
+    -   [Detailed safety strategy update, including expected tradeoffs and prioritization](#detailed-safety-strategy-update-including-expected-tradeoffs-and-prioritization)
+    -   [Design for compile-time temporal and mutation memory safety](#design-for-compile-time-temporal-and-mutation-memory-safety)
+    -   [Give talks at 2-3 conferences about Carbon topics, expanding our audience](#give-talks-at-2-3-conferences-about-carbon-topics-expanding-our-audience)
+-   [Beyond 2025](#beyond-2025)
+    -   [Potential 2026 goals: ship a working 0.1 language for evaluation](#potential-2026-goals-ship-a-working-01-language-for-evaluation)
+    -   [Potential 2027-2028 goals: finish 0.2 language, stop experimenting](#potential-2027-2028-goals-finish-02-language-stop-experimenting)
+    -   [Potential goals _beyond_ 2028: ship 1.0 language & organization](#potential-goals-beyond-2028-ship-10-language--organization)
 
 <!-- tocstop -->
 
-## Objective for 2024: a working toolchain that supports C++ interop
+## Objectives for 2025: demo of C++ interop and design of memory safety
 
-Our focus for 2024 will be to get the Carbon toolchain working, including C++
-interop. We see three key criteria:
+We have two areas of focus for 2025:
 
--   Building realistic Carbon code for interesting interop with C++.
--   Building realistic C++ code for interesting interop with Carbon.
--   The interop itself to allow a single program mixing the two languages.
+1. Get a major chunk of our C++ interop working to the point where we can
+   demonstrate it in realistic scenarios.
+2. Build a concrete and specific design for memory safety in Carbon.
 
-This will both allow folks to explore Carbon using a more traditional and
-realistic compiler model, and allow that exploratory Carbon to lean on C++ for
-libraries and other functionality that doesn't exist in Carbon yet. It will also
-demonstrate how the interop will work in practice.
+We will scope the first one to non-template C++ APIs, and prioritize accessing
+C++ APIs from Carbon. This still will require major progress on the
+implementation of all the relevant Carbon features, and even design in some
+cases.
 
-This objective and focus are oriented around the toolchain and implementation of
-Carbon. We still expect some work on language design, but for its priority to be
-driven largely as a function of being in the critical path of some aspect of our
-implementation work.
+The second is focused on moving from a vague direction of "we will have a memory
+safe dialect of Carbon that is a reasonable default", to a specific and concrete
+design. We want to be able to illustrate exactly what it will look like to
+migrate existing unsafe C++ to Carbon (possibly at large scale), and then begin
+incrementally adopting and integrating memory safety into that otherwise unsafe
+Carbon codebase.
 
-## Key results in 2024
+Achieving these should dramatically reduce the risk around Carbon, especially in
+environments where memory safety is increasingly a necessary part of any future
+software development plans. They will also move the project much closer to our
+0.1 milestone.
 
-### Carbon's toolchain implements enough of the language to build realistic code
+## Key results in 2025
 
-This goal is not necessarily about complete support for the entire language
-design, but rather enough of it to support building the realistic and
-interesting Carbon code that interoperates with C++.
+### Access most non-template C++ APIs in Carbon
 
-Here are some example language features that we think are key to success, but
-this is far from an exhaustive list:
+Beyond excluding templates, this excludes coroutines, and any aspects that
+require accessing Carbon types in C++ such as templates with Carbon types as
+template arguments.
 
--   Imports and a working [prelude] (the earliest stages of a standard library)
--   Operator overloading and dispatch for expressions
--   Generic types and functions
--   Templates (likely only partial support and focused on interop use cases)
+This result includes both the implementation in the toolchain and the underlying
+design underpinning this implementation. It also includes implementation and
+design work on necessary Carbon language features that underpin the interop
+provided.
 
-[prelude]: /docs/design#name-lookup-for-common-types
+### Access non-generic Carbon APIs in C++
 
-### Carbon's toolchain can build C++ code
+This excludes generics to make the scope more tractable, but this remains a bit
+of a stretch goal for 2025, and how much progress we make will depend on how
+many unexpected difficulties we encounter getting the other direction to work,
+and any other delays.
 
-We need the toolchain to be able to build C++ code as if it were Clang in order
-to build the C++ code that Carbon is interoperating with. This isn't about
-building anything new or novel, but about packaging and exposing Clang for this
-purpose.
+### Detailed safety strategy update, including expected tradeoffs and prioritization
 
-### Carbon's toolchain works with existing, simple C++ build systems
+We haven't been focused on the safe side of Carbon for several years and will
+need to refresh our safety strategy to reflect the current plan, as well as
+expanding and making it more detailed to support building our initial memory
+safety design.
 
-We should be able to drop Carbon's toolchain into at least simple `Makefile` or
-CMake build systems as a replacement for the C++ toolchain and provide a Carbon
-toolchain. This doesn't include supporting everything or even moderately complex
-builds; only the simplest of builds using these build systems need to work at
-first.
+### Design for compile-time temporal and mutation memory safety
 
-### Carbon has a design and toolchain implementation of basic C++ interop
+We expect our memory safety story for temporal memory safety to at the highest
+level follow the direction of Rust, using the type system to ensure compile-time
+guarantees of safety without the runtime overhead of garbage collection or
+reference counting. We want our design here to cover both temporal and mutation
+safety. While the exact level of safety and the tradeoffs we're willing to
+accept will be part of updating our safety strategy, at a fundamental level we
+need to fully address the security requirements on memory safety, much like
+other modern languages including Swift, Kotlin, Go, or Rust. A significantly
+lower security bar won't be acceptable for the expected users of safe Carbon.
 
-Our end goal is to compile a minimal but non-trivial example of bi-directionally
-mixing C++ and Carbon code such as our main example and run it successfully.
-However, completing everything involved in this example isn't expected to be
-realistic by the end of the year. We expect to work towards this example and in
-rough priority order across the following interop features and all the Carbon
-features they depend on:
+### Give talks at 2-3 conferences about Carbon topics, expanding our audience
 
--   Calling C++ functions from Carbon.
--   Importing concrete C++ types as Carbon types.
--   (stretch) Using Carbon generics with a C++ type in Carbon.
--   (stretch) Calling Carbon functions from C++.
--   (stretch) Importing concrete Carbon types into C++.
+Beyond continuing to share details about Carbon with the open source and C++
+communities, we also want to expand our audience reach in 2025. We want to give
+talks at a conference in the Asia/Pacific region, and at a conference in the
+broader open source world beyond LLVM and C++ specific conferences.
 
-### Give talks at 2-3 conferences covering 3-4 different Carbon topics
-
-We want to continue to engage with the external C++ community as the Carbon
-toolchain becomes a more real and complete toolchain. We specifically want to
-share when interop becomes something people can experiment with and explore.
-
-### Start building our initial tutorial and introductory material
-
-Because of the nature of Carbon's experiment, the tutorial and introductory
-material won't be focused on typical teaching of the language to general
-developers. Instead, it will be focused on enabling C++ developers to start
-evaluating specific aspects of Carbon for interoperating with existing C++
-codebases.
-
-We only expect to _start_ building this material in 2024. We want to learn what
-any critical gaps are for folks to start evaluating C++ interop and how best to
-close them going into 2025.
-
-## Beyond 2024
+## Beyond 2025
 
 Longer term goals are hard to pin down and always subject to change, but we want
 to give an idea of what kinds of things are expected at a high level further out
-in order to illustrate how the goals and priorities we have in 2024 feed into
+in order to illustrate how the goals and priorities we have in 2025 feed into
 subsequent years.
 
-### Potential 2025 goals: ship a working [0.1 language] for evaluation
+### Potential 2026 goals: ship a working [0.1 language] for evaluation
 
 [0.1 language]:
     /docs/project/milestones.md#milestone-01-a-minimum-viable-product-mvp-for-evaluation
 
-As we adjust our schedule and roadmap to reflect the realistic rate of progress,
-the _earliest_ it seems feasible to have everything we need to evaluate the 0.1
-language is 2025. We're starting to be optimistic in 2024 that we'll be able to
-hit this in 2025, but ultimately this remains a lower bound. As we progress, we
-may discover things that push the schedule out further. That is the nature of an
-experimental project like Carbon.
+Because we are adding a design for memory safety to our 0.1 milestone, we are
+also expecting to push it out by at least a year. Shipping 0.1 in 2026 will be a
+very ambitious goal and may not be possible, but the end of 2026 is now the
+_soonest_ that 0.1 could realistically be ready to ship.
 
 We expect that once we reach this milestone the community will be able to start
 realistically evaluating Carbon as a C++ successor language. Of course, this
 evaluation will take some time.
 
-### Potential 2026-2027 goals: finish [0.2 language], stop experimenting
+### Potential 2027-2028 goals: finish [0.2 language], stop experimenting
 
 [0.2 language]:
     /docs/project/milestones.md#milestone-02-feature-complete-product-for-evaluation
@@ -159,7 +142,7 @@ Some concrete goals that might show up in this time frame:
 -   Create a foundation or similar organization to manage the Carbon project,
     separate from any corporate entities that fund work on Carbon.
 
-### Potential goals _beyond_ 2027: ship [1.0 language] & organization
+### Potential goals _beyond_ 2028: ship [1.0 language] & organization
 
 [1.0 language]:
     /docs/project/milestones.md#milestone-10-no-longer-an-experiment-usable-in-production

proposals/p4880.md

@@ -0,0 +1,141 @@
+# Safety milestones and a 2025 roadmap
+
+<!--
+Part of the Carbon Language project, under the Apache License v2.0 with LLVM
+Exceptions. See /LICENSE for license information.
+SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+-->
+
+[Pull request](https://github.com/carbon-language/carbon-lang/pull/4880)
+
+<!-- toc -->
+
+## Table of contents
+
+-   [Abstract](#abstract)
+-   [Proposal](#proposal)
+-   [Retrospective on 2024](#retrospective-on-2024)
+
+<!-- tocstop -->
+
+## Abstract
+
+We propose updating our milestones to accelerate design and implementation of
+memory safety in Carbon, and a roadmap for 2025 reflecting this change. We also
+provide a retrospective for 2024's progress.
+
+## Proposal
+
+Recently, we have seen several new contributors join the Carbon project and a
+corresponding up-tick in activity across the project. We are all really excited
+both by the new folks participating and seeing things move faster.
+
+However, we have also had some good conversations with potential users and other
+interested parties about the project and our roadmap. Everyone has been very
+happy to see the rapid progress on a realistic toolchain, but we have also
+gotten even stronger feedback than previously around memory safety, and we
+propose updating our plans to better address that feedback.
+
+For the past two years we have been working on the C++-interop focused aspect of
+Carbon, and deferring the work to build out a strong memory safety story. While
+there was strong interest in memory safety, and for many an essential
+requirement long term, it seemed reasonable to tackle first interop, and then
+look at safety. This was heavily informed both by having a very small set of
+contributors and a desire to ship an 0.1 milestone with C++ interop as quickly
+as possible.
+
+Now we both have a larger set of contributors, and even stronger and more
+specific feedback asking for a concrete and detailed design for memory safety in
+Carbon sooner rather than later. We propose as a consequence to add a concrete
+_design_ for memory safe Carbon to our 0.1 milestone, and begin working on this
+in parallel to the toolchain in 2025.
+
+Our projected timeline for reaching 0.1 was historically based on growing the
+set of contributors while keeping all of them focused on the smaller prior 0.1
+milestone. Adding a memory safety design to our target for 0.1 will make it
+impossible to ship in 2025 even with the larger contributor base. We are
+shifting our target from 2025 for 0.1 to, at the soonest, the end of 2026; and
+as always with these projections, they should be understood as a lower bound.
+
+With these updated milestones, we propose the following goals for 2025:
+
+-   Complete design and implementation for remaining Carbon features needed for
+    non-template C++ interop
+-   Complete most of the interop layer for non-template Carbon ↔ C++,
+    prioritizing calling/using C++ APIs from Carbon, and then exposing Carbon
+    APIs back to C++.
+-   Full support for compiling C++ code with the Carbon toolchain using Clang
+-   Update our safety strategy, and establish a detailed strategy for memory
+    safety
+-   Design for compile-time & type-system based temporal and mutation safety
+
+Both the C++ interop and the memory safety designs are expected to have
+dependencies that we will need to prioritize in 2025. The C++ interop will
+depend on finishing the implementation of many parts of the Carbon language in
+our toolchain, and the safety designs will build on top of the existing Carbon
+type-system design, including refinements to it from our implementation
+experience.
+
+As with previous years, we will also continually share our progress and what we
+learn with the broader open source and C++ communities.
+
+## Retrospective on 2024
+
+Our [roadmap for 2024] was an ambitious pivot to focus heavily on implementing
+the Carbon language in a realistic compiler and toolchain. At a high level, this
+pivot was _very_ successful. Carbon's compiler has made massive strides over the
+year, and all of the active contributors to Carbon have thoroughly ramped up on
+our implementation and are making meaningful contributions to it.
+
+[roadmap for 2024]:
+    https://github.com/carbon-language/carbon-lang/blob/10189bbb78db7b143a6d9d62797fc9698363fe4d/docs/project/roadmap.md
+
+Going into 2024, Carbon's compiler didn't have any support for generics,
+importing, expression categories, debug info, mangling, a prelude, or so many
+other things now in flight. Constant evaluation didn't know about aggregates,
+and classes couldn't be initialized at all. We had no examples, much less
+working ones. By the end of 2024, with a surprisingly minimal number of hacks,
+it was possible to solve much of Advent of Code! The progress here has been
+phenomenal, and we made awesome strides towards our goal of a working toolchain.
+
+Looking in more detail at our key results for 2024:
+
+-   [Carbon's toolchain implements enough of the language to build realistic code](https://github.com/carbon-language/carbon-lang/blob/10189bbb78db7b143a6d9d62797fc9698363fe4d/docs/project/roadmap.md#carbons-toolchain-implements-enough-of-the-language-to-build-realistic-code)
+    -   A smashing success, overall.
+    -   We have imports, a working prelude, generic types and functions.
+    -   We have many of the building blocks of dispatching through interfaces
+        for operator overloading, but there are still a few gaps left. Despite
+        the gaps, we're actually using our generic building blocks effectively,
+        including with a working `Core.Int` generic integer type that backs
+        `i32` instead of a hard coded type!
+    -   We didn't get to the more stretch parts of this goal like templates
+        though.
+-   [Carbon's toolchain can build C++ code](https://github.com/carbon-language/carbon-lang/blob/10189bbb78db7b143a6d9d62797fc9698363fe4d/docs/project/roadmap.md#carbons-toolchain-can-build-c-code)
+    -   We have Clang integrated into the toolchain!
+    -   It works reasonably well for C++ code without system `#include`s.
+    -   But we still need to get some critical headers and other data that Clang
+        depends on to get this fully working with system `#include`s.
+-   [Carbon's toolchain works with existing, simple C++ build systems](https://github.com/carbon-language/carbon-lang/blob/10189bbb78db7b143a6d9d62797fc9698363fe4d/docs/project/roadmap.md#carbons-toolchain-works-with-existing-simple-c-build-systems)
+    -   Until we have all the system `#include` support work done, we can't drop
+        our toolchain into a C++ build system. =/ This one ended up largely not
+        landing this year.
+    -   We did end up integrating Carbon's toolchain into very simple Bazel
+        build rules that we use to continuously build and test a collection of
+        example Carbon code. This made sure the compilation model does work in a
+        realistic toolchain situation for Carbon code.
+-   [Carbon has a design and toolchain implementation of basic C++ interop](https://github.com/carbon-language/carbon-lang/blob/10189bbb78db7b143a6d9d62797fc9698363fe4d/docs/project/roadmap.md#carbon-has-a-design-and-toolchain-implementation-of-basic-c-interop)
+    -   This was too ambitious of a goal, but we actually got very close. The
+        first importing of C++ headers (but without doing anything) has already
+        landed, so we weren't too far away from the most basic parts of this.
+-   [Give talks at 2-3 conferences covering 3-4 different Carbon topics](https://github.com/carbon-language/carbon-lang/blob/10189bbb78db7b143a6d9d62797fc9698363fe4d/docs/project/roadmap.md#give-talks-at-2-3-conferences-covering-3-4-different-carbon-topics)
+    -   We had a really strong year delivering talks at both C++ and LLVM
+        conferences: 5 talks on different topics plus a panel session.
+-   [Start building our initial tutorial and introductory material](https://github.com/carbon-language/carbon-lang/blob/10189bbb78db7b143a6d9d62797fc9698363fe4d/docs/project/roadmap.md#start-building-our-initial-tutorial-and-introductory-material)
+    -   We got an important start here with newsletters every other month,
+        packaged pre-releases, and even some "first look" courses introducing
+        people to Carbon.
+    -   However, tutorial material still seems somewhat far away.
+
+Overall, we achieved the majority of what we set out to for 2024. While our
+roadmaps are always a bit ambitious to push us, this year didn't seem excessive
+and is likely to reflect roughly how ambitious we want to be in our roadmaps.