Update and expand `README` content and motivation for Carbon (#1270)

Some review feedback from outside the team directly working on Carbon suggested
two pretty significant updates here. First, we didn't do a good job of
motivating Carbon. This takes two parts, first explaining what we'd like to
accomplish with this approach generally, and second explaining why alternative
approaches don't work. A particularly difficult case here is articulating
effectively the difficulties that motivate an approach other than improving C++
incrementally.

Co-authored-by: Jon Meow <jperkins@google.com>
Co-authored-by: josh11b <josh11b@users.noreply.github.com>

README.md

@@ -1,4 +1,4 @@
-# Carbon language
+# Carbon Language: <br/> An experimental successor to C++
 
 <!--
 Part of the Carbon Language project, under the Apache License v2.0 with LLVM
@@ -6,12 +6,10 @@ Exceptions. See /LICENSE for license information.
 SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 -->
 
-## **The Carbon Language project is an experiment exploring a future direction for the C++ programming language.**
-
 <p align="center">
-  <a href="#carbon-goals">Carbon goals</a> |
-  <a href="#carbon-and-c">Carbon and C++</a> |
-  <a href="#take-a-look">Take a look</a> |
+  <a href="#why-build-carbon">Why?</a> |
+  <a href="#language-goals">Goals</a> |
+  <a href="#getting-started">Getting started</a> |
   <a href="#join-us">Join us</a>
 </p>
 
@@ -52,32 +50,114 @@ GitHub will autolink `img`, but won't produce a link when `href="#"`.
 -   Batteries-included approach: compiler, libraries, docs, tools, package
     manager, and more
 
-## Carbon goals
-
-We believe Carbon must support:
-
-1. Performance-critical software
-2. Software and language evolution
-3. Code that is easy to read, understand, and write
-4. Practical safety and testing mechanisms
-5. Fast and scalable development
-6. Modern OS platforms, hardware architectures, and environments
-7. Interoperability with and migration from existing C++ code
+## Why build Carbon?
+
+C++ remains the dominant programming language for performance-critical software,
+with massive and growing codebases and investments. However, it is struggling to
+improve and meet developers' needs outlined above, in no small part due to
+accumulating decades of technical debt. Incrementally improving C++ is
+[extremely difficult](/docs/project/difficulties_improving_cpp.md), both due to
+the technical debt itself and challenges with its evolution process. The best
+way to address these problems is to avoid inheriting the legacy of C or C++
+directly, and instead start with solid language foundations like a
+[modern generics system](#generics), modular code organization, and consistent,
+simple syntax.
+
+Existing modern languages already provide an excellent developer experience: Go,
+Swift, Kotlin, Rust, and many more. **Developers that _can_ use one of these
+existing languages _should_.** Unfortunately, the designs of these languages
+present significant barriers to adoption and migration from C++. These barriers
+range from changes in the idiomatic design of software to performance overhead.
+
+Carbon is fundamentally **a successor language approach**, rather than an
+attempt to incrementally evolve C++. It is designed around interoperability with
+C++ as well as large-scale adoption and migration for existing C++ codebases and
+developers. A successor language for C++ requires:
+
+-   **Performance matching C++**, an essential property for our developers.
+-   **Seamless, bidirectional interoperability with C++**, such that a library
+    anywhere in an existing C++ stack can adopt Carbon without porting the rest.
+-   **A gentle learning curve** with reasonable familiarity for C++ developers.
+-   **Comparable expressivity** and support for existing software's design and
+    architecture.
+-   **Scalable migration**, with some level of source-to-source translation for
+    idiomatic C++ code.
+
+With this approach, we can build on top of C++'s existing ecosystem, and bring
+along existing investments, codebases, and developer populations. There are a
+few languages that have followed this model for other ecosystems, and Carbon
+aims to fill an analogous role for C++:
+
+-   JavaScript → TypeScript
+-   Java → Kotlin
+-   C++ → **_Carbon_**
+
+## Language Goals
+
+We are designing Carbon to support:
+
+-   Performance-critical software
+-   Software and language evolution
+-   Code that is easy to read, understand, and write
+-   Practical safety and testing mechanisms
+-   Fast and scalable development
+-   Modern OS platforms, hardware architectures, and environments
+-   Interoperability with and migration from existing C++ code
 
 While many languages share subsets of these goals, what distinguishes Carbon is
-their combination. For the Carbon project, they are prioritized in the above
-order to help make clear what tradeoffs we intend to make. However, each and
-every goal remains critically important: **Carbon _must_ have excellent C++
-interoperability and migration to be successful.**
-
-Read the [language overview](docs/design/) for more on the language design
-itself, and the [goals](docs/project/goals.md) for more on these values.
+their combination.
+
+We also have explicit _non-goals_ for Carbon, notably including:
+
+-   A stable ABI for the entire language and library
+-   Perfect backwards or forwards compatibility
+
+Our detailed [goals](/docs/project/goals.md) document fleshes out these ideas
+and provides a deeper view into our goals for the Carbon project and language.
+
+## Project status
+
+Carbon is currently an experimental project. We want to better understand
+whether we can build a language that meets our successor language criteria, and
+whether the resulting language can gather a critical mass of interest within the
+larger C++ industry and community.
+
+Currently, we have fleshed out several core aspects of both Carbon the project
+and the language:
+
+-   The strategy of the Carbon Language and project.
+-   An open-source project structure, governance model, and evolution process.
+-   Critical and foundational aspects of the language design informed by our
+    experience with C++ and the most difficult challenges we anticipate. This
+    includes designs for:
+    -   Generics
+    -   Class types
+    -   Inheritance
+    -   Operator overloading
+    -   Lexical and syntactic structure
+    -   Code organization and modular structure
+-   A prototype interpreter demo that can both run isolated examples and gives a
+    detailed analysis of the specific semantic model and abstract machine of
+    Carbon. We call this the [Carbon Explorer](/explorer/).
+
+We are currently focused on getting more broad feedback and participation from
+the C++ community,
+[completing the 0.1 language design](/docs/project/roadmap.md#completing-the-language-design),
+and
+[completing the Carbon Explorer implementation of this design](/docs/project/roadmap.md#demo-implementation-of-core-features-with-working-examples).
+Beyond that, we plan to prioritize C++ interoperability and a realistic
+toolchain that implements the 0.1 language and can be used to evaluate Carbon in
+more detail.
+
+You can see our [full roadmap](/docs/project/roadmap.md) for more details.
 
 ## Carbon and C++
 
-If you're already a C++ developer, Carbon should have a short learning curve. It
-is built out of a consistent set of language constructs that should feel
-familiar. C++ code like this:
+If you're already a C++ developer, Carbon should have a gentle learning curve.
+It is built out of a consistent set of language constructs that should feel
+familiar and be easy to read and understand.
+
+C++ code like this:
 
 <a href="docs/images/snippets.md#c">
 <!--
@@ -88,7 +168,7 @@ https://drive.google.com/corp/drive/folders/1CsbHo3vamrxmBwHkoyz1kU0sGFqAh688
      alt="A snippet of C++ code. Follow the link to read it.">
 </a>
 
-can be mechanically transformed to Carbon, like so:
+corresponds to this Carbon code:
 
 <a href="docs/images/snippets.md#carbon">
 <!--
@@ -99,11 +179,9 @@ https://drive.google.com/corp/drive/folders/1CsbHo3vamrxmBwHkoyz1kU0sGFqAh688
      alt="A snippet of converted Carbon code. Follow the link to read it.">
 </a>
 
-without loss of performance or readability. Yet, translating C++ to Carbon isn't
-necessary; you can call Carbon from C++ without overhead and the other way
-around. You can port your library to Carbon, or write new Carbon on top of your
-existing C++ investment. Carbon won't add a sea of dependencies or slow down
-your performance-critical code. For example:
+You can call Carbon from C++ without overhead and the other way around. This
+means you migrate a single C++ library to Carbon within an application, or write
+new Carbon on top of your existing C++ investment. For example:
 
 <a href="docs/images/snippets.md#mixed">
 <!--
@@ -114,25 +192,82 @@ https://drive.google.com/corp/drive/folders/1CsbHo3vamrxmBwHkoyz1kU0sGFqAh688
      alt="A snippet of mixed Carbon and C++ code. Follow the link to read it.">
 </a>
 
-In terms of safety, any language that can seamlessly call C++ will not be
-perfectly safe in every dimension. However, Carbon's design encourages you to
-use safe constructs where possible.
-
-Ultimately, C++ carries a significant historical legacy, including around ABI
-stability, that constrains its evolution. Carbon is an attempt to set a new
-direction for C++ developers that allows for fast development, flexibility, and
-delight without sacrificing performance, interoperability, and familiarity.
-
 Read more about
-[C++ interop in Carbon](docs/design/interoperability/philosophy_and_goals.md).
-
-## Take a look
-
-Learn more about Carbon's design:
-
--   [Project goals](docs/project/goals.md)
--   [Language overview](docs/design/)
--   [Explorer](explorer/)
+[C++ interop in Carbon](/docs/design/interoperability/philosophy_and_goals.md).
+
+Beyond interoperability between Carbon and C++, we're also planning to support
+migration tools that will mechanically translate idiomatic C++ code into Carbon
+code to help you switch an existing C++ codebase to Carbon.
+
+## Generics
+
+Carbon provides a **modern
+[generics](/docs/design/generics/overview.md#what-are-generics) system** with
+checked definitions, while still **supporting opt-in
+[templates](/docs/design/templates.md) for seamless C++ interop**. Checked
+generics provide several advantages compared to C++ templates:
+
+-   **Generic definitions are fully type checked**, removing the need to
+    instantiate to check for errors and giving greater confidence in code.
+    -   Avoids compile time cost of re-checking the definition for every
+        instantiation.
+    -   When using a definition-checked generic, usage error messages are
+        clearer, directly showing which requirements are not met.
+-   **Enables automatic, opt-in type erasure and dynamic dispatch** without a
+    separate implementation. This can reduce binary size and enables constructs
+    like heterogeneous containers.
+-   **Strong, checked interfaces** mean fewer accidental dependencies on
+    implementation details and a clearer contract for consumers.
+
+Without sacrificing these advantages, **Carbon generics support
+specialization**, ensuring it can fully address performance-critical use cases
+of C++ templates. For more details about Carbon's generics, see their
+[design](/docs/design/generics).
+
+In addition to easy and powerful interop with C++, Carbon templates can be
+constrained and incrementally migrated to checked generics at a fine granularity
+and with a smooth evolutionary path.
+
+## Memory safety
+
+Safety, and especially
+[memory safety](https://en.wikipedia.org/wiki/Memory_safety), remain key
+challenges for C++ and something a successor language needs to address. Our
+initial priority and focus is on immediately addressing important, low-hanging
+fruit in the safety space:
+
+-   Tracking uninitialized states better, increased enforcement of
+    initialization, and systematically providing hardening against
+    initialization bugs when desired.
+-   Designing fundamental APIs and idioms to support dynamic bounds checks in
+    debug and hardened builds.
+-   Having a default debug build mode that is both cheaper and more
+    comprehensive than existing C++ build modes even when combined with
+    [Address Sanitizer](https://github.com/google/sanitizers/wiki/AddressSanitizer).
+
+Once we can migrate code into Carbon, we will have a simplified language with
+room in the design space to add any necessary annotations or features, and
+infrastructure like [generics](#generics) to support safer design patterns.
+Longer term, we will build on this to introduce **a safe Carbon subset**. This
+will be a large and complex undertaking, and won't be in the 0.1 design.
+Meanwhile, we are closely watching and learning from efforts to add memory safe
+semantics onto C++ such as Rust-inspired
+[lifetime annotations](https://discourse.llvm.org/t/rfc-lifetime-annotations-for-c/61377).
+
+## Getting started
+
+You can get started playing with Carbon by checking out the codebase and using
+the Carbon explorer:
+
+```
+TODO: write exact steps to check out, build, and run the explorer on sample Carbon code
+```
+
+Learn more about the Carbon project:
+
+-   [Project goals](/docs/project/goals.md)
+-   [Language design overview](/docs/design)
+-   [Carbon Explorer](/explorer)
 
 ## Join us
 

docs/project/difficulties_improving_cpp.md

@@ -0,0 +1,84 @@
+# Difficulties improving C++
+
+<!--
+Part of the Carbon Language project, under the Apache License v2.0 with LLVM
+Exceptions. See /LICENSE for license information.
+SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+-->
+
+C++ is the dominant programming language for the performance critical software
+our goals prioritize. The most direct way to deliver a modern and excellent
+developer experience for those use cases and developers would be to improve C++.
+
+Improving C++ to deliver the kind of experience developers expect from a
+programming language today is difficult in part because **C++ has decades of
+technical debt** accumulated in the design of the language. It inherited the
+legacy of C, including
+[textual preprocessing and inclusion](https://clang.llvm.org/docs/Modules.html#problems-with-the-current-model).
+At the time, this was essential to C++'s success by giving it instant and high
+quality access to a large C ecosystem. However, over time this has resulted in
+significant technical debt ranging from
+[integer promotion rules](https://shafik.github.io/c++/2021/12/30/usual_arithmetic_confusions.html)
+to complex syntax with
+"[the most vexing parse](https://en.wikipedia.org/wiki/Most_vexing_parse)".
+
+**C++ has also prioritized backwards compatibility** including both syntax and
+[ABI](https://en.wikipedia.org/wiki/Application_binary_interface). This is
+heavily motivated by preserving its access to existing C/C++ ecosystems, and
+forms one of the foundations of common Linux package management approaches. A
+consequence is that rather than changing or replacing language designs to
+simplify and improve the language, features have overwhelmingly been added over
+time. This both creates technical debt due to complicated feature interaction,
+and fails to benefit from on cleanup opportunities in the form of replacing or
+removing legacy features.
+
+Carbon is exploring significant backwards incompatible changes. It doesn't
+inherit the legacy of C or C++ directly, and instead is starting with solid
+foundations, like a modern generics system, modular code organization, and
+consistent, simple syntax. Then, it builds a simplified and improved language
+around those foundational components that remains both interoperable with and
+migratable from C++, while giving up transparent backwards compatibility. This
+is fundamentally **a successor language approach**, rather than an attempt to
+incrementally evolve C++ to achieve these improvements.
+
+Another challenge to improving C++ in these ways is the current evolution
+process and direction. A key example of this is the committee's struggle to
+converge on a clear set of high-level and long-term goals and priorities aligned
+with [ours](https://wg21.link/p2137). When [pushed](https://wg21.link/p1863) to
+address
+[the technical debt caused by not breaking the ABI](https://wg21.link/p2028),
+**C++'s process
+[did not reach any definitive conclusion](https://cor3ntin.github.io/posts/abi/#abi-discussions-in-prague)**.
+This both failed to meaningfully change C++'s direction and priorities towards
+improvements rather than backwards compatibility, and demonstrates how the
+process can fail to make directional decisions.
+
+Beyond C++'s evolution direction, the mechanics of the process also make
+improving C++ difficult. **C++'s process is oriented around standardization
+rather than design**: it uses a multiyear waterfall committee process. Access to
+the committee and standard is restricted and expensive, attendance is necessary
+to have a voice, and decisions are made by live votes of those present. The
+committee structure is designed to ensure representation of nations and
+companies, rather than building an inclusive and welcoming team and community of
+experts and people actively contributing to the language.
+
+Carbon has a more accessible and efficient [evolution process](evolution.md)
+built on open-source principles, processes, and tools. Throughout the project,
+we explicitly and clearly lay out our [goals and priorities](goals.md) and how
+those directly shape our decisions. We also have a clear
+[governance structure](evolution.md#governance-structure) that can make
+decisions rapidly when needed. The open-source model enables the Carbon project
+to expand its scope beyond just the language. We will build a holistic
+collection of tools that provide a rich developer experience, ranging from the
+compiler and standard library to IDE tools and more. **We will even try to close
+a huge gap in the C++ ecosystem with a built-in package manager.**
+
+Carbon is particularly focused on a specific set of [goals](goals.md). These
+will not align with every user of C++, but have significant interest across a
+wide range of users that are capable and motivated to evolve and modernize their
+codebase. Given the difficulties posed by C++'s technical debt, sustained
+priority of backwards compatibility, and evolution process, we wanted to explore
+an alternative approach to achieve these goals -- through a
+backwards-incompatible successor language, designed with robust support for
+interoperability with and migration from C++. We hope other efforts to
+incrementally improve C++ continue, and would love to share ideas where we can.

docs/project/goals.md

@@ -152,21 +152,19 @@ exact form this ends up taking.
 
 ## Language goals and priorities
 
-We believe Carbon must support:
-
-1.  Performance-critical software.
-2.  Software and language evolution.
-3.  Code that is easy to read, understand, and write.
-4.  Practical safety and testing mechanisms.
-5.  Fast and scalable development.
-6.  Modern OS platforms, hardware architectures, and environments.
-7.  Interoperability with and migration from existing C++ code.
-
-While many languages share subsets of these goals, what distinguishes Carbon is
-their combination. For the Carbon project, they are prioritized in the above
-order to help make clear what tradeoffs we intend to make. However, each and
-every goal remains critically important: **Carbon _must_ have excellent C++
-interoperability and migration to be successful.**
+We are designing Carbon to support:
+
+-   Performance-critical software
+-   Software and language evolution
+-   Code that is easy to read, understand, and write
+-   Practical safety and testing mechanisms
+-   Fast and scalable development
+-   Modern OS platforms, hardware architectures, and environments
+-   Interoperability with and migration from existing C++ code
+
+Many languages share subsets of these goals, but what distinguishes Carbon is
+their combination. Where it is necessary to make tradeoffs between these goals,
+we intend to prioritize them in this order.
 
 ### Goals in detail
 

proposals/p1270.md

@@ -0,0 +1,130 @@
+# Update and expand `README` content and motivation for Carbon
+
+<!--
+Part of the Carbon Language project, under the Apache License v2.0 with LLVM
+Exceptions. See /LICENSE for license information.
+SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+-->
+
+[Pull request](https://github.com/carbon-language/carbon-lang/pull/1270)
+
+<!-- toc -->
+
+## Table of contents
+
+-   [Problem](#problem)
+-   [Background](#background)
+-   [Proposal](#proposal)
+-   [Details](#details)
+-   [Rationale](#rationale)
+-   [Alternatives considered](#alternatives-considered)
+    -   [Avoid discussion of motivations](#avoid-discussion-of-motivations)
+    -   [Avoid summarizing key language design areas](#avoid-summarizing-key-language-design-areas)
+    -   [Avoid discussing the difficulty of directly and incrementally improving C++](#avoid-discussing-the-difficulty-of-directly-and-incrementally-improving-c)
+
+<!-- tocstop -->
+
+## Problem
+
+Feedback from folks outside of the immediate team working on Carbon surfaced
+both some problems with the exact phrasing of our main README content, but more
+importantly some major _gaps_ in our overall documentation. Specifically, we
+failed to really explain our motivations for building Carbon and why this
+approach might make sense.
+
+Given the significance of the new content and the importance of these specific
+topics, this level of change seems important to go through the proposal process.
+
+## Background
+
+We've been trying to polish and improve the positioning and explanation of
+Carbon to help understand whether it makes sense to shift the project towards
+being a _public_ experiment instead of private one.
+
+## Proposal
+
+This PR includes a significant update to the [`README`](/README.md) content, as
+well as adding [a new document](/docs/project/difficulties_improving_cpp.md) to
+explain the difficulties with incrementally improving C++.
+
+It also tweaks the wording of our goals to try to further reduce confusion.
+
+## Details
+
+See the pull request for the detailed change.
+
+## Rationale
+
+-   [Community and culture](/docs/project/goals.md#community-and-culture)
+    -   We should document clearly our motivations to ensure that aspect of the
+        project remains transparent and clear.
+-   [Software and language evolution](/docs/project/goals.md#software-and-language-evolution)
+    -   Understanding the _motivations_ of the Carbon project will be important
+        for future language evolution efforst.
+
+## Alternatives considered
+
+### Avoid discussion of motivations
+
+We could instead choose to avoid discussion of the project's motivations. This
+has largely been the status-quo prior to this change.
+
+Advantages:
+
+-   Less text.
+-   Fewer opportunities for a misunderstanding to develop.
+
+Disadvantages:
+
+-   Fails to be transparent. We _do_ have motivations, and we can't
+    realistically pretend otherwise.
+-   Because we all _do_ have motivations, failing to write them down will
+    largely result in an inconsistent and lower-quality presentation of them in
+    informal discussions and forums.
+
+### Avoid summarizing key language design areas
+
+This proposal suggests some brief summaries around both
+[generics](/README.md#generics) and [memory safety](/README.md#memory-safety).
+We could instead skip these or only have a brief mention of these.
+
+Advantages:
+
+-   Less text.
+-   May be inaccurate and will run the risk of drifting out of date.
+    -   This was a larger concern previously when for example generics was
+        undergoing more active development.
+
+Disadvantages:
+
+-   Fails to give people an easily consumed entry into some of the really
+    exciting aspects of the language design.
+-   Memory safety at least will likely be an immediate question for readers
+    where we can front-load a well considered answer.
+
+### Avoid discussing the difficulty of directly and incrementally improving C++
+
+Previously we didn't go into details about the difficulties with incrementally
+improving C++ itself that are an essential component of the motivation for
+Carbon. We could stay with that approach.
+
+Advantages:
+
+-   Less text.
+-   A very contentious subject that will have many divergent, well-reasoned, and
+    strongly held positions.
+-   Prior attempts to articulate this have ended up being easily misunderstood
+    or implying significantly more than was intended in a way that actually
+    reduced alignment between different readers rather than building alignment
+    and shared understanding.
+
+Disadvantages:
+
+-   Despite the _difficulty_ of articulating this, it remains _important_. We
+    shouldn't avoid doing the work here merely because it is difficult.
+-   Omitting the discussion of these difficulties runs a risk of seeming
+    disingenuous -- the premise of the Carbon project makes it clear that there
+    is a significant motivation here.
+-   Overcoming the difficulty of articulating these difficulties well and in an
+    understandable form will significantly strengthen the Carbon project's
+    overall motivation and how it can engage with the broader industry.