Focus implementation effort on the toolchain (#3532)

Proposal to focus implementation effort for the next 1-2 years on the
Carbon
toolchain instead of the explorer. This will impact the explorer in a
few ways:

- We will keep the explorer's code in place, building, and passing its
basic
    tests. It can remain a good baseline for exploring Carbon's language
    features.
- We won't prioritize expanding the explorer's coverage of Carbon
features or
other improvements -- it is good enough for what we need until the
toolchain
    catches up.
- We will stop actively fuzzing and expanding test coverage for the
explorer.
- Eventually, when we want to resume work on the explorer, we'll
evaluate the
best platform to build on -- the current explorer codebase or on top of
the
    toolchain's semantic IR.

Also tries to update the core readme and contributing docs to reflect
this.

CONTRIBUTING.md

@@ -16,7 +16,6 @@ SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
         -   [Comment on proposals](#comment-on-proposals)
         -   [Contribute design ideas to Carbon](#contribute-design-ideas-to-carbon)
     -   [Contributing to the language implementation](#contributing-to-the-language-implementation)
-        -   [Experiment with Carbon and report issues (no code)](#experiment-with-carbon-and-report-issues-no-code)
         -   [Review and comment on Pull Requests (no code)](#review-and-comment-on-pull-requests-no-code)
         -   [Implement Carbon's design](#implement-carbons-design)
         -   [Triage, analyze or address bugs](#triage-analyze-or-address-bugs)
@@ -94,17 +93,6 @@ early, before even writing a proposal, and the process explains how to do that.
 
 ### Contributing to the language implementation
 
-#### Experiment with Carbon and report issues (no code)
-
-You can experiment with the current implementation of the Carbon language using
-the [online version](https://carbon.compiler-explorer.com/) of Carbon Explorer.
-The state of the design implementation for Explorer is documented on the
-["Are we explorer yet"](https://github.com/carbon-language/carbon-lang/wiki/Are-we-explorer-yet%3F)
-wiki page.
-
-If you uncover a bug or implementation problem, you can
-[report an issue on GitHub](https://github.com/carbon-language/carbon-lang/issues/new/choose).
-
 #### Review and comment on Pull Requests (no code)
 
 Helping with
@@ -113,18 +101,11 @@ a good way to provide feedback, while getting a acquainted with the code base.
 
 #### Implement Carbon's design
 
-The implementation of the Carbon language design takes place in two distinct
-locations:
-
--   the [Carbon Explorer](/explorer/) (see Carbon
-    [Explorer issues](https://github.com/carbon-language/carbon-lang/issues?q=is%3Aissue+is%3Aopen+label%3Aexplorer)),
-    and
--   the [Carbon toolchain](/toolchain/) (see Carbon
-    [toolchain issues](https://github.com/carbon-language/carbon-lang/issues?q=is%3Aissue+is%3Aopen+label%3Atoolchain))
+The implementation of the Carbon language design is currently focused on the
+[Carbon toolchain](/toolchain/) (see Carbon
+[toolchain issues](https://github.com/carbon-language/carbon-lang/issues?q=is%3Aissue+is%3Aopen+label%3Atoolchain))
 
-The Carbon Explorer is generally simpler to contribute to, while the toolchain
-implements the compiler and tools needed for the language. **Some issues are
-also marked as
+**Some issues are also marked as
 ["good first issues"](https://github.com/carbon-language/carbon-lang/labels/good%20first%20issue)**.
 These are intended to be a good place to start contributing.
 

README.md

@@ -149,11 +149,14 @@ and the language:
 -   A prototype interpreter demo that can both run isolated examples and gives a
     detailed analysis of the specific semantic model and abstract machine of
     Carbon. We call this the [Carbon Explorer](/explorer/).
+-   Early components of a full [compiler and toolchain](/toolchain/) that can
+    compile Carbon (and eventually C++ code as well) into standard executable
+    code. This is where most of our current implementation efforts are directed.
 
 If you're interested in contributing, we would love help
 [completing the 0.1 language designs](/docs/project/roadmap.md#complete-design-coverage-of-the-01-languages-necessary-features),
 and
-[completing the Carbon Explorer implementation of this design](/docs/project/roadmap.md#complete-01-language-implementation-coverage-in-the-carbon-explorer).
+[completing the Carbon Toolchain implementation of this design](/docs/project/roadmap.md#a-toolchain-that-can-build-a-minimal-mixed-c-and-carbon-program).
 We are also currently working to get more broad feedback and participation from
 the C++ community. Beyond that, we plan to prioritize C++ interoperability and a
 realistic toolchain that implements the 0.1 language and can be used to evaluate
@@ -266,23 +269,54 @@ semantics onto C++ such as Rust-inspired
 
 ## Getting started
 
-As there is no compiler yet, to try out Carbon, you can use the Carbon explorer
-to interpret Carbon code and print its output. You can try it out immediately at
+To try out Carbon, you can use the Carbon explorer to interpret Carbon code and
+print its output. You can try it out immediately at
 [compiler-explorer.com](http://carbon.compiler-explorer.com/).
 
-To build the Carbon explorer yourself, you'll need to install dependencies
-(Bazel, Clang, libc++), and then you can run:
+Because Carbon is an early, experimental project we don't yet have releases you
+can download and try out locally, you'll instead need to build any tools
+yourself from source. We expect to have packaged releases you can try out when
+we reach our
+[0.1 milestone](/docs/project/milestones.md#milestone-01-a-minimum-viable-product-mvp-for-evaluation).
+
+If you do want to try out Carbon locally, you'll need to install our
+[build dependencies](/docs/project/contribution_tools.md#setup-commands) (Bazel,
+Clang, LLD, libc++) and check out the Carbon repository, for example on Debian
+or Ubuntu:
 
 ```shell
+# Update apt.
+sudo apt update
+
+# Install tools.
+sudo apt install \
+  bazel \
+  clang \
+  libc++-dev \
+  lld
+
 # Download Carbon's code.
 $ git clone https://github.com/carbon-language/carbon-lang
 $ cd carbon-lang
+```
+
+Then you can build and run the explorer:
 
+```shell
 # Build and run the explorer.
 $ bazel run //explorer -- ./explorer/testdata/print/format_only.carbon
 ```
 
-For complete instructions, including installing dependencies, see our
+And you can try out our toolchain which has a very early-stage compiler for
+Carbon:
+
+```shell
+# Build and run the toolchain's help to get documentation on the command line.
+$ bazel run //toolchain/driver:carbon -- help
+```
+
+For complete instructions, including installing dependencies on various
+different platforms, see our
 [contribution tools documentation](/docs/project/contribution_tools.md).
 
 Learn more about the Carbon project:
@@ -290,6 +324,7 @@ Learn more about the Carbon project:
 -   [Project goals](/docs/project/goals.md)
 -   [Language design overview](/docs/design)
 -   [Carbon Explorer](/explorer)
+-   [Carbon Toolchain](/toolchain)
 -   [FAQ](/docs/project/faq.md)
 
 ## Conference talks
@@ -337,8 +372,6 @@ You can also directly:
 -   [Contribute to the language design](CONTRIBUTING.md#contributing-to-the-language-design):
     feedback on design, new design proposal
 -   [Contribute to the language implementation](CONTRIBUTING.md#contributing-to-the-language-implementation)
-    -   [Carbon Explorer](/explorer/): bug report, bug fix, language feature
-        implementation
     -   [Carbon Toolchain](/toolchain/), and project infrastructure
 
 You can **check out some

docs/project/contribution_tools.md

@@ -38,7 +38,7 @@ These commands should help set up a development environment on your machine.
 
 ### Debian or Ubuntu
 
-```
+```shell
 # Update apt.
 sudo apt update
 
@@ -72,7 +72,7 @@ bazel test //...:all
 
 ### macOS
 
-```
+```shell
 # Install Hombrew.
 /bin/bash -c "$(curl -fsSL \
   https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

proposals/p3532.md

@@ -0,0 +1,177 @@
+# Focus implementation effort on the toolchain
+
+<!--
+Part of the Carbon Language project, under the Apache License v2.0 with LLVM
+Exceptions. See /LICENSE for license information.
+SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+-->
+
+[Pull request](https://github.com/carbon-language/carbon-lang/pull/3532)
+
+<!-- toc -->
+
+## Table of contents
+
+-   [Abstract](#abstract)
+-   [Problem](#problem)
+-   [Background](#background)
+-   [Proposal](#proposal)
+-   [Details](#details)
+-   [Alternatives considered](#alternatives-considered)
+    -   [Do nothing](#do-nothing)
+    -   [Front-load porting the functionality and output](#front-load-porting-the-functionality-and-output)
+
+<!-- tocstop -->
+
+## Abstract
+
+Proposal to focus implementation effort for the next 1-2 years on the Carbon
+toolchain instead of the explorer. This will impact the explorer in a few ways:
+
+-   We will keep the explorer's code in place, building, and passing its basic
+    tests. It can remain a good baseline for exploring Carbon's language
+    features.
+-   We won't prioritize expanding the explorer's coverage of Carbon features or
+    other improvements -- it is good enough for what we need until the toolchain
+    catches up.
+-   We will stop actively fuzzing and expanding test coverage for the explorer.
+-   Eventually, when we want to resume work on the explorer, we'll evaluate the
+    best platform to build on -- the current explorer codebase or on top of the
+    toolchain's semantic IR.
+
+## Problem
+
+Carbon is still a relatively small project, and we need to avoid spreading
+ourselves out too much over too many efforts. Currently, we're maintaining two
+implementation codebases -- the Carbon Explorer and the Carbon Toolchain. While
+historically these have served importantly different needs of the project, the
+current state has changed. We're now moving slower as a consequence of spreading
+out our energy across both.
+
+## Background
+
+Originally, the Carbon Explorer served two major purposes:
+
+1. A high-level or "abstract machine" executable semantic model for the design
+   of the language.
+
+2. A rapid prototyping platform with a generated parser and maximally simple &
+   traditional internal architecture (ASTs, etc.).
+
+The first purpose and use case remains extremely important and something that we
+should support. However, long-term it may make more sense to build on the same
+core internal representation as the toolchain in order to avoid duplicated
+effort and maximize its utility. We'll understand the tradeoffs there better
+once the toolchain is similarly feature complete.
+
+The second use case is no longer critical. At the time, the toolchain was
+nascent and using a highly experimental architecture with many unknowns. It was
+hard to be confident it would work at all, much less add a feature to it. We
+also had a large number of design features for Carbon that needed to have some
+form of implementation experience in order to validate the designs themselves.
+
+The current state of the project is very different. The driving purpose of
+prototyping many of the core language features, from inheritance, to generics,
+to expression category, has been achieved. While there are large designs that
+are not yet in the explorer, we have proven out the most critical components and
+gotten critical implementation feedback.
+
+The toolchain is also rapidly maturing and its core architecture is holding up
+well. While implementing a new feature on top of the toolchain's architecture is
+significantly more expensive than on the explorer's architecture, it isn't
+starting from nothing and the most uncertain aspects of the core architecture
+and design have become concrete. It is also increasingly necessary for us to get
+features implemented here so that we can evaluate them in a realistic
+compilation context and integrate them with C++ interop which we only anticipate
+to be able to build in the toolchain architecture.
+
+The toolchain has also developed a Semantics IR that is an especially
+interesting potential platform for building support for the executable semantics
+use case. When the toolchain reaches feature coverage and maturity, it seems
+important to carefully consider whether that's the best approach to take, and
+what the tradeoffs are there.
+
+## Proposal
+
+We should focus all of our efforts for the next 1-2 years on the toolchain, in
+order to make as much progress as possible. This in turn is likely to give us
+the best evaluation of the Carbon experiment in the shortest time frame.
+
+We should keep the Explorer's code in place, building, and passing its basic
+regression tests because the built artifacts of the Explorer remain really
+valuable given its coverage of our design's feature sets. But we shouldn't take
+on new work, or drive significant fuzzing, refactoring, or code cleanups in the
+explorer.
+
+We should also explicitly consider re-building the core functionality of the
+Explorer's abstract machine semantic execution on top of the toolchain's
+Semantics IR model. If this is the right approach given all the tradeoffs, it
+should still leverage the existing work done and lessons learned on the Explorer
+for how to model these semantics and how to render them in an understandable way
+for users. This is another reason why we should keep all of the code in place,
+building and passing its tests, until we complete this. While the code itself is
+unlikely to just directly port across in this way, having the artifacts ensures
+we'll have a good point of comparison and reference for the ideas and designs.
+We also may find that it isn't the right direction, and instead resume work on
+top of the explorer's model directly.
+
+## Details
+
+The short term practical details of this proposal are essentially:
+
+-   Update all of our documentation to direct contributors to the toolchain
+    rather than the Explorer.
+-   Document that the explorer's code is largely an archive and we're not
+    planning to do significant bug fixes, much less feature development in it.
+-   Document that we plan to eventually rebuild the explorer's core
+    functionality on top of the toolchain's semantic model, once that model is
+    sufficiently feature complete. And that we will _not_ remove the explorer's
+    code until that replacement is ready.
+-   Close the various bugs open for working on the explorer with links to this
+    proposal & documentation updates.
+-   Close open PRs and new PRs for the explorer linking to this proposal for
+    details. Note that closed PRs aren't deleted and so we can revisit them if
+    our priorities change and they end up remaining relevant.
+-   Update the issue form to make it clear that we're not planning to do more
+    work on the explorer codebase as-is.
+
+Long term, we need to plan to figure out how we want to pursue the core
+functionality of the explorer, whether in its current implementation or on top
+of the toolchain's Semantic IR model. Even if there isn't much code that can be
+directly reused, we should still heavily learn from and incorporate the relevant
+ideas of the design of the explorer's abstract machine model, output rendering,
+and test suite.
+
+## Alternatives considered
+
+### Do nothing
+
+We could simply leave things in the status-quo. However, this has the serious
+downside that the contributors to Carbon are sufficiently stretched currently
+that they are foregoing work on the explorer already. In some ways, at the
+current size of the project, it isn't clear that we can sustain the status quo
+in practice even if we don't update our documentation.
+
+### Front-load porting the functionality and output
+
+We could immediately try to port the functionality and output so that the
+explorer tooling and workflow is immediately available on top of the toolchain's
+semantic representation.
+
+Unfortunately there is a significant amount of implementation work left for the
+toolchain to catch up in terms of feature completeness with the explorer. We
+need to complete the functionality before we can really pursue this path. And
+this proposal is trying to maximize how much energy the project can devote to
+that.
+
+Beyond this fundamental limit, the current project roadmap and priorities focus
+on getting C++ interop working with the toolchain above expanding functionality
+of the explorer. As a consequence, it seems better to focus on that priority in
+the toolchain and to revisit rebuilding the explorer on top of the toolchain's
+semantic model later, at least after the main interop goals are achieved. At
+that point, it is also likely that most if not all of the feature gaps will have
+been closed.
+
+The proposed approach to essentially archiving the explorer code seems like the
+best way to align the effort and energy of the project contributors with these
+priorities.