carbon-lang/toolchain/docs/lex.md

Lex

Table of contents

• Overview
• Bracket matching
• Alternatives considered

Overview

Lexing converts input source code into tokenized output. Literals, such as string literals, have their value parsed and form a single token at this stage.

Bracket matching

The lexer handles matching for (), [], and {}. When a bracket lacks a match, it will insert a "recovery" token to produce a match. As a consequence, the lexer's output should always have matched brackets, even with invalid code.

While bracket matching could use hints such as contextual clues from indentation, that is not yet implemented.

Alternatives considered

• Bracket matching in parser