carbon-lang/docs/design/lexical_conventions/README.md

Lexical conventions

Table of contents

• Lexical elements

Lexical elements

The first stage of processing a source file is the division of the source file into lexical elements.

A lexical element is one of the following:

• a maximal sequence of whitespace characters
• a word
• a literal:
  • a numeric literal
  • a string literal
• a comment
• TODO: operators ...

The sequence of lexical elements is formed by repeatedly removing the longest initial sequence of characters that forms a valid lexical element.