Common Lexical Translations

1 Introduction

The Common Lexical Translations specification provides reusable lexical categories and translation rules. These rules describe how sequences of Unicode code points are recognized, transformed, and classified as words. Language specifications reuse these rules within Michael Heilmann's Arcadia.

Each consuming language defines its own complete lexical grammar, including its start symbol, word alternatives, and ambiguity-resolution rules. This specification supplies the common building blocks, not a universal word production.

This specification is for readers defining or using a language's lexical grammar. The productions use the concepts and notation in the Context-Free Grammars section of Mathematical Definitions, Notations and Proofs. Read that section for production notation, repetition, optional elements and hexadecimal code-point literals.

A grammar incorporating these rules must have the set of all Unicode code points as its input alphabet. Furthermore, the grammar must ensure that ambiguities are resolved.

A word is a general term for an item recognized by a lexical translation. The common categories do not determine a universal set of words for all languages. Each consuming language shall define its own word production and lexical start symbol, selecting the common categories it uses and adding language-specific categories such as keywords or register notation.

The consuming language shall resolve ambiguities among its selected alternatives and specify which recognized items are emitted, ignored, or significant to its syntax. It shall state any refinement of a common rule explicitly. Reusing a common category does not require accepting every other category listed in this specification.

To use this catalog, select the required categories from Section 2, then consult the consuming language's specification for its complete word grammar and refinements. For example, the Data Definition Language (DDL) lexical translation selects double-quoted strings and defines its own ASCII name profile. Those choices do not restrict other consumers.

The Unicode input alphabet does not mean that every category accepts every Unicode code point. Each production states its own character ranges; the string rules distinguish Unicode scalar values from the broader set of Unicode code points.

2 Productions

This section lists reusable lexical productions and their translation rules. A consuming language selects and combines these productions with its own additions to define its lexical grammar, word alternatives, and start symbol.

2.1 Whitespace

The word Lexical.Whitespace is defined by

/* #9 is also known as "CHARACTER TABULATION" */
Lexical.Whitespace : #9
/* #20 is also known as "SPACE" */
Lexical.Whitespace : #20
/* Legacy misspelled label for the space alternative, retained for compatibility. */
Lexial.Whitespace : #20

2.2 End of line

The word Lexical.EndOfLine is defined by

/* #a is also known as "LINEFEED (LF)" */
/* #d is also known as "CARRIAGE RETURN (CR)" */
Lexical.EndOfLine : #a {#d}
Lexical.EndOfLine : #d {#a}

2.3 Comment

A consuming language may select both single-line and multi-line comments. A Lexical.Comment is either a Lexical.SingleLineComment or a Lexical.MultiLineComment. Lexical.Comment is defined by

Lexical.Comment : Lexical.SingleLineComment
Lexical.Comment : Lexical.MultiLineComment

A Lexical.SingleLineComment begins with two solidus characters (//) and extends to the end of the line. Lexical.SingleLineComment is defined by

/* #2f is also known as SOLIDUS */
Lexical.SingleLineComment :
#2f #2f
/* any sequence of characters except for line_terminator */

The Lexical.LineTerminator is not considered as part of the comment text.

A Lexical.MultiLineComment begins with a solidus and an asterisk (/*) and ends with an asterisk and a solidus (*/). Lexical.MultiLineComment is defined by

/* #2f is also known as SOLIDUS */
/* #2a is also known as ASTERISK */
Lexical.MultiLineComment :
#2f #2a
/* any sequence of characters except except for #2a #2f */
#2a #2f

The opening #2f #2a and closing #2a #2f sequences are not part of the comment text.

Within comment text:

• #2f #2f has no special meaning in either kind of comment.
• #2f #2a and #2a #2f have no special meaning in single-line comments.
• Multi-line comments do not nest.

2.4 Parentheses

The words Lexical.LeftParenthesis and Lexical.RightParenthesis, respectively, are defined by

/* #28 is also known as "LEFT PARENTHESIS" */
Lexical.LeftParenthesis : #28
/* #29 is also known as "RIGHT PARENTHESIS" */
Lexical.RightParenthesis : #29

2.5 Curly brackets

The words Lexical.LeftCurlyBracket and Lexical.RightCurlyBracket, respectively, are defined by

/* #7b is also known as "LEFT CURLY BRACKET" */
Lexical.LeftCurlyBracket : #7b
/* #7d is also known as "RIGHT CURLY BRACKET" */
Lexical.RightCurlyBracket : #7d

2.6 Colon

The word Lexical.Colon is defined by

/* #3a is also known as "COLON" */
Lexical.Colon : #3a

2.7 Square brackets

The words Lexical.LeftSquareBracket and Lexical.RightSquareBracket, respectively, are defined by

/* #5b is also known as "LEFT SQUARE BRACKET" */
Lexical.LeftSquareBracket : #5b
/* #5d is also known as "RIGHT SQUARE BRACKET" */
Lexical.RightSquareBracket : #5d

2.8 Comma

The word Lexical.Comma is defined by

/* #2c is also known as "COMMA" */
Lexical.Comma : #2c

2.9 Name

The word Lexical.Name is defined by

Lexical.Name : {Lexical.Underscore} Lexical.Alphabetic {Lexical.NameSuffixCharacter}
Lexical.Underscore : #5f
Lexical.Alphabetic : /* one ASCII letter from #41 to #5a or #61 to #7a */

/* #41 is also known as "LATIN CAPITAL LETTER A" */
/* #5a is also known as "LATIN CAPITAL LETTER Z" */
/* #61 is also known as "LATIN SMALL LETTER A" */
/* #7a is also known as "LATIN SMALL LETTER Z" */
Lexical.NameSuffixCharacter : /* The unicode characters from #41 to #5a and from #61 to #7a. */

/* #30 is also known as "DIGIT ZERO" */
/* #39 is also known as "DIGIT NINE" */
Lexical.NameSuffixCharacter : /* The unicode characters from #30 to #39. */

/* #5f is also known as "LOW LINE" */
Lexical.NameSuffixCharacter : #5f

The shared name production requires an ASCII letter after any leading underscores. DDL's name profile also permits underscore-only names. That language-specific refinement does not replace the shared production above.

2.10 Number literal

The word Lexical.Number is defined by

Lexical.Number : Lexical.IntegerNumber
Lexical.Number : Lexical.RealNumber
Lexical.Digit : Lexical.DecimalDigit
Lexical.IntegerNumber : [Lexical.Sign] Lexical.Digit {Lexical.Digit}
Lexical.RealNumber : [Lexical.Sign] Lexical.Period Lexical.Digit {Lexical.Digit} [Lexical.Exponent]
Lexical.RealNumber : [Lexical.Sign] Lexical.Digit {Lexical.Digit} [Lexical.Period {Lexical.Digit}] [Lexical.Exponent]
Lexical.Exponent : Lexical.ExponentPrefix [Lexical.Sign] Lexical.Digit {Lexical.Digit}

/* #2b is also known as "PLUS SIGN" */
Lexical.Sign : #2b
/* #2d is also known as "MINUS SIGN" */
Lexical.Sign : #2d
/* #65 is also known as "LATIN SMALL LETTER E" */
Lexical.ExponentPrefix : #65
/* #45 is also known as "LATIN CAPITAL LETTER E" */
Lexical.ExponentPrefix : #45

2.11 String literal

The word Lexical.String is defined by

These reusable productions offer both quote styles. A consuming language may restrict the delimiters or explicitly refine the escape set; DDL selects double-quoted strings only, while other languages specify their own profiles.

Lexical.String : Lexical.SingleQuotedString
Lexical.String : Lexical.DoubleQuotedString

Lexical.DoubleQuotedString : Lexical.DoubleQuote {Lexical.DoubleQuotedStringCharacter} Lexical.DoubleQuote
Lexical.DoubleQuotedStringCharacter : /* any Unicode scalar value except #22, #5c and [#00,#1f] */
Lexical.DoubleQuotedStringCharacter : Lexical.EscapeSequence
Lexical.DoubleQuotedStringCharacter : #5c Lexical.DoubleQuote
/* #22 is QUOTATION MARK. */
Lexical.DoubleQuote : #22

Lexical.SingleQuotedString : Lexical.SingleQuote {Lexical.SingleQuotedStringCharacter} Lexical.SingleQuote
Lexical.SingleQuotedStringCharacter : /* any Unicode scalar value except #27, #5c and [#00,#1f] */
Lexical.SingleQuotedStringCharacter : Lexical.EscapeSequence
Lexical.SingleQuotedStringCharacter : #5c Lexical.SingleQuote
/* #27 is APOSTROPHE. */
Lexical.SingleQuote : #27

/* #5c is REVERSE SOLIDUS; #75 is LATIN SMALL LETTER U. */
Lexical.EscapeSequence : #5c 'u' #7b Lexical.HexadecimalDigit [Lexical.HexadecimalDigit] [Lexical.HexadecimalDigit] [Lexical.HexadecimalDigit] [Lexical.HexadecimalDigit] [Lexical.HexadecimalDigit] #7d
Lexical.EscapeSequence : #5c #5c /* reverse solidus */
Lexical.EscapeSequence : #5c #62 /* b */
Lexical.EscapeSequence : #5c #66 /* f */
Lexical.EscapeSequence : #5c #6e /* n */
Lexical.EscapeSequence : #5c #72 /* r */
Lexical.EscapeSequence : #5c #74 /* t */
Lexical.EscapeSequence : #5c #76 /* v */
Lexical.EscapeSequence : #5c #30 /* 0 */

The reverse solidus must introduce a listed escape; it is not an ordinary unescaped string character. Raw U+0000 through U+001F, including line endings, are forbidden inside strings. The opposite quote character may appear literally: for example, "It's ready" needs no escape for its apostrophe.

Lexical translation removes the delimiters and decodes the contents as follows:

• Lexical.DoubleQuote {Lexical.DoubleQuotedStringCharacter} Lexical.DoubleQuote has the leading double quote and the trailing double quote removed.
• Lexical.SingleQuote {Lexical.SingleQuotedStringCharacter} Lexical.SingleQuote has the leading single quote and the trailing single quote removed.
• \" (#5c #22) in a double-quoted string denotes U+0022 (quotation mark).
• \' (#5c #27) in a single-quoted string denotes U+0027 (apostrophe). This production does not enable single-quoted DDL strings.
• \\ (#5c #5c) denotes U+005C (reverse solidus).
• \b (#5c #62) denotes U+0008 (backspace), not U+0007 (bell).
• \f (#5c #66) denotes U+000C (form feed).
• \n (#5c #6e) denotes U+000A (line feed).
• \r (#5c #72) denotes U+000D (carriage return).
• \t (#5c #74) denotes U+0009 (character tabulation).
• \v (#5c #76) denotes U+000B (line tabulation / vertical tab).
• \0 (#5c #30) denotes U+0000 (NUL). It is part of the decoded text, not a string terminator.
• \u{...} contains one to six hexadecimal digits between { (#7b) and } (#7d), and is replaced by the corresponding Unicode scalar value. Let \(d_1,\ldots,d_n\) be the digit values in source order, with \(1 \leq n \leq 6\). The code point is \[ c := \sum_{i=1}^{n} d_i \cdot 16^{n-i}. \] Lexical translation shall fail if braces are missing, the digit count is outside this range, a digit is not hexadecimal, \(c > \texttt{10FFFF}_{16}\), or \(\texttt{D800}_{16} \leq c \leq \texttt{DFFF}_{16}\). Leading zeros count toward the six-digit limit. Spaces, signs and underscores are not allowed inside the braces. The unbraced \uXXXX form is not supported. For example, \u{41} denotes A and \u{1F3EE} denotes U+1F3EE.

For example, the DDL string "A\tB\n" decodes to A, U+0009, B, U+000A; "\0" has one decoded scalar value, U+0000. Unknown escapes such as \q or \d fail lexical translation.

2.12 Boolean literal

The word Lexical.Boolean is defined by

Lexical.Boolean : Lexical.True
Lexical.Boolean : Lexical.False
Lexical.True : true
Lexical.False : false
true : #74 #72 #75 #65
false : #66 #61 #6c #73 #65

When a consuming language selects these alternatives, true and false are keywords: Lexical.Boolean takes priority over Lexical.Name for those spellings.

2.13 Void literal

The word Lexical.Void is defined by

Lexical.Void : #76 #6f #69 #64

When a consuming language selects this alternative, void is a keyword: Lexical.Void takes priority over Lexical.Name for that spelling.

2.14 Decimal digit

The word Lexical.DecimalDigit is defined by

Lexical.DecimalDigit : /* A single Unicode character from the code point range +U0030 to +U0039. */

2.15 Hexadecimal digit

The word Lexical.HexadecimalDigit is defined by

Lexical.HexadecimalDigit : /* A single Unicode character from U+0030 to U+0039, U+0061 to U+0066, or U+0041 to U+0046. */

2.16 Alphanumeric

The word Lexical.Alphanumeric is reserved for future use.

2.17 Period

The word Lexical.Period is defined by

/* #2e is also known as "FULL STOP" */
Lexical.Period : #2e

2.18 End of input

The word Lexical.EndOfInput is a zero-width condition that is satisfied at the position immediately following the last Unicode code point of the input. It does not consume or represent a Unicode code point.

2.19 Start of input

The word Lexical.StartOfInput is a zero-width condition that is satisfied at the position immediately preceding the first Unicode code point of the input. It does not consume or represent a Unicode code point.

2.20 Line continuation

The word Lexical.LineContinuation is defined by

/* #5c is also known as "REVERSE SOLIDUS" */
Lexical.LineContinuation : #5c Lexical.EndOfLine