Common Lexical Translations

1. Introduction

The Common Lexical Translations specification a lexical translation, that is, it defines translations rules for how an input sequence of Unicode code points is translated into an output sequence of transformed and classified sequences of Unicode code points called words. The rules for provided by the Common Lexical Translations specification are reused by other specifications within Michael Heilmann’s Arcadia.

The rules layed out here are described in terms of the concepts and notations of the Context-Free Grammars specification (see https://michaelheilmann.com/specifications/context-free-grammars for more information).

A grammar incorporating these rules must have the set of all Unicode code points as its input alphabet. Furthermore, the grammar must ensure that ambiguities are resolved.

2 Standard Profile

The Full Profile contains all possible rules. Other profiles may be added in future versions. Language designers are encouraged to create their own profiles.

2.1 word

The word word is defined by

2.2 whitespace

The word whitespace is defined by

2.3 line terminator

The word Lexical.LineTerminator is defined by

2.4 comments

The language using the Common Lexical Specification may use both single-line comments and multi-line comments. A Lexical.Comment is either a single_line_comment or a Lexical.MultiLineComment. Lexical.MultiLineComment is defined by

A Lexical.SingleLineComment starts with two solidi. It extends to the end of the line. Lexical.SingleLinecomment is defined by

The Lexical.LineTerminator is not considered as part of the comment text.

A Lexical.MultiLineComment is opened by a solidus and an asterisk and closed by an asterisk and a solidus. Lexical.MultiLineComment is defined by

The #2f #2a and #2a #2f sequences are not considered as part of the comment text.

This implies:

• #2f #2f has no special meaning either comment.
• #2f #2a and #2a #2f have no special meaning in single-line comments.
• Multi-line comments do not nest.

2.5 parentheses

The words Lexical.LeftParenthesis and Lexical.RightParenthesis, respectively, are defined by

2.6 curly brackets

The words Lexical.LeftCurlyBracket and Lexical.RightCurlyBracket, respectively, are defined by

2.7 colon

The word Lexical.Colon is defined by

2.8 square brackets

The words Lexical.LeftSquareBracket and Lexica.RightSquareBracket, respectively, are defined by

2.9 comma

The word Lexical.Comma is defined by

2.10 name

The word Lexical.Name is defined by

2.10 number literal

The word Lexical.Number is defined by

2.11 string literal

The word Lexical.String is defined by

In the lexical translation, several transformations are performed upon the word:

• Lexical.DoubleQuote {Lexical.DoubleQuotedStringCharacter} Lexical.DoubleQuote has the leading double quote and the trailing double quote removed.
• Lexical.SingleQuote {Lexical.SingleQuotedStringCharacter} Lexical.SingleQuote has the leading single quote and the trailing single quote removed.
• #5c #5c (double "REVERSE SOLIDUS") are replaced by #5c ("REVERSE SOLIDUS")
• #5c #64 ("REVERSE SOLIDUS" followed by "LATIN SMALL LETTER B") are replaced by #7 ("BELL")>/li>
• #5c #66 ("REVERSE SOLIDUS" followed by "LATIN SMALL LETTER F") are replaced by #c ("FORM FEED")
• #5c #6e ("REVERSE SOLIDUS" followed by "LATIN SMALL LETTER N") are replaced by #a ("LINE FEED")>/li>
• #5c #72 ("REVERSE SOLIDUS" followed by "LATIN SMALL LETTER R") are replaced by #d ("CARRIAGE RETURN")
• #5c #72 ("REVERSE SOLIDUS" followed by "LATIN SMALL LETTER T") are replaced by #9 ("TAB")
• #5c 'u' Lexical.HexadecimalDigit Lexical.HexadecimalDigit Lexical.HexadecimalDigit Lexical.HexadecimalDigit is replaced by the corresponding Unicode code point. That Unicode code point is computed as follows: Let \(a\), \(b\), \(c\), and \(d\) denote the first, second, third, and fourth hexadecimal digit from left to right following 'u'. Then the Unicode code point is given by \[ c := (a \cdot 16^3) + (b*16^2) + (c*16^1) + (d*16^0) \] If \(c\) is outside of the set of Unicode code points, then the lexical translation shall fail.

2.12 boolean literal

The word Lexical.Boolean is defined by

Lexical.Boolean : Lexical.True
Lexical.Boolean : Lexical.False
true : #74 #72 #75 #65
false : #66 #61 #6c #73 #65

Remark: The word Lexical.Boolean is a so called keyword. It takes priority over the Lexical.Name.

2.13 void literal

The word Lexical.Void is defined by

Lexical.Void : #76 #6f # #69 #64

Remark: The word Lexical.Void is a so called keyword. It takes priority over the Lexical.Name.

2.14 decimal digit

The word Lexical.DecimalDigit is defined by

Lexical.DecimalDigit : /* A single Unicode character from the code point range +U0030 to +U0039. */

2.15 hexadecimal digit

The word Lexical.HexadecimalDigit is defined by

Lexical.HexadecimalDigit : /* A single Unicode character from the code point range +U0030 to +U0039, +U0061 to +U007A, U+0041 to U+005A*/

2.16 alphanumeric

The word Lexical.Alphanumeric is reserved for future use.

2.17 period

The word Lexical.Period is defined by

/* #2e is also known as "FULL STOP" */
Lexical.Period : 2e