Common Lexical Specification

This Common Lexical Specification provides definitions of grammar rules being re-used in multiple language specifications on this website. This document consists of three sections: Section 1 defines how programs are encoded on a Byte level. Section 2 provides an introduction into grammars. Section 3 provides the full profile lexical grammar. Section 4 provides information on profiles.

1. Unicode

A program is a sequence of Unicode code points encoded into a sequence of Bytes using an Unicode encoding. In this version, only UTF-8 NOBOM with sequences of length 1 is supported. The Unicode encoding of a particular program must be determined by consumers of this specification.

2. Grammars

This section describes context-free grammars used in this specification to define the lexical and syntactical structure of a language.

2.1 Context-free grammars

A context-free grammar consists of a number of production. Each production has an abstract symbol called a non-terminal as its left-hand side, and a sequence of one or more non-terminal and terminal symbols as its right-hand side. For each grammar, the terminal symbols are drawn from a specified alphabet.

Starting from a sequence consisting of a single distinguished non-terminal, called the goal symbol, a given context-free grammar specifies a language, namely, the set of possible sequences of terminal smbols that can result from repeatedly replacing any non-terminal in the sequence with a right-hand side of a production for which the non-terminal is the left-hand side.

2.3 Lexical grammars

The lexical grammar uses the Unicode code points from the Unicode decoding phase as its terminal symbols. The non-terminals of the lexical grammar start with the prefix Lexical.. It defines a set of productions, starting from the goal symbol Lexical.Word, that describe how sequences of code points are translated into a word.

2.4 Syntactical grammars

The syntactical grammar for the Data Definition Language uses words of the lexical grammar as its terminal symbols. The non-terminals of the syntactical grammar start with the prefix Syntactical.. It defines a set of productions, starting from the goal symbol sentence, that describe how sequences of words are translated into a sentence.

2.5 Grammar notation

Productions are written in fixed width fonts.

A production is defined by its left-hand side, followed by a colon , followed by its right-hand side definition. The left-hand side is the name of the non-terminal defined by the production. Multiple alternating definitions of a production may be defined. The right-hand side of a production consits of any sequence of terminals and non-terminals. In certain cases the right-hand side is replaced by a comment describing the right-hand side. This comment is opened by /* and closed by */.

A terminal is a sequence of Unicode symbols. A Unicode symbol is denoted by a shebang # followed by a hexadecimal number denoting its code point.

The syntax {x} on the right-hand side of a production denotes zero or more occurrences of x.

The syntax [x] on the right-hand side of a production denotes zero or one occurrences of x.

The empty string is denoted by ε.

3 Full Profile Lexical Specification

The lexical grammar describes the translation of Unicode code points into words. The goal non-terminal of the lexical grammar is the Lexical.Word symbol.

3.1 word

The word word is defined by

3.2 whitespace

The word whitespace is defined by

3.3 line terminator

The word Lexical.LineTerminator is defined by

3.4 comments

The language using the Common Lexical Specification may use both single-line comments and multi-line comments. A Lexical.Comment is either a single_line_comment or a Lexical.MultiLineComment. Lexical.MultiLineComment is defined by

A Lexical.SingleLineComment starts with two solidi. It extends to the end of the line. Lexical.SingleLinecomment is defined by

The Lexical.LineTerminator is not considered as part of the comment text.

A Lexical.MultiLineComment is opened by a solidus and an asterisk and closed by an asterisk and a solidus. Lexical.MultiLineComment is defined by

The #2f #2a and #2a #2f sequences are not considered as part of the comment text.

This implies:

• #2f #2f has no special meaning either comment.
• #2f #2a and #2a #2f have no special meaning in single-line comments.
• Multi-line comments do not nest.

3.5 parentheses

The words Lexical.LeftParenthesis and Lexical.RightParenthesis, respectively, are defined by

3.6 curly brackets

The words Lexical.LeftCurlyBracket and Lexical.RightCurlyBracket, respectively, are defined by

3.7 colon

The word Lexical.Colon is defined by

3.8 square brackets

The words Lexical.LeftSquareBracket and Lexica.RightSquareBracket, respectively, are defined by

3.9 comma

The word Lexical.Comma is defined by

3.10 name

The word Lexical.Name is defined by

3.10 number literal

The word Lexical.Number is defined by

3.11 string literal

The word Lexical.String is defined by

3.12 boolean literal

The word Lexical.Boolean is defined by

Remark: The word Lexical.Boolean is a so called keyword. It takes priority over the Lexical.Name.

3.13 void literal

The word Lexical.Void is defined by

Remark: The word Lexical.Void is a so called keyword. It takes priority over the Lexical.Name.

3.14 decimal digit

The word Lexical.DecimalDigit is defined by

3.15 hexadecimal digit

The word Lexical.HexadecimalDigit is defined by

3.16 alphanumeric

The word Lexical.Alphanumeric is reserved for future use.

3.17 period

The word Lexical.Period is defined by