fr.jeremyschoffen.textp.alpha.reader.grammar — fr.jeremyschoffen/textp-reader-alpha 1

Textp's grammar.

We construct here textp's grammar using instaparse. Our grammar is then constructed here in two parts:

• a lexical part or lexer made of regular expressions.
• a set of grammatical rules tyring the lexer together into the grammar.

The lexer.

Our lexer is made of regular expression constructed with the [[textp.reader.alpha.grammar/defregex]] macro which uses the Regal library under the covers. We then assemble a lexer from these regular expressions with the [[textp.reader.alpha.grammar/make-lexer]] macro.

For instance we could construct the following 2 rules lexer:

(def-regex number [:* :digit])

(def-regex word [:* [:class ["a" "z"]]])

(def lexer (make-lexer number word))

lexer
;=> {:number {:tag :regexp
              :regexp #"\d*"}
     :word {:tag :regexp
            :regexp #"[a-z]*"}}

The grammatical rules

We use the [[instaparse.combinators/ebnf]] function to produce grammatical rules. This allows use to write these rules in the ebnf format.

For instance we could write the following:

(def rules
  (instac/ebnf
     "
     doc = (token <':'>)*
     token = (number | word)
     "))

rules
;=>{:doc {:tag :star
          :parser {:tag :cat
                   :parsers ({:tag :nt :keyword :token}
                            {:tag :string :string ":" :hide true})}}
    :token {:tag :alt
            :parsers ({:tag :nt :keyword :number}
                      {:tag :nt :keyword :word})}}

This way of writing the grammatical rules is way easier than using function combinators and still gives us these rules in map form.

The combining trick

Now that we have both a lexer and and grammatical rules, we can simply merge them to have the full grammar.

We actually get a instparse parser this way:

(def parser
  (insta/parser (merge lexer rules)
                :start :doc))

(parser "abc:1:def:2:3:")
;=> [:doc
     [:token [:word "abc"]]
     [:token [:number "1"]]
     [:token [:word "def"]]
     [:token [:number "2"]]
     [:token [:number "3"]]]
```

With the exception of some details, this is how this namespace is made.

all-delimitors clj/s

all-grammatical-rules clj/s

Merging of the lexer rules and the grammatical rules.

any-char clj/s

Regex that recognizes any character.

anything clj/s

braces clj/s

brackets clj/s

comment-g clj/s

Grammatical rule for commented text:

This text is normal text.
◊/The text here is kept commented out/◊

def-regex clj/s macro

(def-regex n xeger-expr)

(def-regex n doc xeger-expr)

Macro used to short hand:

(def a-regex (make-regex "a regal expression"))

into

(def-regex a-regex "a regal expression")

diamond clj/s

double-quote clj/s

embedded-g clj/s

Grammatical rules descripbing clojure code embedded in text.

We can embed clojure calls: ◊(def ex 1)◊ and clojure values ◊|x|◊

Not that the embedded call syntax is mutually recursive with the tag syntax.
We can have :
◊(def home ◊a[:href "www.home.com"]{Home})◊

and use it here: ◊|home|◊

embedded-g-masked clj/s

end-comment clj/s

end-embedded-value clj/s

end-embeded-code clj/s

end-verbatim clj/s

escaper clj/s

escaping-char clj/s

The backslash used to escaped characters in plain text.

general-g clj/s

general-g-masked clj/s

grammar clj/s

Final grammar with all the rules that need to be hidden specified as such.

grammar-masked clj/s

The set of the rule names that need to be hidden. These rules won't produce nodes in the parse tree. In compiler parlance these are the node you'd find in a the syntax tree but not in the abstract syntax tree.

hide-all clj/s

(hide-all g)

Hide all rules in a instaparse grammar in its data (map) form by applying [[instaparse.combinators/hide-tag]] to all values of the map.

hide-rules clj/s

(hide-rules g rule-names)

Selectively hide rules instaparse grammar in its data (map) form. It applies [[instaparse.combinators/hide-tag]] to the rules whose names are in rule-names.

lexer clj/s

Lexer of our grammar. Its the raw lexer with all rules are hidden by default (they won't materialize as a node of a parse tree).

lexer* clj/s

Raw lexer of our grammar. It's an instaparse grammar in data (map) form containing all the regular expressions used in the final parser.

macro-reader-char clj/s

make-complex-symbol-regex clj/s

(make-complex-symbol-regex rep)

Regex for a full symbol name with namespace. Parse an optional ns name followed by the character / then a simple symbol. The repetition for the character of the symbol name is parameterized to allow fo reluctant repetition.

make-lexer clj/s macro

(make-lexer & regexes)

Make a sequence of named regular expression into a intaparse map of named regex rules.

make-simple-symbol-regex clj/s

(make-simple-symbol-regex rep)

Regex for simple symbols without namespaces. The character repetition is parameterized to allow for reluctant repetition.

non-special clj/s

normal-text clj/s

ns-end clj/s

parens clj/s

parser clj/s

Our parser with the starting rule specified as the :doc rule and the output format tree set to :enlive.

plain-text clj/s

Text to be interpreted as plain text, neither clojure code, nor special blocks of text. Basically any character excluding diamond and backslash which have special meaning.

symbol-first-char clj/s

In the case of the first character of a symbol name, there are more forbidden chars:

• digits aren't allowed as first character
• the macro reader char # isn't allowed either.

symbol-ns-part clj/s

Regex for the ns name of a symbol, parses dot separated names until a final name.

symbol-regular-char clj/s

Characters that are always forbidden in symbol names:

• spaces
• diamond char since it starts another grammatical rule
• delimitors: parens, brackets, braces and double quotes.
• / since it the special meaning of separating the namespace from the symbol name.
• . since it has the special meaning of separating symbol names.
• \ since it is reserved by clojure to identify a literal character.

symbol-regular-char-set clj/s

tag-g clj/s

Grammatical rules for tag syntax.

A tag is meant to ultimately be a clojure call. It starts with the character ◊ followed by a symbol then followed by arguments. Arguments can be clojure arguments enclosed in brackets or text argument enclosed in braces.

Clojure arguments allow clojure code to be passed argument as embedded code which can contain other tags. Text argument are block of text which can recursively contain tags and embedded code.

tag-g-masked clj/s

tag-plain-text clj/s

The text found inside curly braces in tags. Can be anything but the chars:

• ◊: diamond will start a new grammatical rule
• }: right curly brace closes the text arg to the tag
• '' : backslash with start an escaped char grammatical rule

Allows for the forbidden chars to appear when escaped with a backslash.

text-comment clj/s

Regex used to parse text inside a comment block. All characters allowed terminated by "/◊"

text-e-code clj/s

Regex used when parsing parsing text in embedded code.

text-e-value clj/s

Regex used when parsing a symbol in the case of embedded values. It basically is the same as text-symbol except for the use of reluctant repetition for the symbol name and the use of a lookahead at the end to search for the end of an embedded value block.

text-escaped-char clj/s

text-g clj/s

Grammatical rules for top level text. Basically any character except "◊" or any escaped character.

text-g-masked clj/s

text-spaces clj/s

Spaces found inbetween tag args.

text-symbol clj/s

Regex used when parsing a symbol in the case of tag names.

text-t-clj clj/s

Regex used when parsing a the text inside a clojure argument to a tag. Can be anything but the chars:

• ◊: diamond will start a new grammatical rule
• [], brackets: theses characters will start a new grammatical rule
• ": double quote will start a new grammatical rule

Allows for the forbidden char to appear when escaped with a backslash.

text-t-clj-non-special clj/s

text-t-clj-str clj/s

The text inside a clojure string. Can be anything but the char:

• ": double quote will close the string

Allows for the forbidden chars to appear when escaped with a backslash.

text-t-clj-str-non-special clj/s

text-t-txt-non-special clj/s

text-verbatim clj/s

Regex used to parse text inside a verbatim block. All characters allowed terminated by "!◊"

verbatim-g clj/s

Grammatical rule for verbatim text:

This text is normal text.
◊!The text here is kept ◊verbatim!◊