fr.jeremyschoffen.prose.alpha.reader.grammar — fr.jeremyschoffen/prose-alpha 7

Prose's grammar.

The Grammar propsed here is heavily inspired by Pollen's.

We construct it using in 2 parts:

• a lexical part or lexer made of regular expressions.
• a set of grammatical rules tyring the lexer together into the grammar.

The lexer.

Our lexer is made of regular expressions constructed with the fr.jeremyschoffen.prose.alpha.reader.grammar.utils/def-regex macro. It uses the Regal library under the covers.

Then, to assemble these regexes into a grammar we use the fr.jeremyschoffen.prose.alpha.reader.grammar.utils/make-lexer macro.

For instance we could construct the following 2 rules lexer:

(def-regex number [:* :digit])

(def-regex word [:* [:class ["a" "z"]]])

(def lexer (make-lexer number word))

lexer
;=> {:number {:tag :regexp
              :regexp #"\d*"}
     :word {:tag :regexp
            :regexp #"[a-z]*"}}

The grammatical rules

Most of the grammatical rules are created using the ebnf notation as follows

(def rules
  (instac/ebnf
    "
    doc = (token <':'>)*
    token = (number | word)
    "))

rules
;=>{:doc {:tag :star
          :parser {:tag :cat
                   :parsers ({:tag :nt :keyword :token}
                             {:tag :string :string ":" :hide true})}}
           :token {:tag :alt
                   :parsers ({:tag :nt :keyword :number}
                             {:tag :nt :keyword :word})}}

The combining trick

Now that we have both a lexer and and grammatical rules, we can simply merge them to have the full grammar.

(def parser
  (insta/parser (merge lexer rules)
                :start :doc))

(parser "abc:1:def:2:3:")
;=> [:doc
      [:token [:word "abc"]]
      [:token [:number "1"]]
      [:token [:word "def"]]
      [:token [:number "2"]]
      [:token [:number "3"]]]
```

With the exception of some details, this is how this namespace is organized.

all-delimitors clj/s

all-grammatical-rules clj/s

Merging of the lexer rules and the grammatical rules.

any-char clj/s

Any character whatsoever.

anything clj/s

braces clj/s

brackets clj/s

clojure-call-text clj/s

Clojure text found in clojure calls (enclosed in parens).

clojure-string clj/s

Text found in clojure strings.

close-brace clj/s

close-bracket clj/s

close-paren clj/s

complex-symbol clj/s

double-quote clj/s

enclosed-g clj/s

Grammatical describing text enclosed in balanced marker: quotes, parenthesis..;

escape clj/s

escape-char clj/s

The escaping character

general-g clj/s

The general grammar, tying the lexer and the enclosed rules together with the top grammatical rules.

grammar clj/s

The proper grammar in map form.

It is an updated all-grammatical-rules in regards to hidden-tags hidden-results.

hidden-results clj/s

Names of rules whose productions are completely hidden.

hidden-tags clj/s

Names of additional grammatical rules that have their tag hidden.

lexer clj/s

The proper lexer, a version of lexer* where each rule has its tag hidden.

lexer* clj/s

Our lexer, an incomplete instaparse grammar in map form containing only regex rules.

macro-reader-char clj/s

ns-end clj/s

open-brace clj/s

open-bracket clj/s

open-paren clj/s

parens clj/s

parser clj/s

Instaparse parser made from grammar

pipe clj/s

pipe-char clj/s

The pipe character found between the special character and a symbol when directly using a symbol in text.

plain-text clj/s

Text to be interpreted as plain text, neither clojure code, nor special blocks of text. Basically any character excluding diamond which have special meaning.

simple-symbol clj/s

Regex for the ns name of a symbol, parses dot separated names until a final name.

special clj/s

special-char clj/s

The special character that denotes embedded code.

symbol-excluded-charset clj/s

symbol-first-char clj/s

In the case of the first character of a symbol name, there are more forbidden chars:

• digits aren't allowed as first character
• the macro reader char # isn't allowed either.

symbol-regular-char clj/s

Characters that are always forbidden in symbol names:

• spaces
• diamond char since it starts another grammatical rule
• delimitors: parens, brackets, braces and double quotes.
• / since it the special meaning of separating the namespace from the symbol name.
• . since it has the special meaning of separating symbol names.
• \ since it is reserved by clojure to identify a literal character.

symbol-text clj/s

Regex used when parsing a symbol.

tag-clj-arg-text clj/s

Clojure text found in clojure arguments to tag fns (enclosed in brackets).

tag-spaces clj/s

Spaces found in-between tag args.

tag-text-arg-text clj/s

Regular text found in text arguments to tag fns (enclosed in braces).

verbatim-text clj/s

Text found in verbatim blocks.