Slides from my talk from #jQueryBulgaria 2013

1. Creating own language
made easy
Ingvar Stepanyan
@RReverser

2. Everything sucks

3. Everything sucks – human version

4. Everything sucks – developer version

5. Scary magic
???
??
?

7. Not so scary magic
JS
JS
JS
Parser
Generator

8. Parsers

9. Parser generators

jison Bison in javascript, used by Coffeescript

PEG.js parser generator for JavaScript based on the parsing expression grammar formalism

OMeta/JS (source) metacompiler for many languages to many targets, including js.

languagejs - PEG parser written in JavaScript with first class errors

Canopy Self-hosting PEG parser compiler for JavaScript, influenced by Ruby parser generators such as Treetop and
Citrus

JS/CC LALR(1) parser generator

jsparse PEG by Grandmaster Chris Double

ReParse parser combinator library for Javascript like Haskell's Parsec

p4js Monadic parser library for JavaScript

JSGLR Scannerless, Generalized Left-to-right Rightmost (SGLR) derivation parser for JavaScript

antlr has a javascript target

Cruiser.Parse LL(k) parser

10. Parser generators
Top-down (Jison)
Bottom-up (PEG.js)
start
= additive
%left "+"
%left "*"
additive
= left:multiplicative "+" right:additive { return
left + right; }
/ multiplicative
%start program;
multiplicative
= left:primary "*" right:multiplicative { return
left * right; }
/ primary
program
: expression { return $$ }
;
primary
= integer
/ "(" additive:additive ")" { return additive; }
integer "integer"
= digits:[0-9]+ { return parseInt(digits.join(""),
10); }
%%
expression
: expression "+" expression -> $1 + $3
| expression "*" expression -> $1 * $3
| "(" expression ")" -> $2
| NUMBER -> $1
;

11. Choice ordering
Top-down (Jison)
Bottom-up (PEG.js)
start
= additive
%left "+"
%left "*"
additive
= left:multiplicative "+" right:additive { return
left + right; }
/ multiplicative
%start program;
multiplicative
= left:primary "*" right:multiplicative { return
left * right; }
/ primary
program
: expression { return $$ }
;
primary
= integer
/ "(" additive:additive ")" { return additive; }
integer "integer"
= digits:[0-9]+ { return parseInt(digits.join(""),
10); }
%%
expression
: expression "+" expression -> $1 + $3
| expression "*" expression -> $1 * $3
| "(" expression ")" -> $2
| NUMBER -> $1
;

12. Ambiguity
if a then (if b then s) else s2
or
if a then if b then s else s2:
if a then (if b then s else s2)
Top-down (Jison)
Bottom-up (PEG.js)
start
= additive
%left "+"
%left "*"
additive
= left:multiplicative "+" right:additive { return
left + right; }
/ multiplicative
%start program;
multiplicative
= left:primary "*" right:multiplicative { return
left * right; }
/ primary
program
: expression { return $$ }
;
primary
= integer
/ "(" additive:additive ")" { return additive; }
integer "integer"
= digits:[0-9]+ { return parseInt(digits.join(""),
10); }
%%
expression
: expression "+" expression -> $1 + $3
| expression "*" expression -> $1 * $3
| "(" expression ")" -> $2
| NUMBER -> $1
;

13. Left recursion
x=1-2-3
Top-down (Jison)
Bottom-up (PEG.js)
start
= additive
%left "+"
%left "*"
additive
= left:multiplicative "+" right:additive { return
left + right; }
/ multiplicative
%start program;
multiplicative
= left:primary "*" right:multiplicative { return
left * right; }
/ primary
program
: expression { return $$ }
;
primary
= integer
/ "(" additive:additive ")" { return additive; }
integer "integer"
= digits:[0-9]+ { return parseInt(digits.join(""),
10); }
%%
expression
: expression "+" expression -> $1 + $3
| expression "*" expression -> $1 * $3
| "(" expression ")" -> $2
| NUMBER -> $1
;

14. Left recursion
x=1-2-3
Top-down (Jison)
Bottom-up (PEG.js)
[
[
"x",
"=",
[
[
"1",
"-",
"2"
],
"-",
"3"
]
"x",
"=",
[
"1",
"-",
[
"2",
"-",
"3"
]
]
]
]

15. Summary choice
Top-down (Jison)
Bottom-up (PEG.js)
start
= additive
%left "+"
%left "*"
additive
= left:multiplicative "+" right:additive { return
left + right; }
/ multiplicative
%start program;
multiplicative
= left:primary "*" right:multiplicative { return
left * right; }
/ primary
program
: expression { return $$ }
;
primary
= integer
/ "(" additive:additive ")" { return additive; }
integer "integer"
= digits:[0-9]+ { return parseInt(digits.join(""),
10); }
%%
expression
: expression "+" expression -> $1 + $3
| expression "*" expression -> $1 * $3
| "(" expression ")" -> $2
| NUMBER -> $1
;

16. Jison syntax: helpers
%{
var scope = {};
%}
…

17. Jison syntax: lexer
…
%lex
%%
s+
/* skip whitespace */
[A-Za-z_]w+
return 'ID';
d+
return ‘NUMBER’;
[+*;=]
return yytext;
<<EOF>>
return 'EOF';
/lex
…

18. Jison syntax: operator precedence
…
%left ';‘
%right ‘=‘
%left ‘+’
%left ‘*’
…
/*
“x=a*b+c”
-> assign(“x”, “a*b+c”)
-> assign(“x”, add(“a*b”, “c”))
-> assign(“x”, add(mul(“a”, “b”), “c”))
*/

19. Jison syntax: rules
%start program
program
: stmt* EOF { return $1 }
;
stmt
: expr ‘;’ -> $1
;
expr
: expression "+" expression -> $1 + $3
| expression "*" expression -> $1 * $3
| NUMBER -> $1
;

20. Code generation

21. Methods
 string concatenation
 building AST object + escodegen (http://github.com/Constellation/escodegen)
 using SourceNode from source-map (https://github.com/mozilla/source-map)

22. Debugging: source maps

23. Methods
 string concatenation
 building AST object + escodegen (http://github.com/Constellation/escodegen)
 using SourceNode from source-map (https://github.com/mozilla/source-map)

24. AST way

25. Methods
 string concatenation
 building AST object + escodegen (http://github.com/Constellation/escodegen)
 using SourceNode from source-map (https://github.com/mozilla/source-map)

26. SourceNode
 new SourceNode(line, column, filename, jsChunk)
 line, column – position in original file
 filename – name of original file
 jsChunk – JavaScript code string, another SourceNode instance or array of those

27. Resulting stack
JS
JS
JS
Jison
sourcemap

28. Live demo