Have you ever wondered what does it take to create new language? Have you ever wanted to contribute to your favourite one? Don't have time to read "the dragon book"? No problem. During this talk I will gently introduce you to world of token streams, lexers, parsers and compilers. I will try to show you patterns and tools you have at your disposal, at the same time not diving to deep into theory, rather focusing on practical aspects. Of course I will focus on JVM, bytecode and latest changes in a world of JVM, which make life of language developers easier.

1. The following presentation is provided under DHMB
license (Do Not Hurt My Brain).
Lecturer is not responsible for the financial and
moral damages resulting from taking too seriously
the content of the presentation.
Including permanent damage to neurons, reduction of
neurotransmitter activity at the molecular level
and group dismissal.

2. Patterns for JVM languages
Polish JUG, 2014

3. about me
work://chief_architect@lumesse
owner://symentis.pl
twitter://j_palka
blog://geekyprimitives.wordpress.com
scm:bitbucket://kcrimson
scm:github://kcrimson

6. JVM
is
the
new
assembler

7. translator
interpreter
compiler

9. source code
↓
target language source code
↓
machine code || interpreter

11. source code
↓
execution results

13. source code
↓
machine code

14. So how does it all work?

15. source code
↓lexer
token stream
↓parser
IR

16. Intermediate Representation
visitor
listener
tree

17. Syntax tree

18. local y = 1
x = 1 + y

22. fragment
DIGIT
:
[0-9]
;
fragment
LETTER
:
[a-zA-Z]
;
String
:
''' .*? '''
| '"' .*? '"'
;
Number
:
DIGIT+
;
Name
:
(
'_'
| LETTER
)
(
'_'
| LETTER
| DIGIT
)*
;
COMMENT
:
'--' .*? 'n' -> skip
;
NL
:
'r'? 'n' -> skip
;
WS
:
[ tf]+ -> skip
;

23. [@0,[0..1)='x',<49>]
[@1,[2..3)='=',<36>]
[@2,[4..5)='1',<48>]
[@3,[6..7)='+',<33>]
[@4,[8..15)='"Hello"',<47>]
[@5,[15..16)=';',<37>]
[@6,[17..17)='',<-1=EOF>]

24. varlist returns [Varlist result]
:
var
{$result = createVarlist($var.result);}
(
',' var
{$result = createVarlist($result,$var.result);}
)*
;
//
var returns [Var result]
:
Name
{ $result = createVariableName($Name.text); }
| prefixexp '[' exp ']'
| prefixexp '.' Name
;

25. beware of left recursion

26. translator
interpreter
compiler

27. Interpreter pattern
Non terminal nodes
Terminal nodes

29. package pl.symentis.lua.grammar;
import pl.symentis.lua.runtime.Context;
public interface Statement {
void execute(Context context);
}

30. package pl.symentis.lua.grammar;
import pl.symentis.lua.runtime.Context;
public interface Expression<T> {
T evaluate(Context ctx);
}

31. package pl.symentis.lua.runtime;
public interface Context {
VariableRef resolveVariable(String name);
Scope enterScope();
Scope exitScope();
}

32. package pl.symentis.lua.runtime;
public interface Scope {
Scope getOuterScope();
VariableRef resolveVariable(String name);
void bindVariable(String name, Object value);
}

33. translator
interpreter
compiler

34. compiler → byte code
org.ow2.asm:asm:jar
me.qmx.jitescript:jitescript:jar

35. JiteClass jiteClass = new JiteClass(classname,new String[] { p(Statement.class) });
jiteClass.defineDefaultConstructor();
// parse file and start to walk through AST visitor
final CodeBlock codeBlock = newCodeBlock();
chunk.accept(new CompilerVisitor(codeBlock));
codeBlock.voidreturn();
jiteClass.defineMethod("execute", JiteClass.ACC_PUBLIC,
sig(void.class, new Class[] { Context.class }), codeBlock);
return jiteClass.toBytes(JDKVersion.V1_7);

36. When writing bytecode compiler it helps to think
about Java code
But at the end you need to think like stack machine :)

37. @Override
public void execute(Context context) {
x = valueOf(1).plus(valueOf(2));
}

38. public void visitExpression(CodeBlock codeBlock) {
codeBlock.ldc(2);
codeBlock.invokestatic(p(Integer.class), "valueOf",sig(Integer.class, int.class));
codeBlock.invokestatic(p(LuaTypes.class), "valueOf",sig(LuaType.class,Object.class));
codeBlock.astore(2);
codeBlock.ldc(1);
codeBlock.invokestatic(p(Integer.class), "valueOf",sig(Integer.class, int.class));
codeBlock.invokestatic(p(LuaTypes.class), "valueOf",
sig(LuaType.class, Object.class));
codeBlock.aload(2);
codeBlock.invokeinterface(p(LuaType.class), "plus",sig(LuaType.class,
LuaType.class));
codeBlock.astore(3);
}

39. Symbol table
maps variable locals to its symbolic name

40. Pillars of every language design
Visibility
Types
Runtime

41. What's next?
Just let me know
type analisys and optimizations
invokedynamic
graal + truffle

42. Or interpreter/compiler hackathon