About the design and development of JSLT, a language for querying JSON data, and transforming between JSON formats.

1. JSLT: JSON query & transform
Lars Marius Garshol, lars.marius.garshol@schibsted.com
http://twitter.com/larsga 2018–09–12, JavaZone 2018

2. Data Platform
2
Data Platform
Batch
Streaming
Pulse

3. Data volume
3

4. Routing
• We send data to ~210 different destinations
• Filters on the data specify which data should go
where
• often very detailed conditions on many fields
• Full routing tree has ~600 filter/transform/sink
nodes
4

5. Transforms
• Because GDPR we need to anonymize most incoming data
formats
• Some data has data quality issues that cannot be fixed at
source, requires transforms to solve
• In many cases data needs to be transformed from one format to
another
• Pulse to Amplitude
• Pulse to Adobe Analytics
• ClickMeter to Pulse
• Convert data to match database structures
• …
5

6. Who configures?
• Schibsted has >100 business units
• for Data Platform to do detailed configuration for all of
these isn’t going to scale
• for sites to do it themselves saves lots of time
• Configuration requires domain knowledge
• each site has its own specialities in Pulse tracking
• to transform and route these correctly requires knowing all
this
6

7. Batch config: 1 sink
{
"driver": "anyoffilter",
"name": "image-classification",
"rules": [
{ "name": "ImageClassification", "key": "provider.component", "value": "ImageClassification" },
{ "name": "ImageSimilarity", "key": "provider.component", "value": "ImageSimilarity" }
],
"onmatch": [
{
"driver": "cache",
"name": "image-classification",
"level": "memory+disk"
},
{
"driver": "demuxer",
"name": "image-classification",
"rules": "${pulseSdrnFilterUri}",
"parallel": true,
"onmatch": {
"driver": "textfilewriter",
"uri": "${imageSiteUri}",
"numFiles": {
"eventsPerFile": 500000,
"max": ${numExecutors}
}
}
}
],
"consume": true
7
Early config was 1838 lines
} JSON matching

8. A real transform
8

9. What if?
• We had an expression language for JSON
• something like, say, XPath for JSON
• could write routing filters using that
• We had a tranformation language for JSON
• write as JSON template, using expression language to
compute values to insert
• A custom routing language for both batch and
streaming, based on this language
• designed for easy expressivity & deploy
9

10. • Already existing query language for JSON
• https://stedolan.github.io/jq/
• Originally implemented in C
• there is a Java implementation, too
• Can do things like
• .foo
• .foo.bar
• .foo.bar > 25
• …
10

11. First, fumbling attempt
{
“event_type” : “View”,
“insert_id” : {“__expr__” : “.object.id”},
“source” : {“__if__” : {
“test” : “.source”,
“then” : “.source.id”,
“else” : “.src”
}
}
11

12. Second, fumbling attempt
{
“event_type” : “View”,
“insert_id” : “${.object.id}”,
“source” : “${if (.source) .source.id else .src}”
}
12

13. Third attempt
{
“event_type” : “View”,
“insert_id” : ${ .object.id },
“source” : if ${ .source } ${ .source.id } else ${ .src }
}
13

14. Proof-of-concept
• Implement real-world transforms in this language
• before it was implemented
• Helped improve and solidify the design
• Verified that the language could do what we
needed
• Transforms looked quite reasonable.
14

15. A simple language
• JSON is written in JSON syntax
• evaluates to itself
• if <expr> <expr> else <expr>
• [for <expr> <expr>]
• let <name> = <expr>
• ${ … jq … }
15

16. Matchers
{
“event_type” : “View”,
“insert_id” : ${ .object.id },
* : ${ . }
}
16
{
“event_type” : “View”,
“insert_id” : ${ .object.id },
* - “object” : ${ . }
}

17. Stunt prototype
• Most of it implemented in two days
• Implemented in Scala
• using Antlr 3 to generate the parser
• jackson-jq for jq
• jackson for JSON
• A simple object tree interpreter
• Constructor.construct(Context, JsonNode) => JsonNode
17

18. Object tree{
“event_type” : “View”,
“insert_id” : ${ .object.id },
“source” : if ${ .source } ${ .source.id } else ${ .src }
}
18
ObjectConstructor
PairConstructor(LiteralConstructor)
PairConstructor(JqConstructor)
PairConstructor(IfConstructor(Jq, Jq, Jq))

19. Literal expression
19

20. Object expression
20
Create object
Construct value
Add to object

21. If
21
Evaluate
condition
Construct then
Construct else

22. The parser
• Code that takes a character stream and builds the
expression tree
• Use a parser generator to handle the difficult part
• requires writing a grammar
• Parser generator produces Abstract Syntax Tree
• basically corresponds to the grammar structure
22

23. Antlr Grammar
grammar Jstl;
WS : [ trn]+ -> skip ; // ignore whitespace
COMMENT : '//' ~[rn]* [rn] -> skip ; // ignore comments
STRING : '"' ((~["]) | ('"'))+ '"' ;
INT: '-'? [0-9]+ ;
FLOAT: '-'? [0-9]+ '.' [0-9]+ ;
NULL: 'null';
BOOL: 'true' | 'false';
IDENT: [A-Za-z] [_A-Za-z]* ;
JQ : '$' '{' (~[}"] | '"' (~["] | '' .)* '"')+ '}' ;
jstl : let* expr EOF ;
expr : object | array | STRING | INT | FLOAT | NULL | BOOL |
JQ | ifvalue | forexpr | parenthesis;
23

24. Parser
24

25. Language in use
• Implemented Data Quality Tooling using jq
• filters done in jq
• Implemented routing using jq filters
• and transforms in JSLT
• Wrote some transforms using the language
• anonymization of tracking data
• cleanup transforms to handle bad data
• …
25

26. The good
• The language works
• proven by DQT, routing, and transforms
• Minimal implementation effort required
• Users approved of the language
• general agreement it was a major improvement
• people started writing their own transforms
26

27. The bad
• Performance could be better
• not horrible, but not great, either
• The ${ … } wrappers are really ugly
• jq
• does not handle missing data well
• has dangerous features
• has weird and difficult syntax for some things
• Too many dependencies
• Scala runtime (with versioning issues)
• Antlr runtime
27

28. 2.0
• Implement the complete language ourselves
• goodbye ${ … }
• Get rid of the jq strangeness
• Add some new functionality
• Implement in pure Java with JavaCC
• JavaCC has no runtime dependencies
• only dependency is Jackson
28

29. JSLT expressions
29
.foo Get “foo” key from input object
.foo.bar Get “foo”, then “.bar” on that
.foo == 231 Comparison
.foo and .bar < 12 Boolean operator
$baz.foo Variable reference
test(.foo, “^[a-z0-9]+$”) Functions (& regexps)

30. JSLT transforms
30

31. Anonymization
31

32. Sinks
VG-FrontExperimentsEngagement-1:
eventType: PulseAnonymized
filter: get-client(.) == "vg" and ."@type" == "Engagement" and
contains(.object."@type", ["Article", "SalesPoster"]) and
(contains("df-86-", .origin.terms) or
contains("df-86-", .object."spt:custom".terms))
transform: transforms/vg-article-views.jslt
kinesis:
arn: arn:aws:kinesis:eu-west-1:070941167498:stream/
vg_front_experiments_engagement
role: arn:aws:iam::070941167498:role/data-platform-kinesis-write
32

33. Expressions
• + - / *
• and or
• > < <= != == >=
• literals
• variable references
• function calls (+ function library)
33

34. Dealing with missing data
• 2 + null => null
• size(null) => null
• number(“12”) => 12
• number(null) => null
34

35. Operators
35
Evaluate left & right
Do the operation

36. Numeric operators
36
Null handling
Convert to numbers
or error
Decide if int or float
The actual operations

37. Minus
37

38. Object for expressions
38
{
“event_type” : “View”,
“insert_id” : .object.id,
“source” : if (.source) .source.id else .src
} + {
for (.custom)
“custom_” + .key : .value
}
{
“object” : {“id” : 21323, … },
“src” : “App123”,
“custom” : {
“time” : 4592593492,
“event”: 3433
}
}
{
“event_type” : “View”,
“insert_id” : 21323,
“source” : “App123”,
“custom_time”: 4592593492,
“custom_event” : 3433
}

39. Function declarations
def <name>(<param1>, <param2>)
<let>*
<expr>
Very easy to implement
Very useful
But means going Turing-complete…
39

40. A real function
40

41. Implementation
41

42. Turing-complete?
• Means that the language can express any
computation
• It’s known that all that’s required is
• conditionals (we have if tests)
• recursion (our functions can call themselves)
• But can this really be true?
42

43. N-queens
• Write a function that takes the size of the
chessboard and returns it with queens
• queens(4) =>
[
[ 0, 1, 0, 0 ],
[ 0, 0, 0, 1 ],
[ 1, 0, 0, 0 ],
[ 0, 0, 1, 0 ]
]
43https://github.com/schibsted/jslt/blob/master/examples/queens.jslt

44. Getting started
44

45. Danger?
• It’s possible to implement operations that run
forever
• But in practice the stack quickly gets too deep
• The JVM will then terminate the transform
45

46. Performance
• 5-10 times faster than 1.0
• The main difference: no more jackson-jq
• jackson-jq is not very efficient
• internal model is List<JsonNode>
• creates many unnecessary objects during evaluation
• does work at run-time that should be done at compile-time
46

47. JSLT improvements
• Value model is JsonNode
• can usually just return data from input object or from code
• Efficient internal structures
• all collections are arrays
• very fast to traverse
• Boolean short-circuiting
• once we know the result, stop evaluating
• Cache regular expressions to avoid recompiling
47

48. The optimizer
• An optimizer is a function that takes an expression
and outputs an expression such that
• the new expression is at least as fast, and
• always outputs the same value
• Improves performance quite substantially even with
very simple techniques
48

49. Constant folding
contains(get-client(.), [“vg”, “aftenposten”, “bt”])
49
FunctionExpression
FunctionExpression
DotExpression
ArrayExpression
LiteralExpression
LiteralExpression
LiteralExpression
FunctionExpression
FunctionExpression
DotExpression
LiteralExpression

50. Implementation
50

51. Performance
• Test case: pulse-cleanup.jslt, real data, my laptop
• a complicated transform: 165 lines
• Transforms 132,000 events/second in one thread
• 1.0 did about 20,000 events/second
51

52. Three strategies
• Syntax tree interpreter
• known to be the slowest approach
• Bytecode compiler with virtual machine
• C version of jq does this
• Java does that (until the JIT kicks in)
• Python does this
• Native compilation
• what JIT compiler in Java does
52

53. Designing a VM
53
Opcode Param
DUP
MKOBJ
CALL <func>
int[] bytecode;
JsonNode[] stack;
int top;
switch (opcode) {
case DUP:
stack[++top] = stack[top-1];
break;
case MKOBJ:
stack[++top] = mapper.createObj…
break;
// …
}

54. Compiler
• Traverse down the object tree
• emit bytecode as you go
• Stack nature of the VM matches object tree
structure
• each Expression produces code that leaves the value of
that expression on the stack
• Example:
• MKARR, <first value>, ARRADD, <second>, ARRADD, …
54

55. Prototype
• Stunt implemented over a couple of days
• Depressing result: object tree interpreter ~40%
faster
• Anthony Sparks tried the same thing
• original VM implementation 5x slower
• eventually managed to achieve performance parity
• So far: performance does not justify complexity
55

56. Java bytecode?
• The JVM is actually a stack-based VM
• can simply compile to Java bytecode instead
• Tricky to learn tools for generating bytecode
• no examples, very little documentation
• In the end decided to use the Asm library
• not very nice to use
• very primitive API
• crashes with NullPointerException on bad bytecode
56

57. Compiler
57

58. Compile dot accessor
58

59. Results
• Hard work to build
• many surprising issues in Java bytecode
• Performance boost of 15-25%
• code lives on jvm-bytecode branch in Github
• Ideas for how it could be even faster…
• through type inference
59

60. Type inference benefits
"sdrn:" + $namespace + ":" + $rType + ":" + $rId
Plus
Plus(“sdrn” $namespace)
Plus(
Plus(“:” $rType)
Plus(“:” $rId)
)
60
Plus:
JsonNode -> String
JsonNode -> String
String + String
new String -> new JsonNode
Will make 4 unnecessary TextNode objects
Will wrap and unwrap String repeatedly
Will check types unnecessarily

61. Solution
• + operator can ask both sides: what type will you
produce?
• If one side says “string” then the result will be a string
• When compiling, do compile(generator, String)
• will compile code that produces a Java String object
• + operator will make a new String if that’s what’s
wanted
• or turn it into a TextNode if the context wants Any
61

62. Freedom from Jackson
• The current codebase is bound to Jackson
• JVM bytecode compilation might be a way to
escape that
• Could build compilers that can interface with
different JSON representations
• Have ideas for a more efficient JSON representation
• basically encode everything as arrays of ints
• should save memory, GC, and produce faster code
62

63. Freedom from JSON
• If we aren’t bound to Jackson, why should we be
bound to JSON?
• Could support Avro, too
• Perhaps also other formats
63

64. Conclusion

65. Internal status
• JSLT now used in
• Data Quality Tooling (to express tests on data)
• routing filters
• transforms
• In Schibsted we have
• 52 transforms, 2370 lines of code
• written by many people in different parts of the company
• Data Platform runs ~11 billion transforms/day
65

66. Open source status
• Released in June
• People are using it for real
• one certain case, several more examples
• details unknown
• Useful contributions from outsiders
• several bug fixes to datetime/number handling
• Two alternative implementations being worked on
• one in .NET
• one is virtual machine-based in Java
66

67. Lessons learned
• A custom language can make life much simpler
• if it fits the use case well
• Implementing a language is easier than it seems
• basically doable in a week
• Designing a language is not easy
• unfortunately
67

68. https://github.com/schibsted/jslt
https://www.slideshare.net/larsga
Slides at