#norikra meetup 2

1. Norikra Recent Updates
Norikra meetup #2, 2015/06/03
TAGOMORI Satoshi(@tagomoris)

2. @tagomoris
Treasure Data, Inc.
Fluentd, Norikra, Hadoop, ...

3. Norikra:
Open source stream processing server with SQL
• v1.0.0: 2014/05/20
• "Stream Processing and Norikra"
• http://www.slideshare.net/tagomoris/stream-processing-and-norikra
• "Landscape of Norikra Features"
• http://www.slideshare.net/tagomoris/norikra-meetup-features
• (... 16 releases)
• v1.3.1: 2015/05/07

4. Changes of Norikra
• "Changes" section in README
• only for major updates
• "Changes.md"
• for details
• https://github.com/norikra/norikra/blob/master/Changes.md
• Commit logs

5. Features newly added
• Suspended queries
• NULLABLE fields
• Listener plugins
• Dynamic plugin reloading

6. Suspended queries
• temporarily stop queries / resume these later
• good for tests, try&errors
• [caution]
• suspended queries doesn't saved in stats file

7. NULLABLE fields
• Queries process events just with all fields
• NULLABLE(field)
• queries handle field as NULL if missing
• works well for multi data source situation

8. NULLABLE fields(2)
SELECT a, b,
COUNT(*) AS count
FROM s.win:time(1 min)
GROUP BY a, b
{"a":1, "b":2} {"a":2}
{"a":1, "b":2, "count":1}
{"a":2, "b":2, "count":1}
{"a":2, "b":2}
ignored!

9. NULLABLE fields(3)
SELECT a, NULLABLE(b),
COUNT(*) AS count
FROM s.win:time(1 min)
GROUP BY a, b
{"a":1, "b":2} {"a":2}
{"a":1, "b":2, "count":1}
{"a":2, "b":2, "count":1}
{"a":2, "b":null, "count":1}
{"a":2, "b":2}

10. Listener
• Listener
• process output data of queries
• default listener: memory pool
• Built-in listeners: specified by group
• LOOPBACK(target)
• STDOUT()

11. Listener plugin: do anything
• 3rd party listener plugin
• syntax: NAME(arg)
• JRuby + rubygem
• We can do whatever we want :D
• by writing very simple code
• https://github.com/norikra/norikra-lilstener-mock

12. Listener plugin: patterns
query
push to users
enqueue to
kafka
memory
pool
filter records
other target
sample records

13. Listener plugin: sync vs async
Sync listener:
> called per every query's output
> roughly specified by views (.win:...)
> select this if you wanna deliver events ASAP
Async listener:
> called per specified interval [0.1sec]
> process buffered output data
> very easy to write

14. Dynamic plugin reloading
• SIGHUP
• Norikra loads plugins newly installed in
JRuby runtime
• doesn't reload plugins already installed
• UDF plugins, Listener plugins
• Less restarts!

15. Other updates:
• /engine_statistics, --ui-context-path
• GC stats in engine statistics
• SO_REUSEADDR
• Shutoff mode under high memory pressure
• --log4j-properties-path, -javaagent
• Esper 5.2, msgpack.gem, ...
• and many other bug fixes!

16. Norikra improvement
never lasting!
Next:
• Custom views? Tables?
• Query distribution?
Thanks!