23.05.15 Одесса. Impact Hub Odessa. Конференция AI&BigData Lab Николай Алексеенко (Lynx Capital Partners LCC) Описание, сравнительные характеристики и анализ применимости в Big Data KDB+ - весьма зрелой и мощной, но практически неизвестной в Восточной Европе базы данных. Подробнее: http://geekslab.co/ https://www.facebook.com/GeeksLab.co https://www.youtube.com/user/GeeksLabVideo

1. KDB+: Устрашающее совершенство
Lynx Capital Partners LLC
Николай Алексеенко
Руслан Шевченко
Виктор Советов
Ольга Гриценко

3. REQUIREMENTS
•
Speed
•
Big Data processing
•
InMemory/Disk storage
•
Realtime/Historical data
•
Distributed
•
Simplicity (sic!)

4. KDB+ Customers
•
Goldman Sachs
•
Morgan Stanley
•
J.P. Morgan
•
Deutsche Bank
•
Shenzhen Stock Exchange
•
NYSE

5. What is KDB+ ?
•
Column-Based DBMS
•
Q and K APL-like programming languages
•
SQL emulator
•
HTTP-Server
•
RPC-Server
•
etc.

6. What is Q?
•
APL-style vector language
•
Very fast small interpreter
•
As fast as C/C++
•
Compact code
•
Parallel operations

7. Q: code examples
q)table: ([] sym:`AAPL`IBM`MSFT; price:(132.54;172.22;46.90);
size:(100;300;250))
q)table
sym price size
----------------
AAPL 132.54 100
IBM 172.22 300
MSFT 46.9 250
q)dict:`AAPL`IBM`MSFT!(132.54;172.22;46.9)
q)dict
AAPL| 132.54
IBM | 172.22
MSFT| 46.9
q)10 * (1 2 3 4 5)
10 20 30 40 50
q)1 2 3 * 10 20 30
10 40 90

8. Q: code examples
q)genlist:{x+til y}
q)genlist[10;5]
10 11 12 13 14
q)genlist2:{[frst;lst] frst + til 1+lst-frst}
q)genlist2[5;10]
5 6 7 8 9 10
NormPrice:{
execs:select from orderexec where date in x;
syms:select price_digits by date,symbol from symbols where
date in x;
update price:price*xexp[10;neg price_digits] from
execs lj syms}

9. Q: code examples
q)fix:"8=FIX.4.2|9=0287|35=8|34=000001409|56=BS01|49=STIM|
52=20150504-11:49:34.922|128=ETBV|20=0|17=71|150=2|11=STE-
SHORT-106295118|37=31799|39=2|1=NVNIKHARS|55=CLF|54=5|38=2000|
32=2000|151=0|14=2000|44=5.84|31=5.84|6=5.84|9621=0.0|59=5|
9730=R|9991=R|30=SPDR|9994=ARCA|9995=ARCA|76=BAYP|9992=ARCA|
9993=SPDR_DMA|10=204|"
q)(!/)"I=|"0:fix
8 | "FIX.4.2"
9 | "0287"
35 | ,"8"
34 | "000001409"
56 | "BS01"
49 | "STIM"
52 | "20150504-11:49:34.922"
128| "ETBV"
20 | ,"0"
17 | "71"
150| ,"2"

10. Q code: understanding
q)5*2-2
0
q)(5*2)-2
8
FUNCTION ARG;
FUNCTION2 FUNCTION1 FUNCTION0 ARG0
FUNCTION2 FUNCTION1 ARG1
FUNCTION2 ARG2
q)a: 1 2 3
q)a 1
2
q)a[1]
2
q)5*2-2
0
q)(5*2)-2
8

11. Q/KDB+: interfacing
•
С/С++
•
Java
•
Scala
•
Python
•
Perl
•
C#
•
Lua
•
PHP
•
Mathlab
•
R
•
CUDA
•
etc

12. KDB+: IPC
C APP
KDB+
KDB+
Java APP
Mode:
•
Sync (Request-Response)
•
Async (Data Streaming)

13. KDB+TICK
Feeds Frontend RDB HDB
PUB/SUB
RT Queries
Reports&
Analytics

14. KDB+ problems: price

15. KDB+ problems:
Newbies

16. Q Resources
•
Q For Mortals. A tutorial in Q programming (Jeffry A.
Borror)
•
Q Tips: Fast, Scaleable and Maintainable KDB+ (Nick
Psaris)
•
http://code.kx.com/wiki/Main_Page
•
Q For Gods Whitepaper Series
http://www.firstderivatives.com/
•
http://www.timestored.com

17. Questions ?