Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie 王涛：基于Cloudera impala的非关系型数据库sql执行引擎
Ähnlich wie 王涛：基于Cloudera impala的非关系型数据库sql执行引擎 (20)
Mehr von hdhappy001
Mehr von hdhappy001 (20)
王涛：基于Cloudera impala的非关系型数据库sql执行引擎
- 2. 提纲 1 Cloudera Impala 简介 2 SequoiaDB 简介 3 SequoiaSQL 增强特性 4 使用场景 5 性能指标
- 3. Real Time SQL Query Engine CLOUDERA IMPALA 简介
- 6. Impala 架构
- 10. JSON格式与查询 需要更多的时间被行业用户接受 { name: “Tim”, age: 30, phone: [ 12345, 67890 ] } CREATE TABLE EMP ( NAME CHAR(10), AGE INTEGER, PHONE1 INTEGER, PHONE2 INTEGER ); INSERT INTO EMP VALUES ( “Tim”, 30, 12345, 67890 ) ;
- 12. SequoiaSQL联邦执行引擎 • SequoiaSQL 是基于开源项目开 发的SQL执行引擎 – Cloudera Impala • • • • 不仅仅是“SQL-on-Hadoop” 支持JDBC 兼容Hive驱动 增强功能 – 加入SequoiaDB与关系型数据库 的读写接口 – 将metastore嵌入SequoiaDB – 增加Update/Delete/Merge语句 – 谓词条件下压 Application SQL JDBC SequoiaSQL Engine Federated Database
- 15. 查询谓词下压，利用数据库索引提升性能 查询 SELECT * FROM T, K WHERE T.Price > 1000 AND T.id = K.id 关联 数据存储层 HDFS Database T.Price > 1000 HDFS 批量扫描 索引扫描 优势： 1.减少网络数据传输 2.利用数据库索引提升性能
- 18. NoSQL in Industry USE CASES
- 20. 交易流水归档历史查询 Legacy Systems IBM UDB ESE TCPIP Oracle 10G SAS TCPIP End Users TCPIP TCPIP Management ODS & IDS SPSS Cube Analysts Other Sources New Systems TCPIP TCPIP … Web App SequoiaSQL TCPIP Text Data Manual Input Hive Data Mart Users TCPIP HDFS SequoiaDB R Web App Users
- 23. SQL1: SQL5: SELECT select cttb.appg_mode ac_nm,appg_date,cst_id,dep_acno,tran_ty ,cttb.appg_date pe,tran_amount from ,cttb.tran_code sdbmassdata.local_sdb_ch_bank limit 10; SQL2: select count(cst_id) from sdbmassdata.local_sdb_ch_bank; SQL3: select * from sdbmassdata.local_sdb_ch_bank order by tran_amount limit 10 ; SQL4: select max(tran_amount) from sdbmassdata.local_sdb_ch_bank group by ac_nm,appg_date,cst_id,dep_acno,tran_a mount,tran_type limit 10; 表名 记录数 local_sdb_ch_bank 20,000,000 local_bancs_cusm_f 70,602 …<300+字段> local_ctis_txn_tif_b 22,869,263 ,dhit.cst_tp_code as_f ,dhit.ssn_avg_bal local_dep_hist_inf_t 313,721 mp ,dhit.dpstm_term FROM sdbmassdata.local_ctis_txn_tif_bas_f cttb LEFT OUTER JOIN sdbmassdata.local_dep_hist_inf_tmp dhit ON ( cttb.ACT_NO = dhit.DEP_ACNO AND dhit.STAT_DT = '2012-02-01' ) LEFT OUTER JOIN sdbmassdata.local_BANCS_CUSM_F bcf ON (dhit.CST_ID = bcf.CI03_CUST_ACCT_NO) limit 1;
- 24. SequoiaSQL Hive性能对比 700 SQL SQL SQL SQL SQL 600 500 1 2 3 4 5 SequoiaSQL Hive 0.49 96.903 22.28 147.087 316.26 600.164 38.18 231.108 20.69 578.676 400 SequoiaSQL 300 Hive 200 环境： 3台物理机 每台7个数据分区组 100 0 SQL 1 SQL 2 SQL 3 SQL 4 SQL 5
- 25. Roadmap 产品路标规划