Suche senden
Hochladen
Ocean base --千亿级海量数据库-2011数据库大会
•
Als PPTX, PDF herunterladen
•
40 gefällt mir
•
5,833 views
K
knuthocean
Folgen
Melden
Teilen
Melden
Teilen
1 von 21
Jetzt herunterladen
Empfohlen
”日中EC比較と出店事例からみる中国ECビジネスの成功のポイント"セールス・インチャイナ株式会社 代表取締役社長 浅野 潤 氏 netyeargroup ...
”日中EC比較と出店事例からみる中国ECビジネスの成功のポイント"セールス・インチャイナ株式会社 代表取締役社長 浅野 潤 氏 netyeargroup ...
NetyearGroup
Resume
Shawki Nader09 (Nj)
Shawki Nader09 (Nj)
Nader Shawki
การวิจัยในชั้นเรียนตามโครงการ ทสรช. 2.0
วิจัยในชั้นเรียนครูชัยวัฒน์
วิจัยในชั้นเรียนครูชัยวัฒน์
rpg26
Tom Vlasic Tom helps business incorporate and successfully utilize information technology so they will realize their full potential. As a keen observer, quick study and adaptive professional, Tom works with a wide range of organizations. His work helps them understand the value of a comprehensive IT strategy and as a result, organizations can unleash their full potential with an approach that will ensure both business and shareholder success. His experience in executive-level roles makes Tom more understanding, productive and focused on facilitating successful outcomes with clients. As an active member on not-for-profit boards, Tom also has a record of enhancing both board and organization performance. Most recently Tom led an award winning Project Management firm that specialized in delivering complex technology projects in the Healthcare sector. Tom has a Master of Applied Science degree in Electrical Engineering from the University of Waterloo and is a licenced Professional Engineer
Tom Vlasic: Virtual Teams and Technology
Tom Vlasic: Virtual Teams and Technology
PMI SWOC
ผลงานของนักเรียนมอสอง
เพลงตลกดี
เพลงตลกดี
rpg26
ผลงานของนักเรียนมอสอง
เพลงตลกดี
เพลงตลกดี
rpg26
Singapore2
Singapore2
rpg26
Resume For Tina B Bennett
Resume For Tina B Bennett
tbbennett
Empfohlen
”日中EC比較と出店事例からみる中国ECビジネスの成功のポイント"セールス・インチャイナ株式会社 代表取締役社長 浅野 潤 氏 netyeargroup ...
”日中EC比較と出店事例からみる中国ECビジネスの成功のポイント"セールス・インチャイナ株式会社 代表取締役社長 浅野 潤 氏 netyeargroup ...
NetyearGroup
Resume
Shawki Nader09 (Nj)
Shawki Nader09 (Nj)
Nader Shawki
การวิจัยในชั้นเรียนตามโครงการ ทสรช. 2.0
วิจัยในชั้นเรียนครูชัยวัฒน์
วิจัยในชั้นเรียนครูชัยวัฒน์
rpg26
Tom Vlasic Tom helps business incorporate and successfully utilize information technology so they will realize their full potential. As a keen observer, quick study and adaptive professional, Tom works with a wide range of organizations. His work helps them understand the value of a comprehensive IT strategy and as a result, organizations can unleash their full potential with an approach that will ensure both business and shareholder success. His experience in executive-level roles makes Tom more understanding, productive and focused on facilitating successful outcomes with clients. As an active member on not-for-profit boards, Tom also has a record of enhancing both board and organization performance. Most recently Tom led an award winning Project Management firm that specialized in delivering complex technology projects in the Healthcare sector. Tom has a Master of Applied Science degree in Electrical Engineering from the University of Waterloo and is a licenced Professional Engineer
Tom Vlasic: Virtual Teams and Technology
Tom Vlasic: Virtual Teams and Technology
PMI SWOC
ผลงานของนักเรียนมอสอง
เพลงตลกดี
เพลงตลกดี
rpg26
ผลงานของนักเรียนมอสอง
เพลงตลกดี
เพลงตลกดี
rpg26
Singapore2
Singapore2
rpg26
Resume For Tina B Bennett
Resume For Tina B Bennett
tbbennett
Ocean base海量结构化数据存储系统 hadoop in china
Ocean base海量结构化数据存储系统 hadoop in china
knuthocean
Test
Test
mukda
อบรมไอซีที ณ สิงคโปร์
Singapore 2549
Singapore 2549
rpg26
การไตร่ตรองงานวิจัยในชั้นเรียน ตามการมาอบรมโครงการ ทสรช. 2.0 วันที่ 25 - 28 มกราคม 2553
การไตร่ตรองของครูชัยวัฒน์
การไตร่ตรองของครูชัยวัฒน์
rpg26
George Jucan George Jucan is a well known in the project management community as a successful IT project management consultant, speaker at public events, trainer and author of high-impact project management articles. George is an experienced Project Manager Professional (PMP) with 20 years of technical and management experience in complex environments, both in public and private sector. He provides consulting services in methodology definition and capability assessment, business reengineering and organizational restructuring, projects audit and recovery, as well as hands-on management of projects. George Jucan is currently a member of the Core Team for the upcoming PMBOK® Guide 5th Edition. Moreover, George is currently the Vice-Chair of the Canadian Advisory Committee for the upcoming ISO 21500 project management standard. He is also the Chair of PMI Information Systems SIG, as well as a Board member of PMI Southern Ontario Chapter.
George Jucan: Using PMCDF to Advance One's Career
George Jucan: Using PMCDF to Advance One's Career
PMI SWOC
Coach Cathy: Mind Mapping
Coach Cathy: Mind Mapping
PMI SWOC
Distributed system engineering_practice
Distributed system engineering_practice
knuthocean
A graphic designer specializing brochure design, technical illustration and in website developing, hosting, in the Denver - Conifer, and front range areas of Colorado. Over 20 Years of producing great 4-Color publications, printing, advertising, logos and illustration.
Artimage Keynote
Artimage Keynote
sannis
“云存储系统”赏析系列分享三:Sql与nosql
“云存储系统”赏析系列分享三:Sql与nosql
knuthocean
淘宝Ocean base云存储实践 2011架构师大会
淘宝Ocean base云存储实践 2011架构师大会
knuthocean
Ocean base --千亿级海量数据库-lamper_日照
Ocean base --千亿级海量数据库-lamper_日照
knuthocean
Weitere ähnliche Inhalte
Andere mochten auch
Ocean base海量结构化数据存储系统 hadoop in china
Ocean base海量结构化数据存储系统 hadoop in china
knuthocean
Test
Test
mukda
อบรมไอซีที ณ สิงคโปร์
Singapore 2549
Singapore 2549
rpg26
การไตร่ตรองงานวิจัยในชั้นเรียน ตามการมาอบรมโครงการ ทสรช. 2.0 วันที่ 25 - 28 มกราคม 2553
การไตร่ตรองของครูชัยวัฒน์
การไตร่ตรองของครูชัยวัฒน์
rpg26
George Jucan George Jucan is a well known in the project management community as a successful IT project management consultant, speaker at public events, trainer and author of high-impact project management articles. George is an experienced Project Manager Professional (PMP) with 20 years of technical and management experience in complex environments, both in public and private sector. He provides consulting services in methodology definition and capability assessment, business reengineering and organizational restructuring, projects audit and recovery, as well as hands-on management of projects. George Jucan is currently a member of the Core Team for the upcoming PMBOK® Guide 5th Edition. Moreover, George is currently the Vice-Chair of the Canadian Advisory Committee for the upcoming ISO 21500 project management standard. He is also the Chair of PMI Information Systems SIG, as well as a Board member of PMI Southern Ontario Chapter.
George Jucan: Using PMCDF to Advance One's Career
George Jucan: Using PMCDF to Advance One's Career
PMI SWOC
Coach Cathy: Mind Mapping
Coach Cathy: Mind Mapping
PMI SWOC
Distributed system engineering_practice
Distributed system engineering_practice
knuthocean
A graphic designer specializing brochure design, technical illustration and in website developing, hosting, in the Denver - Conifer, and front range areas of Colorado. Over 20 Years of producing great 4-Color publications, printing, advertising, logos and illustration.
Artimage Keynote
Artimage Keynote
sannis
“云存储系统”赏析系列分享三:Sql与nosql
“云存储系统”赏析系列分享三:Sql与nosql
knuthocean
淘宝Ocean base云存储实践 2011架构师大会
淘宝Ocean base云存储实践 2011架构师大会
knuthocean
Ocean base --千亿级海量数据库-lamper_日照
Ocean base --千亿级海量数据库-lamper_日照
knuthocean
Andere mochten auch
(11)
Ocean base海量结构化数据存储系统 hadoop in china
Ocean base海量结构化数据存储系统 hadoop in china
Test
Test
Singapore 2549
Singapore 2549
การไตร่ตรองของครูชัยวัฒน์
การไตร่ตรองของครูชัยวัฒน์
George Jucan: Using PMCDF to Advance One's Career
George Jucan: Using PMCDF to Advance One's Career
Coach Cathy: Mind Mapping
Coach Cathy: Mind Mapping
Distributed system engineering_practice
Distributed system engineering_practice
Artimage Keynote
Artimage Keynote
“云存储系统”赏析系列分享三:Sql与nosql
“云存储系统”赏析系列分享三:Sql与nosql
淘宝Ocean base云存储实践 2011架构师大会
淘宝Ocean base云存储实践 2011架构师大会
Ocean base --千亿级海量数据库-lamper_日照
Ocean base --千亿级海量数据库-lamper_日照
Ocean base --千亿级海量数据库-2011数据库大会
1.
OceanBase千亿级海量数据库 2011.4 日照:rizhao.ych@taobao.com
1
2.
3.
Oceanbase技术方案
4.
收藏夹应用案例
5.
系统展望2
6.
7.
注册会员:3.7亿,来访人群峰值6000万
8.
日PV:超过20亿
9.
在线商品数:8亿
10.
每分钟销售商品:4.8万
11.
交易额:单日超10亿,光棍节 19.5亿
12.
淘宝商品库、评价库、交易订单库、用户库、店铺库…
13.
今后几年信息量还将增长几倍到几十倍
14.
分库分表也不一定总是奏效3 数据来源:http://www.alibuybuy.com/posts/52702.html
15.
16.
支持80%以上互联网在线应用
17.
数据规模:百TB级,百台机器
18.
OLTP:几十万QPS,几万TPS
19.
OLAP:支持千万级记录实时计算
20.
定义支持的SQL子集标准
21.
支持MapReduce等时髦计算模型
22.
TPC-E
23.
更多,。。。4
24.
25.
数据容量大、可扩展性好、容错能力强
26.
27.
海量数据存储特点的进一步分析
28.
数据量大但修改量较小,一亿次更新 * 100B
= 10G
29.
区分最新修改的数据和老数据?
30.
OceanBase = RDBMS
+ 云存储
31.
动态数据(增删改):单机之内存+SSD
32.
静态数据:静态B+树,多机
33.
数据 = 静态数据+动态数据
34.
事务:集中化写事务+分布式读事务6
35.
36.
动态数据服务器UpdateServer:主+备,实时修改(内存+SSD)
37.
静态数据服务器ChunkServer:多台,静态数据存储 (磁盘或SSD)
38.
39.
随机读,不支持范围查询;
40.
Hash划分均匀;
41.
两种Hash:取模Hash与一致性Hash
42.
实例:Tair,Memcache,Dynamo,Cassandra
43.
分布式B+ Tree
44.
随机读和顺序扫描,支持范围查询;
45.
顺序划分不均匀,需要叶子节点分裂合并
46.
实例:Bigtable & HBase,Google
Megastore
47.
Oceanbase数据结构
48.
动态数据:单机B+树
49.
静态数据:分布式B+树
50.
新的静态数据 = 老的静态数据
+ 动态数据8
51.
52.
静态数据服务器ChunkServer
53.
机器动态上下线
54.
动态数据服务器UpdateServer
55.
内存+SSD服务,多网卡,万兆网卡
56.
备提供读服务
57.
可靠性
58.
静态数据服务器ChunkServer
59.
数据存储多份,一般为3份
60.
动态数据服务器UpdateServer
61.
Commit log +
RAID 1磁盘
62.
实时本地热备(主+备) + 准实时异地热备
63.
定位服务器RootServer
64.
实时本地热备(主+备) + 准实时异地热备9
65.
66.
单机写事务 +分布式读事务
67.
支持跨表事务
68.
一致性选择
69.
弱一致性
70.
最终一致性
71.
强一致性
72.
本地实时同步,异地准实时同步10
73.
74.
RootServer总体协调
75.
负载均衡因素:内存,磁盘等资源占用,读写负载等;
76.
数据迁移:迁移过程不影响对外服务
77.
读写分离
78.
ChunkServer只读,简化设计并提高读性能
79.
UpdateServer采用copy-on-write数据结构,写不影响读
80.
Oceanbase系统读和写基本不干扰11
81.
82.
在线修改schema
83.
没有随机写,SSD友好
84.
内置数据压缩,减少机器数量和网络数据流量
85.
在线(不停机)系统版本升级12
86.
87.
内存容量
88.
新增的记录:1千万条/天,1KB/条10GB/天
89.
记录的修改:1亿条/天,100B/条10GB/天
90.
网络:100,000QPS,100B/条10MB/s
91.
磁盘
92.
Commit log
(bin log):Group commit
93.
改进方案
94.
SSD
95.
多网卡、万兆网卡
96.
…13
97.
98.
需求:查找一个用户的所有收藏的所有商品详情
99.
收藏信息表保存收藏信息条目,40亿+
100.
收藏商品表保存收藏的商品详细信息,4亿+
101.
执行两张表的暴力Join?一个用户可以收藏数千商品
102.
冗余商品详细信息到收藏表?一件商品可被数十万用户收藏14
103.
104.
收藏夹数据 = 静态数据
+ 动态数据
105.
静态数据:收藏信息表冗余存储商品详情信息
106.
动态数据:收藏信息表和商品详情表分别存放到UpdateServer内存中
107.
操作步骤:顺序读取静态数据中用户的收藏信息及商品详情; 将动态数据中的用户收藏信息更新到读取结果中; 将动态数据中的用户收藏的商品信息更新到读取结果中;
15
108.
109.
110.
4 ChunkServer,
2 * E5520 @2.27HZ, 10 * 300GB SAS, 16GB
111.
21亿条记录, 压缩后160GB *
3, 10k块, 随机读, cache基本不命中17/26 响应时间长:同步读 -> 预读 CS load高:全异步模型
112.
113.
2 * E5520
@ 2.27HZ, 24G, 千兆网卡
114.
待优化点
115.
优化网络框架内存分配:优化后 QPS >
10W
116.
减少任务队列导致的上下文切换:优化后 QPS >
20W
117.
结论:UPS不是性能瓶颈18
118.
119.
数据拷贝:Direct IO,权衡接口模块化与性能
120.
内存分配:内存池,线程缓存
121.
锁:线程缓存,减少Cache锁冲突,copy-on-write数据结构
122.
上下文切换:替换基于任务队列的网络模型19
123.
124.
列式存储
125.
Blob支持
126.
MapReduce
127.
TPC-E
128.
代码开源
129.
…20
130.
个人博客:http://nosqlnotes.net 21 谢谢
Jetzt herunterladen