JingYi Yao of Didi @ HBaseConAsia2018

1. hosted by
HBaseConAsia2018
August 17,2018
Gehua New Century Hotel Beijing，China

2. hosted by
HBase at DiDi
JingYi Yao

3. hosted by
Content 01
02
04
03
HBase Improvements
& Applications
Phoenix Improvements
& Applications
GeoMesa Applications
Monitor & Operations

4. hosted by
HBase Improvements
& Applications
01

5. hosted by
HBase Improvements & Applications
◇ 1.HBase Applications at DiDi
◇ 2.ACL Improvements
◇ 3.Replication Improvements
◇ 4.Reduce User Connection

6. hosted by
HBase Improvements & Applications
1.HBase Applications

7. hosted by
HBase Improvements & Applications
1.HBase Applications

8. hosted by
HBase Improvements & Applications
1.HBase Applications

9. hosted by
HBase Improvements & Applications
1.HBase Applications
◇ Message Queue
◇ GeoGraphic Feature data
◇ Heat Map
◇ User Profile
◇ Log Analysis

10. hosted by
HBase Improvements & Applications
2.Replication Improvements

11. hosted by
HBase Improvements & Applications
2.Replication Improvements
◇ (1).Replication in RS Group
◇ (2).Table Metric
◇ (3).RS Group ratio

12. hosted by
HBase Improvements & Applications
3.Connection
◇ (1).users
◇ (2).Coprocessor：
◇ Regionserver clusterConnection

13. hosted by
HBase Improvements & Applications
4.ACL Improvements

14. hosted by
HBase Improvements & Applications
4.ACL Improvements

15. hosted by
HBase Improvements & Applications
4.ACL Improvements

16. hosted by
HBase Improvements & Applications
◇ reduce zookeeper znode
◇ and watcher
4.ACL Improvements

17. hosted by
HBase Improvements & Applications
5.Others
◇ (1)RPC audit log
◇ (2)RSGroup bug fix
◇ (3)quota
◇ (4)safe drop table
◇ …

18. hosted by
Phoenix Improvements &
Applications
02

19. hosted by
Phoenix Improvements & Applications

20. hosted by
Phoenix Improvements & Applications

21. hosted by
Phoenix Improvements & Applications
◇ SLA 99.95%
◇ Secondary indexes
◇ Java-jdbc client P99 35 ms
◇ column default
◇ QueryServer
5.Order History

22. hosted by
Phoenix Improvements & Applications
◇ Secondary indexes ：
◇ INDEX Type: Global Index => Covered Index
◇ ASYNC INDEX
◇ SALT_BUCKETS => reverse(ID) & pre split
◇ coprocessor ：reduce connection
◇ QueryServer metrics

23. hosted by
Phoenix Improvements & Applications

24. hosted by
Phoenix Improvements & Applications
◇ zk connection:
◇ region num * Index num => 1

25. hosted by
GeoMesa Applications03

26. hosted by
GeoMesa Applications
GeoMesa
GeoMesa is an open source suite of
tools that enables large-scale
geospatial querying and analytics on
distributed computing systems.

27. hosted by
GeoMesa Applications
◇ 1.Road Detection
◇ 2.Device ID => Trajectory

28. hosted by
GeoMesa Applications

29. hosted by
GeoMesa Applications
◇ Z2 : latitude and longitude
◇ Z3 : latitude, longitude, and time
◇ XZ2 : line strings or polygons
◇ XZ3 : non-Point geometry and has a time attribute
◇ Record/ID
◇ Attribute
Index

30. hosted by
GeoMesa Applications
◇ GeoHash:
◇ geohashes have been used to first convert latitude and longitude into a single bit vector

31. hosted by
GeoMesa Applications
Z-order curve

32. hosted by
GeoMesa Applications
◇ Weakness
◇ More scan
Z-order curve

33. hosted by
GeoMesa Applications
◇ Hilbert filling curve Inedx
◇ Geomesa plan
◇ Coprocessor
◇ GeoMesa Stats
Future
Experience from Phoenix

34. hosted by
Monitor & Operations04

35. hosted by
Monitor & Operations
◇ (1).HBase
◇ (2).Zookeeper
◇ (3).HDFS
◇ (4).ThriftServer
◇ (5).Queryserver
1.Monitor

36. hosted by
Zookeeper
Monitor & Operations

37. hosted by
◇ echo mntr | nc $zookeeper 2181
◇ zk_num_alive_connections
◇ zk_open_file_descriptor_count
◇ zk_outstanding_requests
◇ zk_min_latency
◇ zk_followers
Monitor & Operations

38. hosted by
◇ [Put | Get | Scan …]_99th_percentile
◇ Slow Read/Write …
ThriftServer
QueryServer
◇ print slow prepareStatement
◇ print slow query
◇ ExecuteQueryTime_99th_percentile
Monitor & Operations

39. hosted by
◇ slow disk
◇ edits log => hdfs path
HDFS
Monitor & Operations

40. hosted by
◇ DHS(Didi HBase Service)：
◇ Visualizing Information: Region hot-spots from Region Metric
◇ Read、Write、StoreSize
Monitor & Operations

41. hosted by
Monitor & Operations
The highest 10
&
The lowest 10

42. hosted by
◇ Automatic computation：input/Output Size per sec.Store Size..
◇ Recommended values：Region Count
Monitor & Operations

43. hosted by
◇ (1).read request、write request、Mutate_99th_percentile、Get_99th_percentile、
ScanTime_99th_percentile、receivedBytes、sentBytes
◇ (2).compactionQueueLength、flushQueueLength、splitQueueLength
◇ (3).jvm metrics…
◇ slow request: Rpc p99、gc、compaction queue 、slow log
Monitor & Operations
HBase Metric

44. hosted by
Monitor & Operations

45. hosted by
Monitor & Operations

46. hosted by
Monitor & Operations
yaojingyi@didichuxing.com

47. hosted by
Thanks