Swarnim Kulkarni (Cerner) Cerner has been an active consumer of HBase for a very long time, storing petabytes of healthcare data in its multiple isolated HBase clusters. This talk will walk through the design of Cerner's enterprise data hub with a focus on the multi-tenant HBase as a service offering within the hub.

1. HBase
in
the
Enterprise
Data
Hub
It’s
all
about
co-­‐exis0ng
peacefully!
Swarnim
Kulkarni
May
24th
2016

2. innova8on

3. innova8on
noun
in·∙no·∙va·∙0on
ˌi-­‐nə-­‐ˈvā-­‐shən
1.
the
introduc0on
of
something
new
2.
a
new
idea,
method
or
device

5. Chart
Search

6. Sepsis

7. Millennium
+

8. Popula8on
Health

9. HBase
sits
at
the
core
of
a
lot
of
these
solu8ons!

10. Each
solu8on
built
with
specific
business
problem
in
mind

11. CHART
SEARCH
SEPSIS
MILLENNIUM
+
POP
HEALTH

12. CHART
SEARCH
SEPSIS
MILLENNIUM
+
POP
HEALTH

13. Prevents
assets
from
being
shared
High
investment
High
maintenance
costs
High
barrier
of
entry
for
innova8on

14. Enter
The
Data
Hub

16. Bob
Tim

17. Bob

18. = +
+

20. Tim

21. = +
+

30. Innova8ons
Source
of
data
Raw
data

31. Data
Hub
Owner
Data
Hub
Consumers
Workers

32. “An
Enterprise
Data
Hub
is
a
centralized
loca0on
to
store
all
data,
for
as
long
as
needed,
in
its
original
fidelity;
with
flexibility
to
run
variety
of
enterprise
workloads
including
batch
processing
and
interac0ve
SQL
-­‐
with
robust
security,
audi0ng
and
management.”

33. “An
Enterprise
Data
Hub
is
a
centralized
loca-on
to
store
all
data,
for
as
long
as
needed,
in
its
original
fidelity;
with
flexibility
to
run
variety
of
enterprise
workloads
including
batch
processing
and
interac0ve
SQL
-­‐
with
robust
security,
audi0ng
and
management.”

34. “An
Enterprise
Data
Hub
is
a
centralized
loca-on
to
store
all
data,
for
as
long
as
needed,
in
its
original
fidelity;
with
flexibility
to
run
variety
of
enterprise
workloads
including
batch
processing
and
interac0ve
SQL
-­‐
with
robust
security,
audi0ng
and
management.”

35. “An
Enterprise
Data
Hub
is
a
centralized
loca-on
to
store
all
data,
for
as
long
as
needed,
in
its
original
fidelity;
with
flexibility
to
run
variety
of
enterprise
workloads
including
batch
processing
and
interac0ve
SQL
-­‐
with
robust
security,
audi0ng
and
management.”

36. “An
Enterprise
Data
Hub
is
a
centralized
loca-on
to
store
all
data,
for
as
long
as
needed,
in
its
original
fidelity;
with
flexibility
to
run
variety
of
enterprise
workloads
including
batch
processing
and
interac0ve
SQL
-­‐
with
robust
security,
audi-ng
and
management.”

37. Mul8-­‐tenant
Secure
and
Compliant
Ac8ve
archive
of
all
data
Low
barrier
of
entry
Hadoop
as
a
Service
Data
Hub
is….

38. So
why
provide
HBase
as
a
service?

39. Maximize
cluster
u8liza8on
Minimum
resource
guarantee
sharing
Low
barrier
of
entry
HBase
as
a
Service
Knowledge
Data
Data

40. Requirements

41. Support
mul8-­‐tenant
environment
for
mul8ple
users
Isolated
deployment
Set
quota
per
consumer
ACLs
for
project
level
administra8on
Appropriate
security
for
user
authen8ca8on

42. Architecture

44. Smaller
HBase
clusters
part
of
the
bigger
“hub”
Scale
as
per
need
Easier
management

45. Resource
Management

46. Making
sure
that
you
have
sufficient
resources
before
you
start
the
job!

47. CGroups
• Control
Groups
• Police
and
limit
CPU,
Disk
I/O
and
Memory
usage
• Resource
guarantee
by
sta8c
par88oning
• Very
useful
in
case
of
conten8on

48. • Value
for
resource-­‐limits
driven
by
individual
tenants
depending
on
the
workload
• Defaults
to
70-­‐30
usage
for
memory
and
CPU
between
(NodeManager
+
Datanode)
and
RegionServer
• Higher
memory
on
RS
(at
least
4
GB)
helps
avoid
swapping
and
happier
GC
CPU
CPU
CPU
MEMORY
MEMORY
MEMORY
SEPSIS
M+
POP
HEALTH

49. What
we
learned
• Recommend
tenants
to
give
atleast
4
GB
of
memory
to
the
regionserver
for
smoother
opera8on
• Coopera8ve
memory
(ex:
via
JVM
heaps,
max.
container
sizes)
works
bejer
than
cgroup
limits
• Disable
swapping
on
HBase
nodes
• Disable
HDFS
load
balancer
• CMS
GC
performs
way
bejer
• Limit
number
of
containers
running
on
nodes
to
maximize
performance

50. Property
Value
yarn.nodemanager.container-­‐
executor.class
org.apache.hadoop.yarn.server.node
manager.LinuxContainerExecutor
yarn.nodemanager.linux-­‐container-­‐
executor.group
hadoop
yarn.nodemanager.linux-­‐container-­‐
executor.resources-­‐handler.class
org.apache.hadoop.yarn.server.node
manager.u8l.CgroupsLCEResourcesHa
ndler
yarn.nodemanager.linux-­‐container-­‐
executor.cgroups.hierarchy
/yarn
yarn.nodemanager.linux-­‐container-­‐
executor.cgroups.mount
true
hjps://hadoop.apache.org/docs/current/hadoop-­‐yarn/hadoop-­‐yarn-­‐site/NodeManagerCgroups.html

51. Request
queues
• Priori8ze
variety
of
workflows
that
need
access
to
HBase
• Important
to
meet
SLAs
for
tenants
• “FIFO”
vs
“Deadline”
queue
type
• “Deadline”(default)
proved
to
work
prejy
well
for
most
of
the
cases
• Cannot
be
set
per
tenant
but
has
to
be
set
per
cluster
–
support
coming
soon

52. Quotas
• Makes
sure
that
no
single
tenant
abuses
the
system
• Peaceful
coexistence
• Promotes
a
pay-­‐per-­‐use
model.
Could
buy
a
higher
throjle
limit
by
contribu8ng
more
number
of
nodes
• Usually
we
set
throjle
on
a
per
namespace
basis
but
could
also
set
on
the
per
table/user
basis
if
needed

53. Tenant
Isola8on

54. Namespace
• Logical
grouping
of
HBase
tables
• Analogous
to
databases
• Provides
tenants
with
individual
space
to
operate
on
• Tied
to
the
AD
group
used
when
onboarding
• Could
also
apply
quotas(max
regions/tables)
per
namespace(HBASE-­‐8410)
but
not
using
that
feature
for
now

55. Security

56. Authen8ca8on
• Cluster
secured
by
Kerberos
• Disallow
impersona8on
• Require
kinit
to
first
authen8cate
with
the
KDC
before
accessing
the
cluster

57. ACLs
• Provides
the
authoriza8on
piece
• Set
per
namespace
and
8ed
to
the
AD
groups
• Required
proper8es

58. ACLs
<ac0on>
-­‐
Determines
the
type
of
ac8on
–
grant
or
revoke
<en0ty>
-­‐
Determines
en8ty
to
grant
access
to
–
user
or
groups
<level>
-­‐
Determines
the
access
level
–
RWXCA
<scope>
-­‐
Determines
the
scope
for
access
–
namespace,
table,
column
family
or
Cell
Must
be
a
super
user
to
run
these
commands
(Determined
by
hbase.supersuser)

59. Example

60. ~33%
of
all
US
healthcare
data

61. Millennium
+
4
CDH
5.5.2
clusters
(soon
to
be
5)
1245
tables
113,982
total
regions
673
regionservers
895
TB
of
data!
(unreplicated)

62. Popula8on
Health
900
tables
2
million
requests/sec/day
780
regionservers
115,000
regions
700
TB
of
data!
(unreplicated)

63. Lots
of
common
data….
• Reference
Data
• Provider
informa8on
• Insurance
informa8on
• SNOMED/
ICD9
/
ICD10
data
• Ac8vity
Data
• Visits
• Procedures
• Vitals

64. Crawler
M
I
L
L
E
N
N
I
U
M
Collector
HBase
Cluster
Crawler
Collector
HBase
Cluster
M+
Pop.
H
Makes
sense
to
be
hosted
in
the
Data
hub
instead!

65. Onboarding
–
Capacity
planning
M+
Popula8on
Health

66. Onboarding
-­‐
Deployment
M+
Pop.
Health
DATA
HUB
Tenants
could
choose
to
modify
the
cgroup
configura8on
depending
on
expected
workloads
or
just
stay
with
defaults

67. Onboarding
-­‐
Isola8on
• Create
AD
groups
• poph_users,
pop_admins
• mplus_users,
mplus_admins
• Create
namespaces
(as
super
user)
hbase(main):001:0> create_namespace ’mplus'
0 row(s) in 0.5650 seconds
hbase(main):001:0> create_namespace ’poph'
0 row(s) in 0.7262 seconds

68. Onboarding
-­‐
Quotas
• Setup
quotas
hbase(main):001:0> set_quota TYPE => THROTTLE, NAMESPACE
=> ’mplus', LIMIT => ’10000 req/sec'
0 row(s) in 0.7255 seconds
hbase(main):001:0> set_quota TYPE => THROTTLE, NAMESPACE
=> ’poph', LIMIT => ’1500 req/sec'
0 row(s) in 0.5677 seconds

69. Onboarding
-­‐
Security
• Kerberos
secured.
• Require
kinit
for
regular
users
to
access
the
cluster
• Deploy
keytabs
for
service
users
• Setup
ACLs
hbase(main):001:0> grant '@poph_users', 'RWCX', ’@poph’
0 row(s) in 0.3250 seconds
hbase(main):001:0> grant '@poph_admins', 'RWACX', ’@poph’
0 row(s) in 0.4332 seconds

70. Future

71. What
we
can
do
bejer
• Namespace
quota
support
(HBASE-­‐8410)
• Limit
tables/regions
per
namespace
• Region
Server
Groups
(HBASE-­‐6721)
• Pin
namespace/tables
to
subset
of
regionservers
• Advanced
namespace
security
(HBASE-­‐9206)
• Higher
flexibility
to
admins
and
tenants
for
namespace
management

72. @CernerEng
hjp://engineering.cerner.com/