1. Introducing
Riak
Kevin A. Smith
Senior Developer
Basho Technologies

2. What Is Riak?

3. What Is Riak?
• Key/Value store

4. What Is Riak?
• Key/Value store
• Document-oriented database

5. What Is Riak?
• Key/Value store
• Document-oriented database
• Web-shaped storage

6. Key/Value Store

7. Key/Value Store
• Data organized by bucket/key pairs

8. Key/Value Store
• Data organized by bucket/key pairs
• Simple REST API (GET, PUT, DELETE)

9. Document Store

10. Document Store
• Store values as JSON

11. Document Store
• Store values as JSON
• Many clients support automatic JSON
encoding/decoding

12. Document Store
• Store values as JSON
• Many clients support automatic JSON
encoding/decoding
• Javascript Map/Reduce built on top of JSON

13. Web-Shaped
Storage

14. Web-Shaped
Storage
• Content neutral

15. Web-Shaped
Storage
• Content neutral
• Highly distributed

16. Web-Shaped
Storage
• Content neutral
• Highly distributed
• Replicated

17. Web-Shaped
Storage
• Content neutral
• Highly distributed
• Replicated
• Fault-tolerant

18. What Is Riak?

19. What Is Riak?
A flexible storage engine...

20. What Is Riak?
A flexible storage engine...
...with a REST API...

21. What Is Riak?
A flexible storage engine...
...with a REST API...
...and map/reduce capability...

22. What Is Riak?
A flexible storage engine...
...with a REST API...
...and map/reduce capability...
....designed to be fault-tolerant...

23. What Is Riak?
A flexible storage engine...
...with a REST API...
...and map/reduce capability...
....designed to be fault-tolerant...
...distributed...

24. What Is Riak?
A flexible storage engine...
...with a REST API...
...and map/reduce capability...
....designed to be fault-tolerant...
...distributed...
...and ops friendly

25. Influences

26. Influences
• CAP Theorem

27. Influences
• CAP Theorem
• Amazon’s Dynamo Paper

28. Influences
• CAP Theorem
• Amazon’s Dynamo Paper
• Experience running large networks
(Akamai)

29. CAP Theorem

30. CAP Theorem
Consistent Reads and writes reflect a
globally consistent system state

31. CAP Theorem
Consistent Reads and writes reflect a
globally consistent system state

32. CAP Theorem
Consistent Reads and writes reflect a
globally consistent system state
Available System is available for reads and
writes

33. CAP Theorem
Consistent Reads and writes reflect a
globally consistent system state
Available System is available for reads and
writes

34. CAP Theorem
Consistent Reads and writes reflect a
globally consistent system state
Available System is available for reads and
writes
Partition Tolerant System can handle
the failure of individual parts

35. Common Wisdom

36. Common Wisdom
Pick two.

37. The Riak Way

38. The Riak Way
Pick Two.

39. The Riak Way
Pick Two.
For each operation.

40. Dynamo
Influences

41. Dynamo
Influences
• N = The number of replicas

42. Dynamo
Influences
• N = The number of replicas
• R = The number of replicas needed for a
successful read

43. Dynamo
Influences
• N = The number of replicas
• R = The number of replicas needed for a
successful read
• W = The number of replicas needed for a
successful write

44. Dynamo Math

45. Dynamo Math
N - R = read fault tolerance

46. Dynamo Math
N - R = read fault tolerance
N - W = write fault tolerance

47. Dynamo Math

48. Dynamo Math
N = 4, W = 2, R = 1

49. Dynamo Math
N = 4, W = 2, R = 1

50. Dynamo Math
N = 4, W = 2, R = 1
4 - 2 = 2 hosts can be down and Riak can
still perform writes.

51. Dynamo Math
N = 4, W = 2, R = 1
4 - 2 = 2 hosts can be down and Riak can
still perform writes.
4 - 1 = 3 hosts can be down and Riak can
still perform reads.

52. Riak Improvements

53. Riak Improvements
• N can vary per bucket

54. Riak Improvements
• N can vary per bucket
• R and W can vary per operation

55. Riak Improvements
• N can vary per bucket
• R and W can vary per operation
Choose your own fault tolerance/performance tradeoff

56. Consistent Hashing
2160 0
node 0
node 1
2160/4
node 2
node 3
hash(<<"artist">>,<<"REM">>)
2160/2

57. R value
get(<<"artist">>,<<"REM">>,
R=2)
(N=3)
{ok, Object}
X

58. W value
put(<<"artist">>,<<"REM">>,
W=2)
(N=3)
ok
X

59. N=10, R/W=2
get/put("artist", "REM",
R/W=2)
(N=10)
{ok, Object}
X X
X
X X
X X X

60. Resolving Conflicts

61. Resolving Conflicts
• Riak focuses on the AP of CAP

62. Resolving Conflicts
• Riak focuses on the AP of CAP
• Data could be briefly inconsistent

63. Resolving Conflicts
• Riak focuses on the AP of CAP
• Data could be briefly inconsistent
• Inconsistency must be resolved

64. Detecting & Resolving
Conflicts
0 1
Object
v0
2 3

65. Detecting & Resolving
Conflicts
Object
0 1
v0
Object
2 3
v0

66. Detecting & Resolving
Conflicts
Object
0 1
v1
Object
2 3
v0

67. Detecting & Resolving
Conflicts
0 1
Object
v1
2 3

68. Detecting & Resolving
Conflicts
Object
0 1
v1
Object
2 3
v1

69. Client Resolution

70. Client Resolution
• Can be set per-bucket or server-wide

71. Client Resolution
• Can be set per-bucket or server-wide
• Conflicting data is “bubbled up” to the
client

72. Client Resolution
• Can be set per-bucket or server-wide
• Conflicting data is “bubbled up” to the
client
• Client picks the winner

73. Server Resolution

74. Server Resolution
• “Last write wins”

75. Server Resolution
• “Last write wins”
• Enabled by default

76. Server Resolution
• “Last write wins”
• Enabled by default
• What most apps need 80% of the time

77. Live Demo!

78. Linking Objects

79. Linking Objects
• Objects can store pointers, or links, to
other objects

80. Linking Objects
• Objects can store pointers, or links, to
other objects
• Doesn’t have to be the same bucket

81. Linking Objects
• Objects can store pointers, or links, to
other objects
• Doesn’t have to be the same bucket
• Object links described in a Link header

82. Link Header Format
Object URL
</riak/demo/test1>; riaktag="userinfo"
Link tag

83. Link Walking

84. Link Walking
• Ask Riak to “walk” a sequence of links

85. Link Walking
• Ask Riak to “walk” a sequence of links
• Optionally, collect objects along the walk
and return them

86. Link Walking
• Ask Riak to “walk” a sequence of links
• Optionally, collect objects along the walk
and return them
• Can be arbitrarily deep

87. Link Walking Examples

88. Link Walking Examples
/riak/demo/test1/_,_,1

89. Link Walking Examples
/riak/demo/test1/_,_,1
Start walking at /demo/test1 and return all
linked objects

90. Link Walking Examples

91. Link Walking Examples
/riak/demo/test1/demo,_,1

92. Link Walking Examples
/riak/demo/test1/demo,_,1
Start walking at /demo/test1 and return all
linked objects contained in the demo bucket

93. Link Walking Examples

94. Link Walking Examples
/riak/demo/test1/_,_,0/_,_,1

95. Link Walking Examples
/riak/demo/test1/_,_,0/_,_,1
Start walking at /demo/test1, find any linked objects,
then find and return any objects linked to those

96. Link Walking Examples

97. Link Walking Examples
/riak/demo/test1/_,child,0/_,_,1

98. Link Walking Examples
/riak/demo/test1/_,child,0/_,_,1
Start walking at /demo/test1, find any linked objects
with the link tag “child”, then find and return any objects
linked to those

99. Map/Reduce
Terms

100. Map/Reduce
Terms
• Phase: A step within a job

101. Map/Reduce
Terms
• Phase: A step within a job
• Job: A sequence of phases and inputs

102. Map/Reduce
Terms
• Phase: A step within a job
• Job: A sequence of phases and inputs
• Map: Data collection phase

103. Map/Reduce
Terms
• Phase: A step within a job
• Job: A sequence of phases and inputs
• Map: Data collection phase
• Reduce: Data collation or processing
phase

104. Map/Reduce
Overview

105. Map/Reduce
Overview
• Map phases execute in parallel w/data
locality

106. Map/Reduce
Overview
• Map phases execute in parallel w/data
locality
• Reduce phases execute in parallel on the
node where job was submitted

107. Map/Reduce
Overview
• Map phases execute in parallel w/data
locality
• Reduce phases execute in parallel on the
node where job was submitted
• Results are not cached or stored

108. Map/Reduce
Overview
• Map phases execute in parallel w/data
locality
• Reduce phases execute in parallel on the
node where job was submitted
• Results are not cached or stored
• Phases can be written in Erlang or
Javascript

109. Map Phase

110. Map Phase
• Inputs must be bucket/key pairs

111. Map Phase
• Inputs must be bucket/key pairs
• Must return a list

112. Map Phase
• Inputs must be bucket/key pairs
• Must return a list
• Parallel results are aggregated into a single
list

113. Parallel Map

114. Parallel Map

115. Parallel Map

116. Parallel Map

117. Erlang Map Phase

118. Erlang Map Phase
• Two types: modfun and qfun

119. Erlang Map Phase
• Two types: modfun and qfun
• modfuns reference the module and name
of the Erlang function to call

120. Erlang Map Phase
• Two types: modfun and qfun
• modfuns reference the module and name
of the Erlang function to call
• qfuns are anonymous Erlang functions*

121. Erlang Map Phase
• Two types: modfun and qfun
• modfuns reference the module and name
of the Erlang function to call
• qfuns are anonymous Erlang functions*
*Must be on the server-side codepath

122. Erlang Map Phase

123. Erlang Map Phase
map_object_value(Obj, _KeyData, _Arg) ->

124. Erlang Map Phase
map_object_value(Obj, _KeyData, _Arg) ->
[riak_object:get_value(Obj)].

125. Erlang Map Phase
map_object_value(Obj, _KeyData, _Arg) ->
[riak_object:get_value(Obj)].

126. Erlang Map Phase
map_object_value(Obj, _KeyData, _Arg) ->
[riak_object:get_value(Obj)].
• Obj:riak_object retrieved from bucket/key

127. Erlang Map Phase
map_object_value(Obj, _KeyData, _Arg) ->
[riak_object:get_value(Obj)].
• Obj:riak_object retrieved from bucket/key
• KeyData: Static argument specified with the bucket/
key

128. Erlang Map Phase
map_object_value(Obj, _KeyData, _Arg) ->
[riak_object:get_value(Obj)].
• Obj:riak_object retrieved from bucket/key
• KeyData: Static argument specified with the bucket/
key
• Arg: Static argument specified with the phase

129. Erlang Map
Built-Ins

130. Erlang Map
Built-Ins
riak_mapreduce:map_object_value/3

131. Erlang Map
Built-Ins
riak_mapreduce:map_object_value/3
• Returns object value wrapped in a list

132. Erlang Map
Built-Ins
riak_mapreduce:map_object_value/3
• Returns object value wrapped in a list
riak_mapreduce:map_object_value_list/3

133. Erlang Map
Built-Ins
riak_mapreduce:map_object_value/3
• Returns object value wrapped in a list
riak_mapreduce:map_object_value_list/3
• Returns object value. Object value must already
be a list

134. Javascript
Map Phase

135. Javascript
Map Phase
• Two types: jsanon and jsfun

136. Javascript
Map Phase
• Two types: jsanon and jsfun
• jsanons are anonymous JS functions:

137. Javascript
Map Phase
• Two types: jsanon and jsfun
• jsanons are anonymous JS functions:
function(value) { return [value]; }

138. Javascript
Map Phase
• Two types: jsanon and jsfun
• jsanons are anonymous JS functions:
function(value) { return [value]; }
• jsfuns are named JS functions:

139. Javascript
Map Phase
• Two types: jsanon and jsfun
• jsanons are anonymous JS functions:
function(value) { return [value]; }
• jsfuns are named JS functions:
Riak.mapValuesJson

140. Erlang & Javascript

141. Erlang & Javascript
• Same environment as Firefox minus
browser bits

142. Erlang & Javascript
• Same environment as Firefox minus
browser bits
• Erlang to Javascript data is JSON encoded

143. Erlang & Javascript
• Same environment as Firefox minus
browser bits
• Erlang to Javascript data is JSON encoded
• Javascript to Erlang data is JSON decoded

144. Javascript Map Phase

145. Javascript Map Phase
function(value, keyData, arg)

146. Javascript Map Phase
function(value, keyData, arg)

147. Javascript Map Phase
function(value, keyData, arg)
• value: JSON-encoded version of
riak_object

148. Javascript Map Phase
function(value, keyData, arg)
• value: JSON-encoded version of
riak_object
• keyData: Same as Erlang

149. Javascript Map Phase
function(value, keyData, arg)
• value: JSON-encoded version of
riak_object
• keyData: Same as Erlang
• arg: Same as Erlang

150. Javascript Map
Built-Ins

151. Javascript Map
Built-Ins
Riak.mapValues

152. Javascript Map
Built-Ins
Riak.mapValues
• Returns object values. Handles detecting
when/if to use list wrapping.

153. Javascript Map
Built-Ins
Riak.mapValues
• Returns object values. Handles detecting
when/if to use list wrapping.
Riak.mapValuesJson

154. Javascript Map
Built-Ins
Riak.mapValues
• Returns object values. Handles detecting
when/if to use list wrapping.
Riak.mapValuesJson
• Returns JSON parsed object values. Also
performs list wrapping, if needed.

155. Reduce Phase

156. Reduce Phase
• Performed on the node coordinating the
map/reduce job

157. Reduce Phase
• Performed on the node coordinating the
map/reduce job
• Two processes per reduce phase to add
minor parallelism

158. Reduce Phase
• Performed on the node coordinating the
map/reduce job
• Two processes per reduce phase to add
minor parallelism
• Must return a list

159. Erlang Reduce
Built-Ins

160. Erlang Reduce
Built-Ins
riak_mapreduce:reduce_set_union/2

161. Erlang Reduce
Built-Ins
riak_mapreduce:reduce_set_union/2
• Returns unique set of values

162. Erlang Reduce
Built-Ins
riak_mapreduce:reduce_set_union/2
• Returns unique set of values
riak_mapreduce:reduce_sum/2

163. Erlang Reduce
Built-Ins
riak_mapreduce:reduce_set_union/2
• Returns unique set of values
riak_mapreduce:reduce_sum/2
• Returns the sum of inputs

164. Erlang Reduce
Built-Ins
riak_mapreduce:reduce_set_union/2
• Returns unique set of values
riak_mapreduce:reduce_sum/2
• Returns the sum of inputs
riak_mapreduce:reduce_sort/2

165. Erlang Reduce
Built-Ins
riak_mapreduce:reduce_set_union/2
• Returns unique set of values
riak_mapreduce:reduce_sum/2
• Returns the sum of inputs
riak_mapreduce:reduce_sort/2
• Returns the sorted list of inputs

166. Javascript Reduce
Built-Ins

167. Javascript Reduce
Built-Ins
Riak.reduceMin

168. Javascript Reduce
Built-Ins
Riak.reduceMin
• Returns the minimum value of the input set

169. Javascript Reduce
Built-Ins
Riak.reduceMin
• Returns the minimum value of the input set
Riak.reduceMax

170. Javascript Reduce
Built-Ins
Riak.reduceMin
• Returns the minimum value of the input set
Riak.reduceMax
• Returns the maximum value of the input set

171. Javascript Reduce
Built-Ins
Riak.reduceMin
• Returns the minimum value of the input set
Riak.reduceMax
• Returns the maximum value of the input set
Riak.reduceSort

172. Javascript Reduce
Built-Ins
Riak.reduceMin
• Returns the minimum value of the input set
Riak.reduceMax
• Returns the maximum value of the input set
Riak.reduceSort
• Returns a sorted list of the input set

173. Building
M/R Job

174. Building
M/R Job
• Job is a list of phases and starting inputs

175. Building
M/R Job
• Job is a list of phases and starting inputs
• Each phase can:

176. Building
M/R Job
• Job is a list of phases and starting inputs
• Each phase can:
• Receive a static argument

177. Building
M/R Job
• Job is a list of phases and starting inputs
• Each phase can:
• Receive a static argument
• Accumulate and return results

178. Submitting Jobs
via HTTP

179. Submitting Jobs
via HTTP
• Riak exposes M/R via its REST API

180. Submitting Jobs
via HTTP
• Riak exposes M/R via its REST API
• Job is described in JSON

181. Submitting Jobs
via HTTP
• Riak exposes M/R via its REST API
• Job is described in JSON
• Submitted via POST

182. Submitting Jobs
via HTTP
• Riak exposes M/R via its REST API
• Job is described in JSON
• Submitted via POST
• Default URL is /mapred

183. Erlang Phase
(JSON)

184. Erlang Phase
(JSON)
{Type:{“language”:”erlang”, “module”: Module,

185. Erlang Phase
(JSON)
{Type:{“language”:”erlang”, “module”: Module,
“function”: Function, “keep”:Flag}}

186. Erlang Phase
(JSON)
{Type:{“language”:”erlang”, “module”: Module,
“function”: Function, “keep”:Flag}}

187. Erlang Phase
(JSON)
{Type:{“language”:”erlang”, “module”: Module,
“function”: Function, “keep”:Flag}}
• Type: “map” or “reduce”

188. Erlang Phase
(JSON)
{Type:{“language”:”erlang”, “module”: Module,
“function”: Function, “keep”:Flag}}
• Type: “map” or “reduce”
• Module: String name of Erlang module

189. Erlang Phase
(JSON)
{Type:{“language”:”erlang”, “module”: Module,
“function”: Function, “keep”:Flag}}
• Type: “map” or “reduce”
• Module: String name of Erlang module
• Function: String name of Erlang function

190. Erlang Phase
(JSON)
{Type:{“language”:”erlang”, “module”: Module,
“function”: Function, “keep”:Flag}}
• Type: “map” or “reduce”
• Module: String name of Erlang module
• Function: String name of Erlang function
• Flag: Boolean accumulation toggle

191. Javascript Phase
(JSON)

192. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,

193. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,
“source”: Source,“keep”:Flag}}

194. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,
“source”: Source,“keep”:Flag}}

195. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,
“source”: Source,“keep”:Flag}}
• Type: “map” or “reduce”

196. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,
“source”: Source,“keep”:Flag}}
• Type: “map” or “reduce”
• Source: Source for anonymous function

197. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,
“source”: Source,“keep”:Flag}}
• Type: “map” or “reduce”
• Source: Source for anonymous function
• Flag: Boolean accumulation toggle

198. Javascript Phase
(JSON)

199. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,

200. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,
“name”:Name,“keep”:Flag}}

201. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,
“name”:Name,“keep”:Flag}}

202. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,
“name”:Name,“keep”:Flag}}
• Type: “map” or “reduce”

203. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,
“name”:Name,“keep”:Flag}}
• Type: “map” or “reduce”
• Name: String name of Javascript function

204. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,
“name”:Name,“keep”:Flag}}
• Type: “map” or “reduce”
• Name: String name of Javascript function
• Flag: Boolean accumulation toggle

205. Putting It
Together
{“inputs”: [[“stocks”, “goog”]],
“query”: [{“map”:{“language”:”javascript”,
“name”: “Riak.mapValuesJson”},
“keep”: true}]}

206. Putting It
Together
{“inputs”: [[“stocks”, “goog”],
[“stocks”, “csco”]],
“query”: [{“map”:{“language”:”javascript”,
“name”: “Riak.mapValuesJson”},
“keep”: true}]}

207. Putting It
Together
{“inputs”: “stocks”,
“query”: [{“map”:{“language”:”javascript”,
“name”: “App.extractTickers”,
“arg”: “GOOG”},
“keep”: false},
{“reduce”:{“language”:”javascript,
“name”: “Riak.reduceMin”},
“keep”: true}]}

208. Live Demo!

209. Thank You
Kevin A. Smith
Email: ksmith@basho.com
Twitter: @kevsmith