Data comes in all forms and shapes. Data also evolves as life and people adapt to new situations, and so should your database. When working with data, traditional relational database systems come to mind because that is how most of us have been trained. However, data is rarely homogeneous, and your database should not force you into a certain schema if your data is not relational. During this talk we analyse the composition of "documents" in the context of a document-based database, and cover the basic principles of Map-Reduce and its potential use in the context of computational statistics. What then happens when the amount of data you have no longer fits on 1 server? How easy is it for your favourite database to currently expand and adapt to your new growing requirements? What is your contingency plan if your server goes down? We then go over some of the features that CouchDB, Riak and MongoDB provide you with, alongside some of David's personal opinions. This is an intermediary talk. Listeners should have a working concept of Bayesian statistics, standard internet protocols as such as HTTP, and a minimum understanding of programming languages as such as JavaScript and Erlang as some of the examples for those database are using those languages.

1. The Artful Business
of Data Mining
Distributed Schema-less
Document-Based Databases
Wednesday 27 March 13

2. David Coallier
@davidcoallier
Wednesday 27 March 13

3. Data Scientist
At Engine Yard (.com)
Wednesday 27 March 13

4. RDBMs
Wednesday 27 March 13

5. Structure
Restrictions
Safety
Wednesday 27 March 13

6. id name age address
1 david 1 315
2 divad 3 51
3 foo 41 31
4 bar 42 98
5 john 3315 85
6 jack 4 11
7 jill 8 66
... ... ... ...
Wednesday 27 March 13

7. id name age address
1 david 1 315
2 divad 3 51
3 foo 41 31
4 bar 42 98
5 john 3315 85
6 jack 4 11
7 jill 8 66
... ... ... ...
Wednesday 27 March 13

8. id name age address
1 david 1 315
2 divad 3 51
3 foo 41 31
4 bar 42 98
5 john 3315 85
6 jack 4 11
7 jill 8 66
... ... ... ...
Wednesday 27 March 13

9. id name age address
1 david 1 315
2 divad 3 51
3 foo 41 31
4 bar 42 98
5 john 3315 85
6 jack 4 11
7 jill 8 66
... ... ... ...
Wednesday 27 March 13

10. id name age address
1 david 1 315
2 divad 3 51
3 foo 41 31
4 bar 42 98
5 john 3315 85
6 jack 4 11
7 jill 8 66
... ... ... ...
Wednesday 27 March 13

11. What If?
Wednesday 27 March 13

12. id name age address phone
1 david 26 IE 353
2 divad 27 US 1
3 foo 42 IE 353
4 bar 31 CA 1
5 john 17 NZ 131
6 jack 128 DK 311
7 jill 21 IE 353
... ... ... ... ...
Wednesday 27 March 13

13. Before
Moving on
Wednesday 27 March 13

14. JSON
Wednesday 27 March 13

15. What is JSON?
Wednesday 27 March 13

16. {
"firstName": "David",
"lastName": "Coallier",
"age": 26,
"address": {
"streetAddress": "Mansfield House",
"city": "Crosshaven",
},
"phoneNumbers": [
{
"type": "mobile",
"number": "0863299999"
}
]
}
Wednesday 27 March 13

17. What is HTTP?
Wednesday 27 March 13

18. What is a Schema?
Wednesday 27 March 13

19. Alternative
Wednesday 27 March 13

20. Schema-less
Wednesday 27 March 13

21. Does
NOT
Mean
Structure-less
Wednesday 27 March 13

22. Documents
and
K-V Buckets
Wednesday 27 March 13

23. CouchDB
Cluster of unreliable commodity hardware
Wednesday 27 March 13

24. Replication Attachments
Generated “random” ids
Dictionary Revisions?
JSON Objects
HTTP CRUD
Wednesday 27 March 13

25. Documents
Wednesday 27 March 13

26. Wednesday 27 March 13

27. {
"_id": "131dafsd1vasd",
"_rev": "12-fva32asdf",
"firstName": "David",
"lastName": "Coallier",
"age": 26,
"address": {
"streetAddress": "Mansfield House",
"city": "Crosshaven",
},
"phoneNumbers": [
{
"type": "mobile",
"number": "0863299999"
}
]
}
Wednesday 27 March 13

28. How do you
find
Anything?
Wednesday 27 March 13

29. Map/Reduce
Wednesday 27 March 13

30. ...
Wednesday 27 March 13

31. Riak
Wednesday 27 March 13

32. Dynamo
Paper
Wednesday 27 March 13

33. CAP
Theorem
Wednesday 27 March 13

34. Key-Value
Buckets
Wednesday 27 March 13

35. Differences?
Wednesday 27 March 13

36. CouchDB Riak
Storage Model append-only bitcask
Access HTTP HTTP, PB
Retrieval Views(M/R) M/R, Indexes, Search
Versioning Eventual Consistency Vector Clocks
Concurrency No Locking Client Resolution
Replication master/master/slave replication, clustering
Scaling In/Out Big Couch Built-in
Management Futon/Fuxton Riak Control
http://guide.couchdb.org http://downloads.basho.com/papers/bitcask-intro.pdf
Wednesday 27 March 13

37. Map/Reduce
Wednesday 27 March 13

38. Mapper:
Executed on document
Reducer:
Receives output from mappers
Wednesday 27 March 13

39. {
{ { {
"_id": "...",
"_id": "...", "_id": "...", "_id": "...",
"_rev": "...",
"_rev": "...", "_rev": "...", "_rev": "...",
"age": "32",
"age": "26" "age": "42" "age": "17"
"heads": "3",
} } }
}
Wednesday 27 March 13

40. {
{ { {
"_id": "...",
"_id": "...", "_id": "...", "_id": "...",
"_rev": "...",
"_rev": "...", "_rev": "...", "_rev": "...",
"age": "32",
"age": "26" "age": "42" "age": "17"
"heads": "3",
} } }
}
Wednesday 27 March 13

41. {
"age": "32",
"heads": "3",
}
Wednesday 27 March 13

42. Map: find-ages
{
{ { {
"_id": "...",
"_id": "...", "_id": "...", "_id": "...",
"_rev": "...",
"_rev": "...", "_rev": "...", "_rev": "...",
"age": "32",
"age": "26" "age": "42" "age": "17"
"heads": "3",
} } }
}
Wednesday 27 March 13

43. Map: find-ages
function find_ages(doc) {
if (typeof(doc.age) != undefined) {
emit(doc._id, doc.age);
}
}
Wednesday 27 March 13

44. Map: find-ages
{
{ { {
"_id": "...",
"_id": "...", "_id": "...", "_id": "...",
"_rev": "...",
"_rev": "...", "_rev": "...", "_rev": "...",
"age": "32",
"age": "26" "age": "42" "age": "17"
"heads": "3",
} } }
}
Wednesday 27 March 13

45. Map: find-ages
{
{ { {
"_id": "...",
"_id": "...", "_id": "...", "_id": "...",
"_rev": "...",
"_rev": "...", "_rev": "...", "_rev": "...",
"age": "32",
"age": "26" "age": "42" "age": "17"
"heads": "3",
} } }
}
26 32 42 17
Wednesday 27 March 13

46. Map: find-ages
26 32 42 17
Reduce: sum
Wednesday 27 March 13

47. Reduce: sum
function sum(values) {
return sum(values);
}
Wednesday 27 March 13

48. Map: find-ages
26 32 42 17
Reduce: sum
117
Wednesday 27 March 13

49. Mapper:
Executed on document
Reducer:
Receives output from mappers
Wednesday 27 March 13

50. So
What?
Wednesday 27 March 13

51. The
Machines
They Lurn.
Wednesday 27 March 13

52. The
Problem
Wednesday 27 March 13

53. Statistics
Example
Wednesday 27 March 13

54. Mean,
Std. Deviation
Age
Wednesday 27 March 13

55. n
1
µ = ∑ xi
n i=1
Wednesday 27 March 13

56. n
1
σ= ∑
n i=1
(xi − µ ) 2
Wednesday 27 March 13

57. Mapper:
Executed on document
Reducer:
Receives output from mappers
Wednesday 27 March 13

58. Mapper:
Retrieve values, pre-process
Reducer:
Receive, process further.
Wednesday 27 March 13

59. {
{ { {
"_id": "...",
"_id": "...", "_id": "...", "_id": "...",
"_rev": "...",
"_rev": "...", "_rev": "...", "_rev": "...",
"age": "32",
"age": "26" "age": "42" "age": "17"
"heads": "3",
} } }
}
Wednesday 27 March 13

60. [
[ 26, 676],
[ 32, 1024],
[ 42, 1764],
[ 17, 289 ]
]
Wednesday 27 March 13

61. /**
* Our mapper function.
*/
map: function(doc) {
emit(null, [doc.age, doc.age * doc.age]);
}
/**
* Our reducer...
*/
reduce: function(keys, values, rereduce) {
var N = 0;
var summed = 0;
var summedSquare = 0;
for (var i in values) {
N += 1;
summed += values[i][0];
summedSquare += values[i][1];
}
var mean = summed / N;
var standard_deviation = Math.sqrt(
(summedSquare / N) - (mean* mean)
)
return [mean, standard_deviation]
}
Wednesday 27 March 13

62. /**
* Our mapper function.
*/
map: function(doc) {
emit(null, [doc.age, doc.age * doc.age]);
}
/**
* Our reducer...
*/
reduce: function(keys, values, rereduce) {
var N = values.length;
var summed = sum(values.map(function(v) { return v[0]; }));
var summedSquares = sum(values.map(function(v) { return v[1];}));
var mean = summed / N;
var standard_deviation = Math.sqrt(
(summedSquares / N) - (mean*mean)
)
return [mean, standard_deviation]
}
Wednesday 27 March 13

63. Naive
Bayes
Wednesday 27 March 13

64. Real Life
Fraud
Wednesday 27 March 13

65. P(x j = k | y = fraudulent)
P(x j = k | y = normal)
P(y)
Wednesday 27 March 13

66. We need to:
Sum x j = k , for each y
to calculate P(x|y)
Wednesday 27 March 13

67. We need:
More than 1 mapper.
Wednesday 27 March 13

68. We need
4
mappers
Wednesday 27 March 13

69. Mapper #1:
∑1i P(x = k | y = fraudulent)
j
Wednesday 27 March 13

70. Mapper #2:
∑1i P(x = k | y = normal)
j
Wednesday 27 March 13

71. Mapper #3:
∑1i P(y = fraudulent)
Wednesday 27 March 13

72. Mapper #4:
∑1i P(y = normal)
Wednesday 27 March 13

73. Reducer
Sums up
results for
parameters
Wednesday 27 March 13

74. Cluster
Analysis
Wednesday 27 March 13

75. k-means
Wednesday 27 March 13

76. Mapper:
Divide vectors into subgroups,
Calculate d(p,q) between
vectors, find centroids,
sum them up.
Reducer:
Sum up the sums,
get new centroids.
Wednesday 27 March 13