Google’s original use case for BigTable was the storage and processing of web graph information, represented as sparse matrices. However, many organizations tend to treat HBase as merely a “web scale” RDBMS. This session will cover several use cases for storing graph data in HBase, including social networks and web link graphs, MapReduce processes like cached traversal, PageRank, and clustering and lastly will look at some lower-level modeling details like row key and column qualifier design, using FullContact’s graph processing systems as a real-world use case.

1. Storing and Manipulating Graphs
in HBase
Dan Lynn
dan@fullcontact.com
@danklynn

2. Keeps Contact Information Current and Complete
Based in Denver, Colorado
CTO & Co-Founder

3. Turn Partial Contacts
Into Full Contacts

4. Refresher: Graph Theory

5. Refresher: Graph Theory

6. Refresher: Graph Theory
rt ex
Ve

7. Refresher: Graph Theory
Edg
e

8. Social Networks

9. Tweets
@danklynn
retweeted
“#HBase rocks”
follows
author
@xorlev

10. Web Links
http://fullcontact.com/blog/
<a href=”...”>TechStars</a>
http://techstars.com/

11. Why should you care?
Vertex Influence
- PageRank
- Social Influence
- Network bottlenecks
Identifying Communities

12. Storage Options

13. neo4j

14. neo4j
Very expressive querying
(e.g. Gremlin)

15. neo4j
Transactional

16. neo4j
Data must fit on a
single machine
:-(

17. FlockDB

18. FlockDB
Scales horizontally

19. FlockDB
Very fast

20. FlockDB
No multi-hop query support
:-(

21. RDBMS
(e.g. MySQL, Postgres, et al.)

22. RDBMS
Transactional

23. RDBMS
Huge amounts of JOINing
:-(

25. HBase
Massively scalable

26. HBase
Data model well-suited

27. HBase
Multi-hop querying?

28. Modeling
Techniques

29. Adjacency Matrix
1
3
2

30. Adjacency Matrix
1 2 3
1 0 1 1
2 1 0 1
3 1 1 0

31. Adjacency Matrix
Can use vectorized libraries

32. Adjacency Matrix
Requires O(n2) memory
n = number of vertices

33. Adjacency Matrix
Hard(er) to distribute

34. Adjacency List
1
3
2

35. Adjacency List
1 2,3
2 1,3
3 1,2

36. Adjacency List Design in HBase
e:dan@fullcontact.com
p:+13039316251
t:danklynn

37. Adjacency List Design in HBase
row key “edges” column family
e:dan@fullcontact.com p:+13039316251= ...
t:danklynn= ...
p:+13039316251 e:dan@fullcontact.com= ...
t:danklynn= ...
t:danklynn e:dan@fullcontact.com= ...
p:+13039316251= ...

38. Adjacency List Design in HBase
row key “edges” column family
e:dan@fullcontact.com p:+13039316251= ...
t:danklynn= ...
at to
W e?h
p:+13039316251 e:dan@fullcontact.com= ...
st or
t:danklynn= ...
t:danklynn e:dan@fullcontact.com= ...
p:+13039316251= ...

39. Custom Writables
package org.apache.hadoop.io;
public interface Writable {
void write(java.io.DataOutput dataOutput);
void readFields(java.io.DataInput dataInput);
}
java

40. Custom Writables
class EdgeValueWritable implements Writable {
EdgeValue edgeValue
void write(DataOutput dataOutput) {
dataOutput.writeDouble edgeValue.weight
}
void readFields(DataInput dataInput) {
Double weight = dataInput.readDouble()
edgeValue = new EdgeValue(weight)
}
// ...
}
groovy

41. Don’t get fancy with byte[]
class EdgeValueWritable implements Writable {
EdgeValue edgeValue
byte[] toBytes() {
// use strings if you can help it
}
static EdgeValueWritable fromBytes(byte[] bytes) {
// use strings if you can help it
}
}
groovy

42. Querying by vertex
def get = new Get(vertexKeyBytes)
get.addFamily(edgesFamilyBytes)
Result result = table.get(get);
result.noVersionMap.each {family, data ->
// construct edge objects as needed
// data is a Map<byte[],byte[]>
}

43. Adding edges to a vertex
def put = new Put(vertexKeyBytes)
put.add(
edgesFamilyBytes,
destinationVertexBytes,
edgeValue.toBytes() // your own implementation here
)
// if writing directly
table.put(put)
// if using TableReducer
context.write(NullWritable.get(), put)

44. Distributed Traversal / Indexing
e:dan@fullcontact.com
p:+13039316251
t:danklynn

45. Distributed Traversal / Indexing
e:dan@fullcontact.com
p:+13039316251
t:danklynn

46. Distributed Traversal / Indexing
e:dan@fullcontact.com
p:+13039316251
Pi v
ot v
e rt
ex
t:danklynn

47. Distributed Traversal / Indexing
e:dan@fullcontact.com
p:+13039316251
Ma pReduce ove r
out bou nd edges
t:danklynn

48. Distributed Traversal / Indexing
e:dan@fullcontact.com
p:+13039316251
Em it vertexes an d edge
dat a gro upe d by
the piv ot t:danklynn

49. Distributed Traversal / Indexing
Re duc e key p:+13039316251
“Ou t” vertex
e:dan@fullcontact.com
t:danklynn
“In” vertex

50. Distributed Traversal / Indexing
e:dan@fullcontact.com t:danklynn
Re duc er em its higher-order edge

51. Distributed Traversal / Indexing
Ite rat ion 0

52. Distributed Traversal / Indexing
Ite rat ion 1

53. Distributed Traversal / Indexing
Ite rat ion 2

54. Distributed Traversal / Indexing
Reuse edges created
during previ ous
iterat ions
Ite rat ion 2

55. Distributed Traversal / Indexing
Ite rat ion 3

56. Distributed Traversal / Indexing
Reuse edges created
during previ ous
iterat ions
Ite rat ion 3

57. Distributed Traversal / Indexing
hop s req uires on ly
ite rat ion s

58. Tips / Gotchas

59. Do implement your own comparator
public static class Comparator
extends WritableComparator {
public int compare(
byte[] b1, int s1, int l1,
byte[] b2, int s2, int l2) {
// .....
}
}
java

60. Do implement your own comparator
static {
WritableComparator.define(VertexKeyWritable,
new VertexKeyWritable.Comparator())
}
java

61. MultiScanTableInputFormat
MultiScanTableInputFormat.setTable(conf,
"graph");
MultiScanTableInputFormat.addScan(conf,
new Scan());
job.setInputFormatClass(
MultiScanTableInputFormat.class);
java

62. TableMapReduceUtil
TableMapReduceUtil.initTableReducerJob(
"graph", MyReducer.class, job);
java

63. Elastic
MapReduce

64. Elastic MapReduce
HFi les

65. Elastic MapReduce
HFi les
Copy to S3
Seq uen ceFiles

66. Elastic MapReduce
HFi les
Copy to S3
Elastic MapReduce
Seq uen ceFiles Seq uen ceFiles

67. Elastic MapReduce
HFi les
Copy to S3
Elastic MapReduce
Seq uen ceFiles Seq uen ceFiles

68. Elastic MapReduce
HFi les
Copy to S3
Elastic MapReduce
Seq uen ceFiles Seq uen ceFiles
HFileOutputFormat.configureIncrementalLoad(job, outputTable)
HFi les

69. Elastic MapReduce
HFi les
Copy to S3
Elastic MapReduce
Seq uen ceFiles Seq uen ceFiles
HFileOutputFormat.configureIncrementalLoad(job, outputTable)
HFi les HBase
$ hadoop jar hbase-VERSION.jar completebulkload

70. Additional Resources
Google Pregel: BSP-based graph processing system
Apache Giraph: Implementation of Pregel for Hadoop
MultiScanTableInputFormat: (code to appear on GitHub)
Apache Mahout - Distributed machine learning on Hadoop

71. Thanks!
dan@fullcontact.com