Building a high performance Neo4j Recommendation Engine using the GraphAware Framework

1. GraphAware
TM
Michal Bachman
graphaware.com
@graph_aware
Recommendations
with Neo4j
Building a high-performance recommendation engine

2. GraphAware
TM
Quick Intro
Why Graphs?
Business and Technical Challenges
GraphAware Recommendation Engine
About This Talk

3. GraphAware
TM
News you should read
Books you should buy
People you may know
People you should date
People you should market a product to
…
Recommendation Engines

4. GraphAware
TM
Content-based (features)
Collaborative filtering (user <-> item relationships)
Recommendation Engines

5. GraphAware
TM
Features as well as relationships can be naturally
represented as a graph.
Good News

6. GraphAware
TM
Example
IS_OF_GENRE
title:
“Love Actually”
Movie
name: “Bob”
User
name:
“Comedy”
Genre
RATED
rating: 5
name: “Alice”
User
name:
“Romance”
Genre
title:
“American Pie”
Movie
IS_OF_GENRE
IS_OF_GENRE
RATEDrating: 5
INTERESTED_IN
rating: 5
RATED

7. GraphAware
TM
Easy to understand
Natural to model
Flexible (schema-free)
Fast to query
Graphs (Neo4j)

8. GraphAware
TM
Great for a quick PoC
Great for smaller data sets
Great for relatively simple logic
Cypher

9. GraphAware
TM
Cypher
MATCH
(u:User)-[:LIKED]->(m:Movie),
(m)<-[:LIKED]-(another:User),
(another)-[:LIKED]->(reco:Movie)
WHERE NOT
(u)-[:LIKED|DISLIKED]->(reco)
RETURN reco;

10. GraphAware
TM
Requirements of real-world recommendation
engines are often much more complex.
The Reality

11. GraphAware
TM
Imagine you’re building the ”people you may
know” feature on LinkedIn.
Example

12. GraphAware
TM
After a brainstorming session, your team came up
with the following ways of finding people one may
know:
Example

13. GraphAware
TM
Common contacts
Facebook friends in common
Email / mobile contacts in common
Each others email / mobile contact
Worked for the same company
Studied at the same school
Share the same interest
Live in the same city
People You May Know

14. GraphAware
TM
But that’s just the beginning! Let’s go back and
re-visit.
Example

15. GraphAware
TM
More contacts in common = better chance?
Same city / school / company = does size matter?
What about emails that don’t represent a person?
What about people already connected?
And pending…
And rejected…
And repeatedly ignored…
People You May Know

16. GraphAware
TM
Finding things to recommend
Serving the most relevant recommendations
Measuring the quality of recommendations
Time to market / cost of development
Business Challenges

17. GraphAware
TM
Performance (real-time!)
Simplicity
Flexibility
Technical Challenges

18. GraphAware
TM
So we came up with an open-source
recommendation engine skeleton that will help you
solve all the challenges.
We’ve done it before

19. GraphAware
TM
plugin to Neo4j (uses GraphAware Framework)
you have to use a JVM-language
opinionated architecture
very fast
very flexible
handles all the plumbing
Recommendation Engine

20. GraphAware
TM
Engine per recommendation “reason” (core logic)
Engine executes a graph traversal to find items
Engines are assembled into higher-level engines
Design Decisions

21. GraphAware
TM
Example
IS_OF_GENRE
title:
“Love Actually”
Movie
name: “Bob”
User
name:
“Comedy”
Genre
RATED
rating: 5
name: “Alice”
User
name:
“Romance”
Genre
title:
“American Pie”
Movie
IS_OF_GENRE
IS_OF_GENRE
RATEDrating: 5
INTERESTED_IN
rating: 5
RATED

22. GraphAware
TM
Engine per recommendation “reason” (core logic)
Engine executes a graph traversal to find items
Engines are assembled to higher-level engines
Items discovered multiple times are more relevant
Relevance depends on how was item discovered
Design Decision

23. GraphAware
TM
Example
IS_OF_GENRE
title:
“Love Actually”
Movie
name: “Bob”
User
name:
“Comedy”
Genre
RATED
rating: 5
name: “Alice”
User
name:
“Romance”
Genre
title:
“American Pie”
Movie
IS_OF_GENRE
IS_OF_GENRE
RATEDrating: 5
INTERESTED_IN
rating: 5
RATED

24. GraphAware
TM
Engine per recommendation “reason” (core logic)
Engine executes a graph traversal to find items
Engines are assembled to higher-level engines
Items discovered multiple times are more relevant
Relevance depends on how was item discovered
Items not to be recommended: “cross-cutting”
concern
Design Decisions

25. GraphAware
TM
Example
IS_OF_GENRE
title:
“Love Actually”
Movie
name: “Bob”
User
name:
“Comedy”
Genre
RATED
rating: 5
name: “Alice”
User
name:
“Romance”
Genre
title:
“American Pie”
Movie
IS_OF_GENRE
IS_OF_GENRE
RATEDrating: 5
INTERESTED_IN
rating: 5
RATED

26. GraphAware
TM
Input -> Engine -> Recommendations
Scores and Score Transformers
Blacklists
Filters
Post-processors
Context (how many, how fast,…?)
Loggers
Architecture

27. GraphAware
TM
In 5 minutes, we’ll build a simple engine that
recommends who you should be friends with.
Let’s Build Something

28. GraphAware
TM
0) Model

30. GraphAware
TM
1) Discover

31. GraphAware
TM
public class FriendsInCommon extends SomethingInCommon {
@Override
public String name() {
return "friendsInCommon";
}
@Override
protected RelationshipType getType() {
return FRIEND_OF;
}
@Override
protected Direction getDirection() {
return BOTH;
}
}

32. GraphAware
TM
2) Score

34. GraphAware
TM
public class FriendsInCommon extends SomethingInCommon {
@Override
protected ScoreTransformer scoreTransformer() {
return new ParetoScoreTransformer(100, 10);
}
@Override
public String name() {
return "friendsInCommon";
}
@Override
protected RelationshipType getType() {
return FRIEND_OF;
}
@Override
protected Direction getDirection() {
return BOTH;
}
}

35. GraphAware
TM
3) Post-Process

36. GraphAware
TM
public class RewardSameLocation extends RewardSomethingShared {
@Override
protected RelationshipType type() {
return LIVES_IN;
}
@Override
protected Direction direction() {
return OUTGOING;
}
@Override
protected float scoreValue(Node reco, Node in, Node shared) {
return 10;
}
@Override
protected String scoreName() {
return "sameLocation";
}
}

37. GraphAware
TM
public class RewardSameLabels implements PostProcessor<Node, Node> {
@Override
public void postProcess(Recommendations<Node> out, Node in) {
Label[] inLabels = toArray(in.getLabels());
for (Recommendation<Node> reco : out.get()) {
if (Arrays.equals(inLabels, toArray(reco.getItem().getLabels()))) {
reco.add("sameGender", 10);
}
}
}
}

38. GraphAware
TM
4) Filter

39. GraphAware
TM
public final class FriendsContextFactory extends Neo4jContextFactory {
@Override
protected List<BlacklistBuilder<Node, Node>> blacklistBuilders() {
return asList(
new ExcludeSelf(),
new ExistingRelationshipBlacklistBuilder(FRIEND_OF, BOTH)
);
}
@Override
protected List<Filter<Node, Node>> filters() {
return asList(
new ExcludeSelf()
);
}
}

40. GraphAware
TM
5) Assemble

41. GraphAware
TM
public final class FriendsComputingEngine extends Neo4jTopLevelDelegatingEngine {
public FriendsComputingEngine() {
super(new FriendsContextFactory());
}
@Override
protected List<RecommendationEngine<Node, Node>> engines() {
return asList(
new FriendsInCommon(),
new RandomPeople()
);
}
@Override
protected List<PostProcessor<Node, Node>> postProcessors() {
return asList(
new RewardSameLabels(),
new RewardSameLocation(),
new PenalizeAgeDifference()
);
}
}

42. GraphAware
TM
?) Precompute

43. GraphAware
TM
public final class FriendsComputingEngine extends Neo4jTopLevelDelegatingEngine {
public FriendsComputingEngine() {
super(new FriendsContextFactory());
}
@Override
protected List<RecommendationEngine<Node, Node>> engines() {
return asList(
new FriendsInCommon(),
new RandomPeople()
);
}
@Override
protected List<PostProcessor<Node, Node>> postProcessors() {
return asList(
new RewardSameLabels(),
new RewardSameLocation(),
new PenalizeAgeDifference()
);
}
@Override
public ParticipationPolicy<Node, Node> participationPolicy(Context<Node, Node> context) {
return ParticipationPolicy.IF_MORE_RESULTS_NEEDED;
}
}

44. GraphAware
TM
public final class FriendsRecoEngine extends Neo4jTopLevelDelegatingEngine {
public FriendsRecommendationEngine() {
super(new FriendsContextFactory());
}
@Override
protected List<RecommendationEngine<Node, Node>> engines() {
return asList(
new Neo4jPrecomputedEngine(),
new FriendsComputingEngine()
);
}
}

45. GraphAware
TM
6) Log

46. GraphAware
TM
public final class FriendsRecoEngine extends Neo4jTopLevelDelegatingEngine {
public FriendsRecommendationEngine() {
super(new FriendsContextFactory());
}
@Override
protected List<RecommendationEngine<Node, Node>> engines() {
return asList(
new Neo4jPrecomputedEngine(),
new FriendsComputingEngine()
);
}
@Override
protected List<Logger<Node, Node>> loggers() {
return asList(
new Slf4jRecommendationLogger<Node, Node>(),
new Slf4jStatisticsLogger<Node, Node>()
);
}
}

47. GraphAware
TM
7) Test

48. GraphAware
TM
List<Recommendation<Node>> reco =
recommendationEngine.recommend(getPersonByName("Adam"), Mode.REAL_TIME, 2);
String expected =
"(Vince {total:19.338144," +
"ageDifference:-5.527864," +
"friendsInCommon:14.866008," +
"sameGender:10.0})," +
"(Luanne {total:11.553411," +
"ageDifference:-3.312597," +
"friendsInCommon:14.866008})";
assertEquals(expected, toString(reco));

49. GraphAware
TM
List<Recommendation<Node>> reco =
recommendationEngine.recommend(getPersonByName("Luanne"), REAL_TIME, 4);
assertEquals("Daniela", reco.get(0).getItem().getProperty("name"));
assertEquals(22, reco.get(0).getScore().getTotalScore(), 0.5);
assertEquals("Adam", reco.get(1).getItem().getProperty("name"));
assertEquals(12, reco.get(1).getScore().getTotalScore(), 0.5);
assertEquals("Vince", reco.get(2).getItem().getProperty("name"));
assertEquals(8, reco.get(2).getScore().getTotalScore(), 0.5);
assertEquals("Bob", reco.get(3).getItem().getProperty("name"));
assertEquals(-9, reco.get(3).getScore().getTotalScore(), 0.5);

50. GraphAware
TM
Finding things to recommend
Serving the most relevant recommendations
Measuring the quality of recommendations
Time to market / cost of development
Business Challenges

51. GraphAware
TM
Performance (real-time!)
Simplicity
Flexibility
Technical Challenges

52. GraphAware
TM
Getting Started
<dependencies>
...
<dependency>
<groupId>com.graphaware.neo4j</groupId>
<artifactId>recommendation-engine</artifactId>
<version>2.1.6.27.2</version>
</dependency>
...
<dependencies>

53. GraphAware
TM
Built-in ability to pre-compute recommendations
Other built-in base-classes
But we need your help!
https://github.com/graphaware/neo4j-reco
There’s More!

54. GraphAware
TM
Built-in algorithms
Time-based ParticipationPolicy
Integration with compute engines
Machine learning
Future

55. GraphAware
TM
GraphAware Framework makes it easy to build,
test, and deploy generic as well as domain-
specific functionality for Neo4j.
GraphAware Framework

56. GraphAware
TM
GraphUnit
& RestTest
RelCount WarmUp Schema (wip)
Recommendation
Engine
GraphAware Framework
ChangeFeed UUID TimeTree Algorithms NodeRank

57. GraphAware
TM
Open Source (GPL)
Active
Production Ready
Github: github.com/graphaware
Our Web: graphaware.com
Maven Central
GraphAware Framework

58. GraphAware
TM
Try it
Give us feedback
Contribute
Build your own modules
Get in touch for support / consultancy
GraphAware Framework

59. GraphAware
TM
GraphAware Events
31
Jan
Recommendation
Engines in Brussels
(FOSDEM)
31
Jan
GraphGen in Brussels
(FOSDEM)
5
Feb
Recommendation
Engines Webinar
5
Feb
Meetup at GraphAware
(build your own
Recommendation Engine)
10
Feb
Neo4j Fundamentals in
Manchester
10
Feb
Neo4j Meetup in
Manchester
17
Feb
Neo4j Fundamentals in
Edinburgh
17
Feb
Neo4j Meetup in
Edinburgh

61. GraphAware
TM
GraphConnect Europe 2015
When:
Where:
Tickets:
Call for Papers:
Sponsors:
Thursday, 7th May, 2015 - main Conference Day
Wednesday, 6th May 2015 - Training Day
Etc venues, 155 Bishopsgate, London
(next to Liverpool Street Station)
now available on www.graphconnect.com
199$ early bird plus 100$ for training
499$ full price plus 100$ for training
open now till 29th January
all Neo4j community members, customers or
general graph enthusiasts are invited to submit their talk
open now till 29th January, email:
gceurope@neotechnology.com

62. graphaware.com
@graph_aware
Thank You!
GraphAware
TM