How Medium is using Neo4j

1. How Medium uses Neo4j
Nathaniel Felsen
May 20th, 2015

2. About Me
Nathaniel Felsen
Data & DevOps engineer
nathanielfelsen
n@medium.com
@faitlezen

3. Agenda
• What is Medium and what problem we are solving through Neo4j ?
• Why did we pick Neo4j ?
• Our architecture
• Steps taken and obstacles encountered while going live
• Improve Neo4j’s performances
• Live Demo

4. • Easiest, fastest way to create a
beautiful story
• Seamless integration of photos,
audio & video
• Optimized for web, tablet &
mobile
Medium is a beautiful
publishing experience.

5. Medium is a home for
influential contributors.

6. Medium is a place for
important ideas.

7. • Follow, share, recommend
• Personalized story feed
• Customized daily emails &
notifications
Medium is a network that
builds audience.

8. Datastore Selection Process

9. DynamoDB
• Need to nail the schema ≠ Experimentation
• Limited ways of querying data
• Things like short path between users won’t
perform well
Pros
• Expertise
• Already used to store user info
• No maintenance
• No hardware
Cons

10. • Using Relational database to
create graphs
• Sharding
Pros
• Used by lots of people and
heavily vetted.
• Less rampup for learning
querying language.
• Strong community.
Cons

11. Pros
Flock DB
• Not maintained anymore
• 2 tiers model
• Deal with sharding in the near
future
• Expertise
• SQL Lite syntax
• Open Source / Free
Cons

12. • Not free
• No expertise in house
• Requires hardware
Pros
• Easy to start
• Easy to experiment
• Good community
• Enteprise edition: HA, Backup, Support
Cons

13. Architecture

14. Our Social Service Architecture

15. Nodes
• User
• Post
• Collection
Relationships
• Edited
• Wrote
• Published
• Recommended
• Followed
• …

16. Use queues for the
writes
Write are done to the master only
If you lose your master, you need
to wait for a new election

17. Productionising Neo4j

18. Capacity planning

19. Initial Data Import

20. Metrics / Monitoring
• Architecture
• Systems
• Neo4j
• Dataset
• Java
• Services that interact with Neo4j

21. Logs aggregation /
Indexing
• ElasticSearch
• Logstash
• Logstash forwarder
• Kibana

22. Backups
• Incremental Backup
• Full Backup

23. Runbook / Playbook

24. Getting optimal performance with
Neo4j

25. Talk to the support

26. What Neo4j is good and not as good at
Long Traversal Where NOT Dense / Super Node

27. Cypher Tricks
http://watch.neo4j.org/video/84900121

28. Tune the configuration
over time
• Java Garbage collection (stop the world)
• Neo4j settings

29. Cache Settings
Neo4J2.0and2.1

30. Neo4J2.2

31. Server Plugins & Unmanaged Extensions
• Easy to Deploy
• The server’s functionality can be extended by adding plugins.
• RESTful Web Services (JAX-RS)
• Put more logic in the code like caching
• Sharp tool

32. Demo

33. Followers who recommended a story

34. Top Recommended stories

35. People Recommended to follow

37. Collaborative Filtering

38. n@medium.com
Questions ? Feedback ?