This presentation describes how NOSQL solutions such as the Neo4j graph database and Lucene/Solr index was used in a classic middleware stack in Telenor to solve perfomance and scalability issues.

1. How NoSQL Paid Off for Telenor
JavaZone
13 September 2012 - Oslo

2. Sebastian Verheughe
Architect and developer
Telenor - mobile middleware services (COS)
Katrina Sponheim
Architect and developer
Telenor – business self service solutions

3. Telenor NoSQL Experience
o The problem
o The business case
o The solution
o The challenges
o The results
o My 5 cents

4. The Problem

5. Min Bedrift
Self service portal where Telenor's corporate
customers can manage their entire portfolio of
products.
From small businesses to large corporations

6. Telenor's Corporate Customer Structure
Customer Acme Corporation
Customer Acme Consulting
Customer Acme Development
Account Construction
Account Demolition
Subscription Huey
Subscription Dewey
Subscription Louie

7. The Challenge With Large Corporate Customers
Customers with large portfolios presented a couple of
challenges for the self service solution Min Bedrift:
1. Middleware Services - Not Designed for Search
The middleware services were not designed for managing
large data volumes, resulting in a lot of processing in the
client, and the need for extensive caching there.
2. Resource Authorization – Long Calculation Time
User access to resources required the middleware to
calculate and cache all accesses at logon, something that
could take up to many minutes.

8. The Nightly Logon & Pre-fetch Solution
In order to achieve acceptable response times in MinBedrift,
administrators were logged on and customer data was pre-
fetched and put in a cache each night.
However, as the usage of the solution grew, it became obvious
that the time window available for pre-fetching each night was
closing fast.
0 0 0
9 3 9 3 9 3
6 6 6
2012 2013 2014

9. The Future - Unhandled
Telenor calculated that the pre-fetch time window would soon
be filled, and a increasing percentage of the customers would
experience logon response times above the acceptable x sec.
Login Not pre-fetched Pre-fetched
time
Portfolio Size
x
Customer Portfolio Size

10. The Business Impact
In the end, Telenor would risk losing corporate customers due
to deteriorated customer experience

11. Other Caching Drawbacks
o Stale data up to 24 hours old
o Refresh/login for new users still takes a lot of time
o Memory challenges in Min Bedrift
o Unwanted network/middleware/database load

12. The Business Case

13. Business Case
The business case is built on the negative consequence of NOT
addressing the problem.
Loss of customers (revenue)
Reduced sales transactions (revenue)
Increased manual support (expenses)
Other

14. The Solution

15. Solution Requirements - High Level
The middleware search services should be designed to support
large data sets in a better way for the all clients.
Resource authorization must be fast enough to deliver real time
calculations on demand.

16. The Previous Architecture
Client Client Client Client Client Client
Middleware Services
Master
Data
RDBMS
Multiple Sources

17. The New Architecture
One Master (r/w) – Several Replicas (r)
Client Client Client Client Client Client
Middleware Services
Master
Data
Sybase RDBMS
Search Res Auth
Solr / Lucene Neo4j
Multiple Sources

18. Domain Event Messaging
Domain Domain
Event Event
Search Messaging Res Auth
Solr / Lucene Apache Camel Neo4j
Raw DB
Event
Master
Data
RDBMS

19. Putting it All Together
Min Bedrift MW Search MW Auth
search
getAuthorizedResources
filteredSearch
authorized match

20. Lucene / Solr
Solution

21. Search Service
Today implemented in Min Bedrift
o Cached nightly
o Simple, and iterates over the nodes when searching
o With memory/GC challenges

22. New Search Service
Data stored in Solr/Lucene search engine
o New middleware module exposing WS using tomcat
o Everything indexed makes search extremely fast
o De-normalized data does not require joins
o Search by relevance, paging, sorting and much more

23. Solr Cores
Customer
Search
Client Account
Service
Subscription

24. Entity Denormalization
An entity may include data from several tables
Customer
Account also contains customer name & id
Subscription also contains account name & id

25. Solr/Lucene - Denormalized List View
has
Subscription
Customer
Account
user
Arthur | Jackson Total 555 21 1234 2341
Lisa | Simpson Youth 555 64 3634 3435
John | Brown Pro 555 25 5433 5352

26. Searching by Relevance
Search some or all rows, and return hits by relevance (or sorted)
Subscription
User Name Subscription Phone Number Account Ref. Score Rank
Jane Youth 555 21 3253 5 3253 10 15 1
Paul Premium 555 23 4365 5262
John Standard 555 95 1436 7346
Nina Standard 555 15 3263 3734
Lydia Youth 555 92 3253
5 7334 5
2
Tom Standard 555 02 6394 3212
Neil Premium 555 03 2583 3523

27. Neo4j
Solution

28. Resource Authorization Service
Stored procedure in RDBMS calculating all accesses
o Uses several minutes to calculate for large customers
o Cached for up to 24 hours
o Extremely complex to understand (1500 lines of sql)
o Tightly coupled with other services querying the database

29. New Resource Authorization Service
Customer structure stored in Neo4j graph database
o New middleware module exposing WS using tomcat
o Designed to focus on the relationships between objects
o Very fast – independent of total amount of objects stored

30. Nodes and Relationships
o Relationships with type and direction
o Nodes (with type as property)
U
USER_ACCESS (with prop inherit: true/false)
C
PART_OF
C
C
CONTROLLED_BY
A
A
A A
S S SUBSCRIBED_BY
S S S S
S S
S S
User Customer Account Subscription

31. Traversal (query)
All traversals start from a single node
The start node is often the
U
user node in our case
C
C
C
A
A A
A A
S S
S S S S S
S S S
S S
User Customer Account Subscription

32. Following the Relationships
One custom PathExpander class
o Only follow valid relationships and direction
o Only follow necessary relationships
o Check inheritance rules for current path
Just override the expand method
Iterable<Relationship> expand(Path, BranchState)

33. Picking the Nodes
Custom Evaluator
o Decide to include or exclude
o Delegate to filter that fits your search
o Filter may further evaluate neighbor nodes
Just override the evaluate method
Evaluation evaluate(Path path){
if (resourceFilter.filter(path)
return Evaluation.INCLUDE_AND_CONTINUE
return Evaluation.EXCLUDE_AND_CONTINUE
}

34. Example Access Authorization
Retrieve all subscriptions using a fan out search
U
C
C
C
A
A A
A A
S S
S S S S S
S S S
S S
User Customer Account Subscription

35. Example Access Authorization
Has access to resource using a reverse search to limit number
of nodes to evaluate. Find all paths, and validate one of them.
U
C
C
C
A
A A
A A
S S
S S S S S
S S S
S S
User Customer Account Subscription

36. The Challenges

37. Lucene/Solr
o Using a document store in a relational world – updates
o Change mindset to search by relevance, not sorting
o The time is in the small stuff – not difficult but needs learning
o What type of queries to search on this platform, and NOT
o Scaling & Distribution – Actually, not a challenge…

38. Neo4j
o Competence
o New way of thinking
o Making them really fast (profile & understand graph impl)
o Getting the classic middleware take use of the new service
o What type of queries to search on this platform, and NOT
o Scaling, not easy across servers (not needed for now)

39. Messaging
o Mapping from a relational model (need cache)

40. The Results

41. Project State
o Phase 1 in production (subscription only, nightly populated)
o Phase 2 in system test (the rest + live population)
The following results are from the test environment now

42. Neo4J Runtime Environment
Initial State: Prewarmed at startup, all data in heap
Population: ~20 M nodes (all indexed)
~20 M node properties (only 1 per node)
~50 M relationships
Batchwise (50 K nodes) in 35 minutes
Base heap usage: 10 GB (of 16 GB)
Load: Minimal (not measured with heavy load)

43. Neo4J Measured Performance
Customers measured for performance:
Corporation Customer Accounts Subscriptions
X 160 1 300 147 000
Y 32 000 23 000 52 000
Z 7 18 95 000
X Y Z

44. Neo4J Measured Performance
Find accounts
10 ms 260 ms 2 ms
X: 1 300 Y: 23 000 Z: 18

45. Neo4J Measured Performance
Find subscriptions
1 700 ms 750 ms 1300 ms
X: 147 000 Y: 52 000 Z: 95 000

46. Neo4J Measured Performance
Has access to subscription
2 ms 2 ms 2 ms
X Y Z

47. Solr/Lucene Measured Performance
Find subscriptions
7 ms 58 ms 4 ms
X Y Z

48. Service Performance from Min Bedrift
“Google” search for corporation x: 120 ms
Min Bedrift
searchAllResources
120 ms Search Solr
findAuthorizedResources
55 ms Auth Graph

49. Old vs. New Resource Authorization Service
Calculate All Resources RDBMS Graph
X 12 min 18 sec < 2 sec
Y 22 min 58 sec < 2 sec
Z 3 min 15 sec < 2 sec
Cold Warm Heap

50. Min Bedrift
Demo

51. Summary
Scalable It allows customer growth
Fast logon On demand resource authorization
Fast search Server side search engine much faster
Reusable All clients may use new services
Fresh data Not up to 24 hours old – almost live

52. Alternatives
In-Memory Database (Sybase)
This option was discussed, but license cost and the uncertainty
if it would be enough made us go for the NoSQL option.
Other NoSQL Solutions
We chose to prototype Neo4j and Lucene/Solr because they
were popular and seemed to fit us well, and since it worked we
stuck with them.

53. How We Started Using NoSQL Technology
o Downloaded and prototyped technology very early
o Got training on site to accelerate the development startup
o At the end of development, did a review/QA of the solution
For Lucene/Solr, we got training and support from local Solr/Lucene
expert consultant Jan Høydahl
For Neo4j, we got training and excellent support from NeoTech directly

54. My 5 Cents

55. Think About…
New Technology
Do you have enough in-house competence, or can you easily buy the
necessary competence? Also when maintaining the code.
No Language Standard for Graph Databases
How simple (or possible) is it to change the NoSQL provider?
Working With Relationships
Graph databases are intuitive and fast to work with when interested
in how objects are related to each other.
Gentle NoSQL Introduction
Easier to start using when supporting a specific and limited services
Complexity
You introduce complexity, so make sure it is worth it!

56. The End
Questions?