How NoSQL Paid Off for TelenorJavaZone13 September 2012 - Oslo
Sebastian Verheughe         Architect and developerTelenor - mobile middleware services (COS)           Katrina Sponheim  ...
Telenor NoSQL Experience            o The problem            o The business case            o The solution            o Th...
The Problem
Min Bedrift   Self service portal where Telenors corporate  customers can manage their entire portfolio of                ...
Telenors Corporate Customer Structure    Customer Acme Corporation         Customer Acme Consulting         Customer Acme ...
The Challenge With Large Corporate CustomersCustomers with large portfolios presented a couple ofchallenges for the self s...
The Nightly Logon & Pre-fetch SolutionIn order to achieve acceptable response times in MinBedrift,administrators were logg...
The Future - UnhandledTelenor calculated that the pre-fetch time window would soonbe filled, and a increasing percentage o...
The Business ImpactIn the end, Telenor would risk losing corporate customers dueto deteriorated customer experience
Other Caching Drawbackso Stale data up to 24 hours oldo Refresh/login for new users still takes a lot of timeo Memory chal...
The Business Case
Business CaseThe business case is built on the negative consequence of NOTaddressing the problem.                         ...
The Solution
Solution Requirements - High LevelThe middleware search services should be designed to supportlarge data sets in a better ...
The Previous Architecture Client   Client   Client            Client   Client   Client                   Middleware Servic...
The New ArchitectureOne Master (r/w) – Several Replicas (r)  Client          Client   Client            Client   Client   ...
Domain Event Messaging                  Domain                    Domain                   Event                     Event...
Putting it All TogetherMin Bedrift                 MW Search                    MW Auth          search                   ...
Lucene / Solr    Solution
Search ServiceToday implemented in Min Bedrift o Cached nightly o Simple, and iterates over the nodes when searching o Wit...
New Search ServiceData stored in Solr/Lucene search engine o New middleware module exposing WS using tomcat o Everything i...
Solr Cores                           Customer                SearchClient                      Account                Serv...
Entity DenormalizationAn entity may include data from several tables     Customer      Account      also contains customer...
Solr/Lucene - Denormalized List View                                                has                              Subsc...
Searching by RelevanceSearch some or all rows, and return hits by relevance (or sorted)  Subscription  User Name   Subscri...
Neo4jSolution
Resource Authorization ServiceStored procedure in RDBMS calculating all accesses o Uses several minutes to calculate for l...
New Resource Authorization ServiceCustomer structure stored in Neo4j graph database o New middleware module exposing WS us...
Nodes and Relationships      o Relationships with type and direction      o Nodes (with type as property)                 ...
Traversal (query)    All traversals start from a single node                                                The start node...
Following the RelationshipsOne custom PathExpander class   o Only follow valid relationships and direction   o Only follow...
Picking the NodesCustom Evaluator   o Decide to include or exclude   o Delegate to filter that fits your search   o Filter...
Example Access Authorization    Retrieve all subscriptions using a fan out search                   U                     ...
Example Access Authorization    Has access to resource using a reverse search to limit number    of nodes to evaluate. Fin...
The Challenges
Lucene/Solro   Using a document store in a relational world – updateso   Change mindset to search by relevance, not sortin...
Neo4jo Competenceo New way of thinkingo Making them really fast (profile & understand graph impl)o Getting the classic mid...
Messagingo Mapping from a relational model (need cache)
The Results
Project State o Phase 1 in production (subscription only, nightly populated) o Phase 2 in system test (the rest + live pop...
Neo4J Runtime EnvironmentInitial State:    Prewarmed at startup, all data in heapPopulation:       ~20 M nodes (all indexe...
Neo4J Measured PerformanceCustomers measured for performance:Corporation    Customer      Accounts   SubscriptionsX       ...
Neo4J Measured PerformanceFind accounts      10 ms       260 ms     2 ms     X: 1 300    Y: 23 000   Z: 18
Neo4J Measured PerformanceFind subscriptions     1 700 ms         750 ms      1300 ms    X: 147 000       Y: 52 000   Z: 9...
Neo4J Measured PerformanceHas access to subscription      2 ms            2 ms   2 ms       X               Y      Z
Solr/Lucene Measured PerformanceFind subscriptions       7 ms          58 ms     4 ms        X             Y            Z
Service Performance from Min Bedrift“Google” search for corporation x: 120 ms                    Min Bedrift              ...
Old vs. New Resource Authorization ServiceCalculate All Resources      RDBMS            GraphX                         12 ...
Min BedriftDemo
Summary Scalable     It allows customer growth Fast logon   On demand resource authorization Fast search Server side searc...
AlternativesIn-Memory Database (Sybase)This option was discussed, but license cost and the uncertaintyif it would be enoug...
How We Started Using NoSQL Technology o Downloaded and prototyped technology very early o Got training on site to accelera...
My 5 Cents
Think About…New TechnologyDo you have enough in-house competence, or can you easily buy thenecessary competence? Also when...
The EndQuestions?
Nächste SlideShare
Wird geladen in ...5
×

How NOSQL Paid off for Telenor

5,183

Published on

This presentation describes how NOSQL solutions such as the Neo4j graph database and Lucene/Solr index was used in a classic middleware stack in Telenor to solve perfomance and scalability issues.

Published in: Technologie, Business
1 Comment
14 Gefällt mir
Statistiken
Notizen
Keine Downloads
Views
Gesamtviews
5,183
Bei Slideshare
0
Aus Einbettungen
0
Anzahl an Einbettungen
4
Aktionen
Geteilt
0
Downloads
101
Kommentare
1
Gefällt mir
14
Einbettungen 0
No embeds

No notes for slide

How NOSQL Paid off for Telenor

  1. 1. How NoSQL Paid Off for TelenorJavaZone13 September 2012 - Oslo
  2. 2. Sebastian Verheughe Architect and developerTelenor - mobile middleware services (COS) Katrina Sponheim Architect and developer Telenor – business self service solutions
  3. 3. Telenor NoSQL Experience o The problem o The business case o The solution o The challenges o The results o My 5 cents
  4. 4. The Problem
  5. 5. Min Bedrift Self service portal where Telenors corporate customers can manage their entire portfolio of products. From small businesses to large corporations
  6. 6. Telenors Corporate Customer Structure Customer Acme Corporation Customer Acme Consulting Customer Acme Development Account Construction Account Demolition Subscription Huey Subscription Dewey Subscription Louie
  7. 7. The Challenge With Large Corporate CustomersCustomers with large portfolios presented a couple ofchallenges for the self service solution Min Bedrift:1. Middleware Services - Not Designed for Search The middleware services were not designed for managing large data volumes, resulting in a lot of processing in the client, and the need for extensive caching there.2. Resource Authorization – Long Calculation Time User access to resources required the middleware to calculate and cache all accesses at logon, something that could take up to many minutes.
  8. 8. The Nightly Logon & Pre-fetch SolutionIn order to achieve acceptable response times in MinBedrift,administrators were logged on and customer data was pre-fetched and put in a cache each night.However, as the usage of the solution grew, it became obviousthat the time window available for pre-fetching each night wasclosing fast. 0 0 0 9 3 9 3 9 3 6 6 6 2012 2013 2014
  9. 9. The Future - UnhandledTelenor calculated that the pre-fetch time window would soonbe filled, and a increasing percentage of the customers wouldexperience logon response times above the acceptable x sec. Login Not pre-fetched Pre-fetched time Portfolio Sizex Customer Portfolio Size
  10. 10. The Business ImpactIn the end, Telenor would risk losing corporate customers dueto deteriorated customer experience
  11. 11. Other Caching Drawbackso Stale data up to 24 hours oldo Refresh/login for new users still takes a lot of timeo Memory challenges in Min Bedrifto Unwanted network/middleware/database load
  12. 12. The Business Case
  13. 13. Business CaseThe business case is built on the negative consequence of NOTaddressing the problem. Loss of customers (revenue) Reduced sales transactions (revenue) Increased manual support (expenses) Other
  14. 14. The Solution
  15. 15. Solution Requirements - High LevelThe middleware search services should be designed to supportlarge data sets in a better way for the all clients.Resource authorization must be fast enough to deliver real timecalculations on demand.
  16. 16. The Previous Architecture Client Client Client Client Client Client Middleware Services Master Data RDBMS Multiple Sources
  17. 17. The New ArchitectureOne Master (r/w) – Several Replicas (r) Client Client Client Client Client Client Middleware Services Master Data Sybase RDBMS Search Res Auth Solr / Lucene Neo4j Multiple Sources
  18. 18. Domain Event Messaging Domain Domain Event Event Search Messaging Res Auth Solr / Lucene Apache Camel Neo4j Raw DB Event Master Data RDBMS
  19. 19. Putting it All TogetherMin Bedrift MW Search MW Auth search getAuthorizedResources filteredSearch authorized match
  20. 20. Lucene / Solr Solution
  21. 21. Search ServiceToday implemented in Min Bedrift o Cached nightly o Simple, and iterates over the nodes when searching o With memory/GC challenges
  22. 22. New Search ServiceData stored in Solr/Lucene search engine o New middleware module exposing WS using tomcat o Everything indexed makes search extremely fast o De-normalized data does not require joins o Search by relevance, paging, sorting and much more
  23. 23. Solr Cores Customer SearchClient Account Service Subscription
  24. 24. Entity DenormalizationAn entity may include data from several tables Customer Account also contains customer name & id Subscription also contains account name & id
  25. 25. Solr/Lucene - Denormalized List View has Subscription Customer Account user Arthur | Jackson Total 555 21 1234 2341 Lisa | Simpson Youth 555 64 3634 3435 John | Brown Pro 555 25 5433 5352
  26. 26. Searching by RelevanceSearch some or all rows, and return hits by relevance (or sorted) Subscription User Name Subscription Phone Number Account Ref. Score Rank Jane Youth 555 21 3253 5 3253 10 15 1 Paul Premium 555 23 4365 5262 John Standard 555 95 1436 7346 Nina Standard 555 15 3263 3734 Lydia Youth 555 92 3253 5 7334 5 2 Tom Standard 555 02 6394 3212 Neil Premium 555 03 2583 3523
  27. 27. Neo4jSolution
  28. 28. Resource Authorization ServiceStored procedure in RDBMS calculating all accesses o Uses several minutes to calculate for large customers o Cached for up to 24 hours o Extremely complex to understand (1500 lines of sql) o Tightly coupled with other services querying the database
  29. 29. New Resource Authorization ServiceCustomer structure stored in Neo4j graph database o New middleware module exposing WS using tomcat o Designed to focus on the relationships between objects o Very fast – independent of total amount of objects stored
  30. 30. Nodes and Relationships o Relationships with type and direction o Nodes (with type as property) U USER_ACCESS (with prop inherit: true/false) C PART_OFC C CONTROLLED_BY A A A AS S SUBSCRIBED_BY S S S S S S S S User Customer Account Subscription
  31. 31. Traversal (query) All traversals start from a single node The start node is often the U user node in our case CC C A A A A AS S S S S S S S S S S S User Customer Account Subscription
  32. 32. Following the RelationshipsOne custom PathExpander class o Only follow valid relationships and direction o Only follow necessary relationships o Check inheritance rules for current pathJust override the expand method Iterable<Relationship> expand(Path, BranchState)
  33. 33. Picking the NodesCustom Evaluator o Decide to include or exclude o Delegate to filter that fits your search o Filter may further evaluate neighbor nodesJust override the evaluate method Evaluation evaluate(Path path){ if (resourceFilter.filter(path) return Evaluation.INCLUDE_AND_CONTINUE return Evaluation.EXCLUDE_AND_CONTINUE }
  34. 34. Example Access Authorization Retrieve all subscriptions using a fan out search U CC C A A A A AS S S S S S S S S S S S User Customer Account Subscription
  35. 35. Example Access Authorization Has access to resource using a reverse search to limit number of nodes to evaluate. Find all paths, and validate one of them. U CC C A A A A AS S S S S S S S S S S S User Customer Account Subscription
  36. 36. The Challenges
  37. 37. Lucene/Solro Using a document store in a relational world – updateso Change mindset to search by relevance, not sortingo The time is in the small stuff – not difficult but needs learningo What type of queries to search on this platform, and NOTo Scaling & Distribution – Actually, not a challenge…
  38. 38. Neo4jo Competenceo New way of thinkingo Making them really fast (profile & understand graph impl)o Getting the classic middleware take use of the new serviceo What type of queries to search on this platform, and NOTo Scaling, not easy across servers (not needed for now)
  39. 39. Messagingo Mapping from a relational model (need cache)
  40. 40. The Results
  41. 41. Project State o Phase 1 in production (subscription only, nightly populated) o Phase 2 in system test (the rest + live population)The following results are from the test environment now
  42. 42. Neo4J Runtime EnvironmentInitial State: Prewarmed at startup, all data in heapPopulation: ~20 M nodes (all indexed) ~20 M node properties (only 1 per node) ~50 M relationships Batchwise (50 K nodes) in 35 minutesBase heap usage: 10 GB (of 16 GB)Load: Minimal (not measured with heavy load)
  43. 43. Neo4J Measured PerformanceCustomers measured for performance:Corporation Customer Accounts SubscriptionsX 160 1 300 147 000Y 32 000 23 000 52 000Z 7 18 95 000 X Y Z
  44. 44. Neo4J Measured PerformanceFind accounts 10 ms 260 ms 2 ms X: 1 300 Y: 23 000 Z: 18
  45. 45. Neo4J Measured PerformanceFind subscriptions 1 700 ms 750 ms 1300 ms X: 147 000 Y: 52 000 Z: 95 000
  46. 46. Neo4J Measured PerformanceHas access to subscription 2 ms 2 ms 2 ms X Y Z
  47. 47. Solr/Lucene Measured PerformanceFind subscriptions 7 ms 58 ms 4 ms X Y Z
  48. 48. Service Performance from Min Bedrift“Google” search for corporation x: 120 ms Min Bedrift searchAllResources 120 ms Search Solr findAuthorizedResources 55 ms Auth Graph
  49. 49. Old vs. New Resource Authorization ServiceCalculate All Resources RDBMS GraphX 12 min 18 sec < 2 secY 22 min 58 sec < 2 secZ 3 min 15 sec < 2 sec Cold Warm Heap
  50. 50. Min BedriftDemo
  51. 51. Summary Scalable It allows customer growth Fast logon On demand resource authorization Fast search Server side search engine much faster Reusable All clients may use new services Fresh data Not up to 24 hours old – almost live
  52. 52. AlternativesIn-Memory Database (Sybase)This option was discussed, but license cost and the uncertaintyif it would be enough made us go for the NoSQL option.Other NoSQL SolutionsWe chose to prototype Neo4j and Lucene/Solr because theywere popular and seemed to fit us well, and since it worked westuck with them.
  53. 53. How We Started Using NoSQL Technology o Downloaded and prototyped technology very early o Got training on site to accelerate the development startup o At the end of development, did a review/QA of the solutionFor Lucene/Solr, we got training and support from local Solr/Luceneexpert consultant Jan HøydahlFor Neo4j, we got training and excellent support from NeoTech directly
  54. 54. My 5 Cents
  55. 55. Think About…New TechnologyDo you have enough in-house competence, or can you easily buy thenecessary competence? Also when maintaining the code.No Language Standard for Graph DatabasesHow simple (or possible) is it to change the NoSQL provider?Working With RelationshipsGraph databases are intuitive and fast to work with when interestedin how objects are related to each other.Gentle NoSQL IntroductionEasier to start using when supporting a specific and limited servicesComplexityYou introduce complexity, so make sure it is worth it!
  56. 56. The EndQuestions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×