SlideShare ist ein Scribd-Unternehmen logo
1 von 37
Downloaden Sie, um offline zu lesen
Graph Connect San Francisco – 4 Oct 2013
by Sebastian Verheughe
Using Graph Databases in Real-Time to
Solve Resource Authorization at Telenor
Telenor Norway
Subsidiary of the Telenor Group
2 billions USD in mobile revenues 2012
Sebastian Verheughe
Lead Developer for Neo4j solution
Coding Architect
Disclaimer
The presentation is not identical to the implementation
due to security reasons but shows how we have
modeled and solved the problem in general.
However, all presented data (numbers & charts) are
real, unfiltered and extracted from the production logs
A very aspect of
our business
Telenor Norway Middleware Services
MOBILE
MW
BUSINESS
LOGIC
& DATA
Providing business logic
and data for all channels
in the mobile value chain
Handles users with
access to X00,000
resources
Backend Backend BackendBackend Backend Backend
Channel Channel ChannelChannel Channel Channel
used by 42 channels
calls 35 sub-systems
10,000 code classes
500 requests/second
20,000 orders/day
Our Problem
20 minutes to calculate all accessible resources
1500 lines of SQL to implement the authorization logic
“solved” by caching data going stale
and the solution did not scale…
Finance Production HR
Why a Graph Database?
The questions we wanted answered
required traversal of tree structures.
Sub Sub
Tablet Phone
Access
User
Which resources
does the user
have access to?
Sales
Parent Company
Part of Company
Subscription
Owner
Uses
Subscription
Tablet
Tailored Read Model
The Model makes read queries
as simple and efficient as possible.
First find your questions
then model your graph
graph model = relational model
High Level Architecture
Classic MW
Services
RDBMS
Resource
Authorization
Message Queue
check
access
Clients
tx log
Neo4j
other
sources
Conditional Rules
ACCESS is given with the following include parameters:
access to subsidiaries and access to content
User
Only find children of PARENT COMPANY
given access to subsidiaries is allowed
Only look at PART OF COMPANY
given access to content is allowed
Only look at SUBSCRIPTION OWNER
given access to content is allowed
Different Access Needs
Umbrella Admin
Access ContentAccess Subsidiaries & Content
Super Admin
Admin
Access S&C
Graph Algorithm
Prerequisite: The user node
1. Follow all ACCESS relationships and
read the access parameters on the relationship
2. Follow all PARENT COMPANY relationships given
access to subsidiaries is allowed
3. Follow all PART OF COMPANY relationships given
access to content is allowed
4. Follow all SUBSCRIPTION OWNER relationships given
access to content is allowed
Solution Value
1. Performance optimized from minutes to seconds.
2. Simplicity of writing and understanding business rules
for the query traversal.
3. Scalability by performance allowing us to onboard
more corporate customers (project business case)
Autonomous Service
with it’s own life-cycle and data repository.
Authorization Complexity
• Not a collection of isolated customer trees *
• Not all users of a customer have equal access
• Not a fixed schema, form or size for all
customers
• Real-time updated with customer & product
data
The data form a highly connected living graph
* Covered later in Technical Details
How we Started with Neo4j
1. Searched the internet for articles about graph
database and different solutions.
2. Downloaded and quickly prototyped the solution we
liked that matched our requirements (Neo4j).
3. Workshop with Neo4j and our project developers to
quickly gain competence and ensure design QA.
4. Solution QA with Neo4j before production and help
with performance issues / tuning.
Lessons Learned
• Choose a solution/technology that fits your problem
• New way of thinking – build competence in org.
• Profile your java code to make it really fast
• Don’t put everything into the graph (functional creep)
• Need to know how traversal works (e.g. shortest
path)
• Benchmark the graph to evaluate your traversal
speed
Alternative In-Memory RDBMS
Option 1: Use existing database
- Performance issues due to shared data / suboptimal
structure
- Complexity since SQL not designed for traversal
Option 2: Separate database
+ Might reach same performance as graph db
+ Familiar technology
- Complexity since SQL not designed for traversal
Decided to go with our instinct
Graph Database
Different Graph Structures
Company X: 147 000 Company Y: 52 000 Company Z: 95 000
1 700 ms 750 ms 1300 ms
get all accessible subscriptions
2 000
1 000
Data from test – repeated prod sampling gave ~2.4 sec for 215,000 subscriptions
Different Graph Structures
1 ms 1 ms 1 ms
check access to single subscription
Company X: 147 000 Company Y: 52 000 Company Z: 95 000
2 000
1 000
Production Performance
retrieve all accessible resources
Check single resource access
1 ms
No operational problems in production
RDBMS Disk RDBMS (mem cached) Graph In-Heap
Company X 12 min 18 sec < 2 sec
Company Y 22 min 58 sec < 2 sec
Company Z 3 min 15 sec < 2 sec
Technical Details
Production Details
Graph Size 27 million nodes (pre-warmed in
heap)
~1x properties, ~2x relationships
Traffic Volume ~1000 req/min during biz hours
~ 40K daily real-time updates
Performance Avg: 1 ms, 99% < 4 ms, 99.9% < 9
ms
JVM Sun 6, 20 GB Heap (~15 GB pre-
warmed)
Production Has Access Query
Time (ms)
Time (ms)
Production All Queries
Time (ms)
Garbage collection
Implementing the Algorithm
Lets look at the Neo4j Traversal Framework
Iterable<Node> getAccessibleResources(…) {
Evaluator myEvaluator = …
Expander myExpander = …
return Traversal.description()
.evaluator(myEvaluator)
.expander(myExpander)
.traverse(startNode).nodes();
}
Implementing the Algorithm
Evaluator is a simple filter, e.g. for Node
type
class MyEvaluator implements Evaluator {
public Evaluation evaluate(Path path) {
if <I am interested in this node>
return Evaluation.INCLUDE_AND_CONTINUE;
else
return Evaluation.EXCLUDE_AND_CONTINUE;
}
}
Implementing the Algorithm
The custom Expander contains business
rules!
class ResAuthExpander implements PathExpander<PathExpander> {
…
public … expand(Path path, BranchState<…> state) {
if (path.lastRelationship rel == ACCESS)
accToSub = rel.getProperty(ACCESS_TO_SUBSIDIARIES);
accToCont = rel.getProperty(ACCESS_TO_CONTENT);
state.set( getExpander(accToSub, accToCont) );
}
return state.get().expand(…)
}
Single expander class to control business
Implementing the Algorithm
Generates the valid relationships to traverse.
public getExpander(boolean accToSub, boolean accToCont) {
PathExpander exp = StandardExpander.DEFAULT.add(ACCESS,…);
if (accToSub)
exp.add(PARENT_COMPANY,…)
if (accToCont)
exp.add(PART_OF_COMPANY,…).add(SUBSCRIPTION_OWNER,…);
return exp;
}
}
U-Turn Strategy
X
Subscription
Access
User
Reversing the traversal increases performance from n/2 to
2d where n and d are tree size and depth (we went from 1s to
1.
2.
3.
4.
Does the user
have access to
subscription X?
5.
6.
7.
8.
Up to find path quickly
Down to check access
The Zigzag Problem
Op IT
Subscriptions
User
Solvable by adding state to the traversal (or check path)
What if we also have reversed access to the subscription
payer?
E
d
Jo
The Many-to-Many Problem
Traversal becomes time consuming (e.g. M2M market)
However, we only needed to implement the rule for direct access to
sub.
The nodes Op & IT may be connected through many
subscriptions
Op IT
Subscription
Access
User
Does the user
have access to
department Op?
Deployment View
• Two equal instances of Neo4j embedded in Tomcat
• Access through Java API due to need for custom logic
• Using Neo4j 1.8 without HA (did not like ZooKeeper)
RDBMS
Resource
Authorization
Message Queue
tx log
Neo4j
Resource
Authorization
Neo4j
Dual Model Cost
There are some drawbacks with dual models
also
• Not possible to simply join the ACL with resource
tables in the relational database - queries needed
redesign
• The complexity added by code and infrastructure
necessary to manage an additional model.
• Not ordinary competence (in Norway at least)
Unexplored Areas
Combining Access Control List & Graph
• Best of both worlds (simple logic, fast lookup)
Algorithm
– Find all affected users when the graph is updated
– Invalidate users access control list
– Calculate all accessible resources for each user
– Store result in users access control list
Could then skip the U-turn and many-to-many problem.
Was is worth it?
Yes!
The user experience is important in
Telenor
Questions?
Web References
• Telenor Norway
• The Project - How NOSQL Paid off for
Telenor
• JavaWorld - Graphs for Security

Weitere ähnliche Inhalte

Was ist angesagt?

Cloudstate - Towards Stateful Serverless
Cloudstate - Towards Stateful ServerlessCloudstate - Towards Stateful Serverless
Cloudstate - Towards Stateful ServerlessLightbend
 
Webinar: Large Scale Graph Processing with IBM Power Systems & Neo4j
Webinar: Large Scale Graph Processing with IBM Power Systems & Neo4jWebinar: Large Scale Graph Processing with IBM Power Systems & Neo4j
Webinar: Large Scale Graph Processing with IBM Power Systems & Neo4jNeo4j
 
Databus: LinkedIn's Change Data Capture Pipeline SOCC 2012
Databus: LinkedIn's Change Data Capture Pipeline SOCC 2012Databus: LinkedIn's Change Data Capture Pipeline SOCC 2012
Databus: LinkedIn's Change Data Capture Pipeline SOCC 2012Shirshanka Das
 
What is happening with my microservices?
What is happening with my microservices?What is happening with my microservices?
What is happening with my microservices?Israel Blancas
 
50 Shades of Data - how, when and why Big, Fast, Relational, NoSQL, Elastic, ...
50 Shades of Data - how, when and why Big, Fast, Relational, NoSQL, Elastic, ...50 Shades of Data - how, when and why Big, Fast, Relational, NoSQL, Elastic, ...
50 Shades of Data - how, when and why Big, Fast, Relational, NoSQL, Elastic, ...Lucas Jellema
 
Micro services Architecture
Micro services ArchitectureMicro services Architecture
Micro services ArchitectureAraf Karsh Hamid
 
Oracle Service Bus and Oracle SOA Suite in the Mobile World
Oracle Service Bus and Oracle SOA Suite in the Mobile WorldOracle Service Bus and Oracle SOA Suite in the Mobile World
Oracle Service Bus and Oracle SOA Suite in the Mobile WorldGuido Schmutz
 
Microservices, Kafka Streams and KafkaEsque
Microservices, Kafka Streams and KafkaEsqueMicroservices, Kafka Streams and KafkaEsque
Microservices, Kafka Streams and KafkaEsqueconfluent
 
Lessons from Large-Scale Cloud Software at Databricks
Lessons from Large-Scale Cloud Software at DatabricksLessons from Large-Scale Cloud Software at Databricks
Lessons from Large-Scale Cloud Software at DatabricksMatei Zaharia
 
Oracle ZDM KamaleshRamasamy Sangam2020
Oracle ZDM KamaleshRamasamy Sangam2020Oracle ZDM KamaleshRamasamy Sangam2020
Oracle ZDM KamaleshRamasamy Sangam2020Kamalesh Ramasamy
 
Which Change Data Capture Strategy is Right for You?
Which Change Data Capture Strategy is Right for You?Which Change Data Capture Strategy is Right for You?
Which Change Data Capture Strategy is Right for You?Precisely
 
NoSQL and Spatial Database Capabilities using PostgreSQL
NoSQL and Spatial Database Capabilities using PostgreSQLNoSQL and Spatial Database Capabilities using PostgreSQL
NoSQL and Spatial Database Capabilities using PostgreSQLEDB
 
Lean Enterprise, Microservices and Big Data
Lean Enterprise, Microservices and Big DataLean Enterprise, Microservices and Big Data
Lean Enterprise, Microservices and Big DataStylight
 
Development of concurrent services using In-Memory Data Grids
Development of concurrent services using In-Memory Data GridsDevelopment of concurrent services using In-Memory Data Grids
Development of concurrent services using In-Memory Data Gridsjlorenzocima
 
CNCF Live Webinar: Kubernetes 1.23
CNCF Live Webinar: Kubernetes 1.23CNCF Live Webinar: Kubernetes 1.23
CNCF Live Webinar: Kubernetes 1.23LibbySchulze
 
#dbhouseparty - Should I be building Microservices?
#dbhouseparty - Should I be building Microservices?#dbhouseparty - Should I be building Microservices?
#dbhouseparty - Should I be building Microservices?Tammy Bednar
 
Un guide complet pour la migration de bases de données héritées vers PostgreSQL
Un guide complet pour la migration de bases de données héritées vers PostgreSQLUn guide complet pour la migration de bases de données héritées vers PostgreSQL
Un guide complet pour la migration de bases de données héritées vers PostgreSQLEDB
 

Was ist angesagt? (20)

Oracle GoldenGate
Oracle GoldenGate Oracle GoldenGate
Oracle GoldenGate
 
Cloudstate - Towards Stateful Serverless
Cloudstate - Towards Stateful ServerlessCloudstate - Towards Stateful Serverless
Cloudstate - Towards Stateful Serverless
 
Webinar: Large Scale Graph Processing with IBM Power Systems & Neo4j
Webinar: Large Scale Graph Processing with IBM Power Systems & Neo4jWebinar: Large Scale Graph Processing with IBM Power Systems & Neo4j
Webinar: Large Scale Graph Processing with IBM Power Systems & Neo4j
 
Databus: LinkedIn's Change Data Capture Pipeline SOCC 2012
Databus: LinkedIn's Change Data Capture Pipeline SOCC 2012Databus: LinkedIn's Change Data Capture Pipeline SOCC 2012
Databus: LinkedIn's Change Data Capture Pipeline SOCC 2012
 
What is happening with my microservices?
What is happening with my microservices?What is happening with my microservices?
What is happening with my microservices?
 
50 Shades of Data - how, when and why Big, Fast, Relational, NoSQL, Elastic, ...
50 Shades of Data - how, when and why Big, Fast, Relational, NoSQL, Elastic, ...50 Shades of Data - how, when and why Big, Fast, Relational, NoSQL, Elastic, ...
50 Shades of Data - how, when and why Big, Fast, Relational, NoSQL, Elastic, ...
 
Micro services Architecture
Micro services ArchitectureMicro services Architecture
Micro services Architecture
 
Oracle Service Bus and Oracle SOA Suite in the Mobile World
Oracle Service Bus and Oracle SOA Suite in the Mobile WorldOracle Service Bus and Oracle SOA Suite in the Mobile World
Oracle Service Bus and Oracle SOA Suite in the Mobile World
 
Microservices, Kafka Streams and KafkaEsque
Microservices, Kafka Streams and KafkaEsqueMicroservices, Kafka Streams and KafkaEsque
Microservices, Kafka Streams and KafkaEsque
 
Lessons from Large-Scale Cloud Software at Databricks
Lessons from Large-Scale Cloud Software at DatabricksLessons from Large-Scale Cloud Software at Databricks
Lessons from Large-Scale Cloud Software at Databricks
 
Oracle ZDM KamaleshRamasamy Sangam2020
Oracle ZDM KamaleshRamasamy Sangam2020Oracle ZDM KamaleshRamasamy Sangam2020
Oracle ZDM KamaleshRamasamy Sangam2020
 
Which Change Data Capture Strategy is Right for You?
Which Change Data Capture Strategy is Right for You?Which Change Data Capture Strategy is Right for You?
Which Change Data Capture Strategy is Right for You?
 
NoSQL and Spatial Database Capabilities using PostgreSQL
NoSQL and Spatial Database Capabilities using PostgreSQLNoSQL and Spatial Database Capabilities using PostgreSQL
NoSQL and Spatial Database Capabilities using PostgreSQL
 
Lean Enterprise, Microservices and Big Data
Lean Enterprise, Microservices and Big DataLean Enterprise, Microservices and Big Data
Lean Enterprise, Microservices and Big Data
 
Development of concurrent services using In-Memory Data Grids
Development of concurrent services using In-Memory Data GridsDevelopment of concurrent services using In-Memory Data Grids
Development of concurrent services using In-Memory Data Grids
 
IBM Cloud Direct Link 2.0
IBM Cloud Direct Link 2.0IBM Cloud Direct Link 2.0
IBM Cloud Direct Link 2.0
 
CNCF Live Webinar: Kubernetes 1.23
CNCF Live Webinar: Kubernetes 1.23CNCF Live Webinar: Kubernetes 1.23
CNCF Live Webinar: Kubernetes 1.23
 
Zero Downtime Migration
Zero Downtime MigrationZero Downtime Migration
Zero Downtime Migration
 
#dbhouseparty - Should I be building Microservices?
#dbhouseparty - Should I be building Microservices?#dbhouseparty - Should I be building Microservices?
#dbhouseparty - Should I be building Microservices?
 
Un guide complet pour la migration de bases de données héritées vers PostgreSQL
Un guide complet pour la migration de bases de données héritées vers PostgreSQLUn guide complet pour la migration de bases de données héritées vers PostgreSQL
Un guide complet pour la migration de bases de données héritées vers PostgreSQL
 

Ähnlich wie Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor - GraphConnect San Francisco 2013

Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor...
Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor...Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor...
Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor...Neo4j
 
Using Graph Databases in Real-time to Solve Resource Authorization at Telenor...
Using Graph Databases in Real-time to Solve Resource Authorization at Telenor...Using Graph Databases in Real-time to Solve Resource Authorization at Telenor...
Using Graph Databases in Real-time to Solve Resource Authorization at Telenor...Neo4j
 
Maximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical DemonstrationMaximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical DemonstrationDenodo
 
IT PRO | Connections 2020 : Introduction to Logic Apps and automation solutio...
IT PRO | Connections 2020 : Introduction to Logic Apps and automation solutio...IT PRO | Connections 2020 : Introduction to Logic Apps and automation solutio...
IT PRO | Connections 2020 : Introduction to Logic Apps and automation solutio...George Grammatikos
 
PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?
PLNOG 3: Tomasz Mikołajczyk -  Data scalability. Why you should care?PLNOG 3: Tomasz Mikołajczyk -  Data scalability. Why you should care?
PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?PROIDEA
 
Peek into Neo4j Product Strategy and Roadmap
Peek into Neo4j Product Strategy and RoadmapPeek into Neo4j Product Strategy and Roadmap
Peek into Neo4j Product Strategy and RoadmapNeo4j
 
Strata+Hadoop 2015 NYC End User Panel on Real-Time Data Analytics
Strata+Hadoop 2015 NYC End User Panel on Real-Time Data AnalyticsStrata+Hadoop 2015 NYC End User Panel on Real-Time Data Analytics
Strata+Hadoop 2015 NYC End User Panel on Real-Time Data AnalyticsSingleStore
 
Building data pipelines at Shopee with DEC
Building data pipelines at Shopee with DECBuilding data pipelines at Shopee with DEC
Building data pipelines at Shopee with DECRim Zaidullin
 
RESUME_NEHA _SELENIUM
RESUME_NEHA _SELENIUMRESUME_NEHA _SELENIUM
RESUME_NEHA _SELENIUMNeha Samal
 
Apricot2017 Request tracing in distributed environment
Apricot2017 Request tracing in distributed environmentApricot2017 Request tracing in distributed environment
Apricot2017 Request tracing in distributed environmentHieu LE ☁
 
Logging/Request Tracing in Distributed Environment
Logging/Request Tracing in Distributed EnvironmentLogging/Request Tracing in Distributed Environment
Logging/Request Tracing in Distributed EnvironmentAPNIC
 
Decomposing the Monolith (Riga Dev Days 2019)
Decomposing the Monolith (Riga Dev Days 2019)Decomposing the Monolith (Riga Dev Days 2019)
Decomposing the Monolith (Riga Dev Days 2019)Dennis Doomen
 
Importing Large Sets of Content from Trusted Partners into your Repository
Importing Large Sets of Content from Trusted Partners into your RepositoryImporting Large Sets of Content from Trusted Partners into your Repository
Importing Large Sets of Content from Trusted Partners into your RepositoryBlueFish
 
2010/09 - Database Architechs - Performance & Tuning Tool
2010/09 - Database Architechs - Performance & Tuning Tool2010/09 - Database Architechs - Performance & Tuning Tool
2010/09 - Database Architechs - Performance & Tuning ToolDatabase Architechs
 

Ähnlich wie Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor - GraphConnect San Francisco 2013 (20)

Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor...
Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor...Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor...
Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor...
 
Using Graph Databases in Real-time to Solve Resource Authorization at Telenor...
Using Graph Databases in Real-time to Solve Resource Authorization at Telenor...Using Graph Databases in Real-time to Solve Resource Authorization at Telenor...
Using Graph Databases in Real-time to Solve Resource Authorization at Telenor...
 
Maximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical DemonstrationMaximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
Maximizing Data Lake ROI with Data Virtualization: A Technical Demonstration
 
CV_DebarpanMukherjee
CV_DebarpanMukherjeeCV_DebarpanMukherjee
CV_DebarpanMukherjee
 
IT PRO | Connections 2020 : Introduction to Logic Apps and automation solutio...
IT PRO | Connections 2020 : Introduction to Logic Apps and automation solutio...IT PRO | Connections 2020 : Introduction to Logic Apps and automation solutio...
IT PRO | Connections 2020 : Introduction to Logic Apps and automation solutio...
 
PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?
PLNOG 3: Tomasz Mikołajczyk -  Data scalability. Why you should care?PLNOG 3: Tomasz Mikołajczyk -  Data scalability. Why you should care?
PLNOG 3: Tomasz Mikołajczyk - Data scalability. Why you should care?
 
Peek into Neo4j Product Strategy and Roadmap
Peek into Neo4j Product Strategy and RoadmapPeek into Neo4j Product Strategy and Roadmap
Peek into Neo4j Product Strategy and Roadmap
 
VamsiKrishna Maddiboina
VamsiKrishna MaddiboinaVamsiKrishna Maddiboina
VamsiKrishna Maddiboina
 
Strata+Hadoop 2015 NYC End User Panel on Real-Time Data Analytics
Strata+Hadoop 2015 NYC End User Panel on Real-Time Data AnalyticsStrata+Hadoop 2015 NYC End User Panel on Real-Time Data Analytics
Strata+Hadoop 2015 NYC End User Panel on Real-Time Data Analytics
 
Building data pipelines at Shopee with DEC
Building data pipelines at Shopee with DECBuilding data pipelines at Shopee with DEC
Building data pipelines at Shopee with DEC
 
RESUME_NEHA _SELENIUM
RESUME_NEHA _SELENIUMRESUME_NEHA _SELENIUM
RESUME_NEHA _SELENIUM
 
Apricot2017 Request tracing in distributed environment
Apricot2017 Request tracing in distributed environmentApricot2017 Request tracing in distributed environment
Apricot2017 Request tracing in distributed environment
 
Logging/Request Tracing in Distributed Environment
Logging/Request Tracing in Distributed EnvironmentLogging/Request Tracing in Distributed Environment
Logging/Request Tracing in Distributed Environment
 
Madhu_Resume
Madhu_ResumeMadhu_Resume
Madhu_Resume
 
Decomposing the Monolith (Riga Dev Days 2019)
Decomposing the Monolith (Riga Dev Days 2019)Decomposing the Monolith (Riga Dev Days 2019)
Decomposing the Monolith (Riga Dev Days 2019)
 
Subbu_WM
Subbu_WMSubbu_WM
Subbu_WM
 
mayank_unix_sql_Jboss
mayank_unix_sql_Jbossmayank_unix_sql_Jboss
mayank_unix_sql_Jboss
 
Importing Large Sets of Content from Trusted Partners into your Repository
Importing Large Sets of Content from Trusted Partners into your RepositoryImporting Large Sets of Content from Trusted Partners into your Repository
Importing Large Sets of Content from Trusted Partners into your Repository
 
2010/09 - Database Architechs - Performance & Tuning Tool
2010/09 - Database Architechs - Performance & Tuning Tool2010/09 - Database Architechs - Performance & Tuning Tool
2010/09 - Database Architechs - Performance & Tuning Tool
 
DevOps Case Studies
DevOps Case StudiesDevOps Case Studies
DevOps Case Studies
 

Kürzlich hochgeladen

KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 

Kürzlich hochgeladen (20)

KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 

Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor - GraphConnect San Francisco 2013

  • 1. Graph Connect San Francisco – 4 Oct 2013 by Sebastian Verheughe Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor
  • 2. Telenor Norway Subsidiary of the Telenor Group 2 billions USD in mobile revenues 2012 Sebastian Verheughe Lead Developer for Neo4j solution Coding Architect
  • 3. Disclaimer The presentation is not identical to the implementation due to security reasons but shows how we have modeled and solved the problem in general. However, all presented data (numbers & charts) are real, unfiltered and extracted from the production logs
  • 4. A very aspect of our business
  • 5. Telenor Norway Middleware Services MOBILE MW BUSINESS LOGIC & DATA Providing business logic and data for all channels in the mobile value chain Handles users with access to X00,000 resources Backend Backend BackendBackend Backend Backend Channel Channel ChannelChannel Channel Channel used by 42 channels calls 35 sub-systems 10,000 code classes 500 requests/second 20,000 orders/day
  • 6. Our Problem 20 minutes to calculate all accessible resources 1500 lines of SQL to implement the authorization logic “solved” by caching data going stale and the solution did not scale…
  • 7. Finance Production HR Why a Graph Database? The questions we wanted answered required traversal of tree structures. Sub Sub Tablet Phone Access User Which resources does the user have access to? Sales Parent Company Part of Company Subscription Owner Uses Subscription Tablet
  • 8. Tailored Read Model The Model makes read queries as simple and efficient as possible. First find your questions then model your graph graph model = relational model
  • 9. High Level Architecture Classic MW Services RDBMS Resource Authorization Message Queue check access Clients tx log Neo4j other sources
  • 10. Conditional Rules ACCESS is given with the following include parameters: access to subsidiaries and access to content User Only find children of PARENT COMPANY given access to subsidiaries is allowed Only look at PART OF COMPANY given access to content is allowed Only look at SUBSCRIPTION OWNER given access to content is allowed
  • 11. Different Access Needs Umbrella Admin Access ContentAccess Subsidiaries & Content Super Admin Admin Access S&C
  • 12. Graph Algorithm Prerequisite: The user node 1. Follow all ACCESS relationships and read the access parameters on the relationship 2. Follow all PARENT COMPANY relationships given access to subsidiaries is allowed 3. Follow all PART OF COMPANY relationships given access to content is allowed 4. Follow all SUBSCRIPTION OWNER relationships given access to content is allowed
  • 13. Solution Value 1. Performance optimized from minutes to seconds. 2. Simplicity of writing and understanding business rules for the query traversal. 3. Scalability by performance allowing us to onboard more corporate customers (project business case) Autonomous Service with it’s own life-cycle and data repository.
  • 14. Authorization Complexity • Not a collection of isolated customer trees * • Not all users of a customer have equal access • Not a fixed schema, form or size for all customers • Real-time updated with customer & product data The data form a highly connected living graph * Covered later in Technical Details
  • 15. How we Started with Neo4j 1. Searched the internet for articles about graph database and different solutions. 2. Downloaded and quickly prototyped the solution we liked that matched our requirements (Neo4j). 3. Workshop with Neo4j and our project developers to quickly gain competence and ensure design QA. 4. Solution QA with Neo4j before production and help with performance issues / tuning.
  • 16. Lessons Learned • Choose a solution/technology that fits your problem • New way of thinking – build competence in org. • Profile your java code to make it really fast • Don’t put everything into the graph (functional creep) • Need to know how traversal works (e.g. shortest path) • Benchmark the graph to evaluate your traversal speed
  • 17. Alternative In-Memory RDBMS Option 1: Use existing database - Performance issues due to shared data / suboptimal structure - Complexity since SQL not designed for traversal Option 2: Separate database + Might reach same performance as graph db + Familiar technology - Complexity since SQL not designed for traversal Decided to go with our instinct Graph Database
  • 18. Different Graph Structures Company X: 147 000 Company Y: 52 000 Company Z: 95 000 1 700 ms 750 ms 1300 ms get all accessible subscriptions 2 000 1 000 Data from test – repeated prod sampling gave ~2.4 sec for 215,000 subscriptions
  • 19. Different Graph Structures 1 ms 1 ms 1 ms check access to single subscription Company X: 147 000 Company Y: 52 000 Company Z: 95 000 2 000 1 000
  • 20. Production Performance retrieve all accessible resources Check single resource access 1 ms No operational problems in production RDBMS Disk RDBMS (mem cached) Graph In-Heap Company X 12 min 18 sec < 2 sec Company Y 22 min 58 sec < 2 sec Company Z 3 min 15 sec < 2 sec
  • 22. Production Details Graph Size 27 million nodes (pre-warmed in heap) ~1x properties, ~2x relationships Traffic Volume ~1000 req/min during biz hours ~ 40K daily real-time updates Performance Avg: 1 ms, 99% < 4 ms, 99.9% < 9 ms JVM Sun 6, 20 GB Heap (~15 GB pre- warmed)
  • 23. Production Has Access Query Time (ms) Time (ms)
  • 24. Production All Queries Time (ms) Garbage collection
  • 25. Implementing the Algorithm Lets look at the Neo4j Traversal Framework Iterable<Node> getAccessibleResources(…) { Evaluator myEvaluator = … Expander myExpander = … return Traversal.description() .evaluator(myEvaluator) .expander(myExpander) .traverse(startNode).nodes(); }
  • 26. Implementing the Algorithm Evaluator is a simple filter, e.g. for Node type class MyEvaluator implements Evaluator { public Evaluation evaluate(Path path) { if <I am interested in this node> return Evaluation.INCLUDE_AND_CONTINUE; else return Evaluation.EXCLUDE_AND_CONTINUE; } }
  • 27. Implementing the Algorithm The custom Expander contains business rules! class ResAuthExpander implements PathExpander<PathExpander> { … public … expand(Path path, BranchState<…> state) { if (path.lastRelationship rel == ACCESS) accToSub = rel.getProperty(ACCESS_TO_SUBSIDIARIES); accToCont = rel.getProperty(ACCESS_TO_CONTENT); state.set( getExpander(accToSub, accToCont) ); } return state.get().expand(…) } Single expander class to control business
  • 28. Implementing the Algorithm Generates the valid relationships to traverse. public getExpander(boolean accToSub, boolean accToCont) { PathExpander exp = StandardExpander.DEFAULT.add(ACCESS,…); if (accToSub) exp.add(PARENT_COMPANY,…) if (accToCont) exp.add(PART_OF_COMPANY,…).add(SUBSCRIPTION_OWNER,…); return exp; } }
  • 29. U-Turn Strategy X Subscription Access User Reversing the traversal increases performance from n/2 to 2d where n and d are tree size and depth (we went from 1s to 1. 2. 3. 4. Does the user have access to subscription X? 5. 6. 7. 8. Up to find path quickly Down to check access
  • 30. The Zigzag Problem Op IT Subscriptions User Solvable by adding state to the traversal (or check path) What if we also have reversed access to the subscription payer? E d Jo
  • 31. The Many-to-Many Problem Traversal becomes time consuming (e.g. M2M market) However, we only needed to implement the rule for direct access to sub. The nodes Op & IT may be connected through many subscriptions Op IT Subscription Access User Does the user have access to department Op?
  • 32. Deployment View • Two equal instances of Neo4j embedded in Tomcat • Access through Java API due to need for custom logic • Using Neo4j 1.8 without HA (did not like ZooKeeper) RDBMS Resource Authorization Message Queue tx log Neo4j Resource Authorization Neo4j
  • 33. Dual Model Cost There are some drawbacks with dual models also • Not possible to simply join the ACL with resource tables in the relational database - queries needed redesign • The complexity added by code and infrastructure necessary to manage an additional model. • Not ordinary competence (in Norway at least)
  • 34. Unexplored Areas Combining Access Control List & Graph • Best of both worlds (simple logic, fast lookup) Algorithm – Find all affected users when the graph is updated – Invalidate users access control list – Calculate all accessible resources for each user – Store result in users access control list Could then skip the U-turn and many-to-many problem.
  • 35. Was is worth it? Yes! The user experience is important in Telenor
  • 37. Web References • Telenor Norway • The Project - How NOSQL Paid off for Telenor • JavaWorld - Graphs for Security

Hinweis der Redaktion

  1. Why: access to secret numbers, access to modify/delete subscriptions, possibility to send/receive messages
  2. We use Neo4j for our business critical services, both customer/product services , but also operational services.A channel is client type, e.g. the web solution for corporate customer, or helpdesk solution, or app, and may consist of many clients
  3. The project business case was based on a future point in time where we could not any more onboard any large corporate customers
  4. Drawing on the white board the required logic made us understand that a graph database might be a good solution
  5. Take the hit on write, and make read easy! (for us, read performance is the problem – not write performance)Also, don’t blindly copy tables/foreign keys into nodes and relationships – drop what’s not needed and remember that relationships may have properties in graph
  6. RDBMS is still mastering the data as it is used in many different use-cases where that is beneficial.
  7. TIME LEFT: 30 minutes (10 used)
  8. The last part is important to us. It was really hard to extract the resource authorization out of the relational database, but not we can much more easy replace the current implementation with another one in the future if neccesary.
  9. Production logs does not contain user data, so just one big organization was sampled to get production data for a specific customer
  10. TIME LEFT: 20 minutes (20 used)Graphperformance based on test environment, see charts in the technical section for production numbers not specific for a unique customer
  11. We only have detailed logs for a short while back – so we cannot review all data since production.First production two years ago with limited traffic, full production since spring 2013
  12. We always continue, since we also have our custom expander. This way, we have a clean separation of concern in our code.We also have more advanced filters peaking around the node before it decides to include or exclude the node
  13. This is the most important part of the code, the one place where we now are able to write down the business logic in a simple and natural way.Note that we only have ONE class containing the business rules independently of which use-case we are running.
  14. The relationships and directions that are allowed to traverse given the different switch parameters.
  15. This is possible since we have a tree graph. Demonstrates the importance of understanding how a graph works, because than you may greatly improve performance by smart traversal strategies.
  16. TIME LEFT: 10 minutes (30 used)
  17. Extra knowledge, such as which subscription you are