SlideShare ist ein Scribd-Unternehmen logo
1 von 37
Downloaden Sie, um offline zu lesen
REST::Neo4p
A Perl Driver for Neo4j
Mark A. Jensen
SRA International, Inc.
1
https://github.com/majensen/rest-neo4p.git
• Perler since 2000
• CPAN contributor (MAJENSEN) since 2009
• BioPerl Core Developer
• Scientific Project Director, The Cancer Genome Atlas
Data Coordinating Center
• @thinkinator, LinkedIn
2
Motivation
• TCGA: Biospecimen, Clinical, Genomic Data
– complex
– growing
– evolving technologies
– evolving policies
– need for precise accounting
• Customer suggested Neo4j
– I wanted to play with it, but in Perl
3
Feeping Creaturism
• 2012 - Some Perl experiments out there,
nothing complete
• Got excited
• People started using it
• Sucked into the open-source attractor
MaintenanceGlory
4
Dog Food I
5
Dog Food II
6
Woof!
Neo4p Classes
7
Design Goals
• "OGM" – Perl 5 objects backed by the graph
• User should never have to deal with a REST endpoint*
*Unless she wants to.
• User should never have to deal with a Cypher query†
†Unless he wants to.
• Robust enough for production code
– System should approach complete coverage of the REST
service
– System should be robust to REST API changes and server
backward-compatible (or at least version-aware)
• Take advantage of the self-describing features of the API
8
REST::Neo4p core objects
• Are Node, Relationship, Index
– Index objects represent legacy (v1.0) indexes
– v2.0 “background” indexes handled in Schema
• Are blessed scalar refs : "Inside-out object" pattern
– the scalar value is the item ID (or index name)
– For any object $obj, $$obj (the ID) is exactly what you need
for constructing the API calls
• Are subclasses of Entity
– Entity does the object table handling, JSON-to-object
conversion and HTTP agent calls
– Isolates most of the kludges necessary to handle the few
API inconsistencies that exist(ed)
9
Auto-accessors
• You can treat properties as object fields if
desired
• Caveat: this may not make sense for your application (not
every node needs to have the same properties, but every
object will possess the accessors currently)
10
Batch Calls
• Certain situations (database loading, e.g.)
make sense to batch : do many things in one
API call rather than many single calls
• REST API provides this functionality
• How to make it "natural" in the context of
working with objects?
– Use Perl prototyping sugar to create a "batch
block"
11
Example:
Rather than call the server for every line, you can mix in
REST::Neo4p::Batch, and then use a batch {} block:
12
Calls within
block are
collected and
deferred
13
You can execute more complex logic within the
batch block, and keep the objects beyond it:
14
But miracles are not yet implemented:
Object here doesn't really exist yet…
How does that work?
• Agent module isolates all bona fide calls
– very few kludges to core object modules req'd
• batch() puts the agent into “batch mode” and
executes wrapped code
– agent stores incoming calls as JSON in a queue
• After wrapped code is executed, batch() switches
agent back to normal mode and has it call the
batch endpoint with the queue contents
• Batch processes the response and creates objects
if requested
15
Batch Profiling
• Used Devel::NYTProf, nytprofhtml
– Flame graph:
16
Vertical : unique call stack (call on top is on CPU)
Horizontal : relative time spent in that call stack configuration
Color : makes it look like a flame
Batch Profiling
17
Batch Profiling
18
batch keep : 1.1 of 1.2s
batch discard: 1.0 of 1.1s
no batch: 13.0 of 13.9 s
Batch/keep
Batch Profiling
19
No batch
HTTP Agent
20
Agent
• Is transparent
– But can always see it with REST::Neo4p->agent
– Agent module alone meant to be useful and independent
• Elicits and uses the API self-discovery feature on
connect()
• Isolates all HTTP requests and responses
• Captures and distinguishes API and HTTP errors
– emits REST::Neo4p::Exceptions objects
• [Instance] Is a subclass of a "real" user agent:
– LWP::UserAgent
– Mojo::UserAgent, or
– HTTP::Thin
21
Working within API Self-Description
22
Get first level of
actions
Register actions
Get ‘data’ level
of actionsRegister more
actions
Kludge around
missing actions
Working within API Self-Description
23
• Get the list of actions with
– $agent->available_actions
• And AUTOLOAD will provide (see pod for args):
– $agent->get_<action>()
– $agent->put_<action>()
– $agent->post_<action>()
– $agent->delete_<action>()
• Other accessors, e.g. node(), return the
appropriate URL for your server
Agent Profiling
24
lwp: 2.5 of 2.7 s
mojo: 3.3 of 3.6s
thin: 2.4 of 2.6s
App-level Constraints
25
Use Case
You start out with a set of well categorized things, that
have some well defined relationships.
Each thing will be represented as a node, that's fine. But,
You want to guarantee (to your client, for example) that
1. You can classify every node you add or read
unambiguously into a well-defined group;
2. You never relate two nodes belonging to particular
groups in a way that doesn't make sense according to
your well-defined relationships.
26
Constrain/Constraint
• Now, v2.0 allows integrated Labels and unique
constraints and prevents deletion of connected
nodes, but…
• REST::Neo4p::Constrain - An add-in for
constraining (or validating)
– property values
– connections (relationships) based on node properties
– relationship types
according to flexible specifications
27
Constrain/Constraint
• Multiple modes:
– Automatic (throws exception if constraint
violated)
– Manual (validation function returns false if
constraint violated)
– Suspended (lift constraint processing when
desired)
• Freeze/Thaw (in JSON) constraint
specifications for reuse
28
29
Open the POD now, HAL.
Cypher Queries
• REST::Neo4p::Query takes a familiar, DBI-like
approach
– Prepare, execute, fetch
– "rows" returned are arrays containing scalars,
Node objects, and/or Relationship objects
– If a query returns a path, a Path object (a simple
container) is returned
30
31
Cypher Queries
• Prepare and execute with parameter substitutions
32
Do This!
Not This!
Cypher Queries
• Transactions are supported when you have
v2.0.1 server or greater
– started with REST::Neo4p->begin_work()
– committed with REST::Neo4p->commit()
– canceled with REST::Neo4p->rollback()
(here, the class looks like the database handle in
DBI, in fact…)
33
DBI – DBD::Neo4p
• Yes, you can really do this:
34
35
Glory!
Maintenance.
Future Directions/Contribution Ideas
• Get it onto GitHub
https://github.com/majensen/rest-neo4p.git
• Make batch response parsing more efficient
– e.g., don't stream if response is not huge
• Beautify and deodorize
• Completely touch-free testing
• Add traversal functionality
• Could Neo4p play together with DBIx::Class? (i.e.,
could it be a real OGM?)
36
Thanks!
37
https://github.com/majensen/rest-neo4p.git

Más contenido relacionado

Was ist angesagt?

Ractor's speed is not light-speed
Ractor's speed is not light-speedRactor's speed is not light-speed
Ractor's speed is not light-speedSATOSHI TAGOMORI
 
FlutterでGraphQLを扱う
FlutterでGraphQLを扱うFlutterでGraphQLを扱う
FlutterでGraphQLを扱うIgaHironobu
 
RxJava from the trenches
RxJava from the trenchesRxJava from the trenches
RxJava from the trenchesPeter Hendriks
 
PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...
PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...
PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...Puppet
 
Java JSON Parser Comparison
Java JSON Parser ComparisonJava JSON Parser Comparison
Java JSON Parser ComparisonAllan Huang
 
Stream processing from single node to a cluster
Stream processing from single node to a clusterStream processing from single node to a cluster
Stream processing from single node to a clusterGal Marder
 
Data collection in AWS at Schibsted
Data collection in AWS at SchibstedData collection in AWS at Schibsted
Data collection in AWS at SchibstedLars Marius Garshol
 
Oak, the architecture of Apache Jackrabbit 3
Oak, the architecture of Apache Jackrabbit 3Oak, the architecture of Apache Jackrabbit 3
Oak, the architecture of Apache Jackrabbit 3Jukka Zitting
 
Building Distributed Systems in Scala
Building Distributed Systems in ScalaBuilding Distributed Systems in Scala
Building Distributed Systems in ScalaAlex Payne
 
Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...
Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...
Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...Lucidworks
 
Open Ldap Integration and Configuration with Lifray 6.2
Open Ldap Integration and Configuration with Lifray 6.2Open Ldap Integration and Configuration with Lifray 6.2
Open Ldap Integration and Configuration with Lifray 6.2Vinaykumar Hebballi
 
Scala.js for large and complex frontend apps
Scala.js for large and complex frontend appsScala.js for large and complex frontend apps
Scala.js for large and complex frontend appsOtto Chrons
 
libAttachSQL, The Next-Generation C Connector For MySQL
libAttachSQL, The Next-Generation C Connector For MySQLlibAttachSQL, The Next-Generation C Connector For MySQL
libAttachSQL, The Next-Generation C Connector For MySQLAndrew Hutchings
 
How to develop the Standard Libraries of Ruby?
How to develop the Standard Libraries of Ruby?How to develop the Standard Libraries of Ruby?
How to develop the Standard Libraries of Ruby?Hiroshi SHIBATA
 
Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...
Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...
Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...Lucidworks
 
I18nize Scala programs à la gettext
I18nize Scala programs à la gettextI18nize Scala programs à la gettext
I18nize Scala programs à la gettextNgoc Dao
 
Writing Well-Behaved Unix Utilities
Writing Well-Behaved Unix UtilitiesWriting Well-Behaved Unix Utilities
Writing Well-Behaved Unix UtilitiesRob Miller
 
Search-time Parallelism: Presented by Shikhar Bhushan, Etsy
Search-time Parallelism: Presented by Shikhar Bhushan, EtsySearch-time Parallelism: Presented by Shikhar Bhushan, Etsy
Search-time Parallelism: Presented by Shikhar Bhushan, EtsyLucidworks
 

Was ist angesagt? (20)

Ractor's speed is not light-speed
Ractor's speed is not light-speedRactor's speed is not light-speed
Ractor's speed is not light-speed
 
FlutterでGraphQLを扱う
FlutterでGraphQLを扱うFlutterでGraphQLを扱う
FlutterでGraphQLを扱う
 
RxJava from the trenches
RxJava from the trenchesRxJava from the trenches
RxJava from the trenches
 
PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...
PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...
PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces an...
 
Java JSON Parser Comparison
Java JSON Parser ComparisonJava JSON Parser Comparison
Java JSON Parser Comparison
 
Stream processing from single node to a cluster
Stream processing from single node to a clusterStream processing from single node to a cluster
Stream processing from single node to a cluster
 
Data collection in AWS at Schibsted
Data collection in AWS at SchibstedData collection in AWS at Schibsted
Data collection in AWS at Schibsted
 
Oak, the architecture of Apache Jackrabbit 3
Oak, the architecture of Apache Jackrabbit 3Oak, the architecture of Apache Jackrabbit 3
Oak, the architecture of Apache Jackrabbit 3
 
Building Distributed Systems in Scala
Building Distributed Systems in ScalaBuilding Distributed Systems in Scala
Building Distributed Systems in Scala
 
Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...
Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...
Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn...
 
Ruby on the JVM
Ruby on the JVMRuby on the JVM
Ruby on the JVM
 
Open Ldap Integration and Configuration with Lifray 6.2
Open Ldap Integration and Configuration with Lifray 6.2Open Ldap Integration and Configuration with Lifray 6.2
Open Ldap Integration and Configuration with Lifray 6.2
 
Scala.js for large and complex frontend apps
Scala.js for large and complex frontend appsScala.js for large and complex frontend apps
Scala.js for large and complex frontend apps
 
libAttachSQL, The Next-Generation C Connector For MySQL
libAttachSQL, The Next-Generation C Connector For MySQLlibAttachSQL, The Next-Generation C Connector For MySQL
libAttachSQL, The Next-Generation C Connector For MySQL
 
How to develop the Standard Libraries of Ruby?
How to develop the Standard Libraries of Ruby?How to develop the Standard Libraries of Ruby?
How to develop the Standard Libraries of Ruby?
 
Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...
Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...
Solr Troubleshooting - Treemap Approach: Presented by Alexandre Rafolovitch, ...
 
I18nize Scala programs à la gettext
I18nize Scala programs à la gettextI18nize Scala programs à la gettext
I18nize Scala programs à la gettext
 
Reaching the lambda heaven
Reaching the lambda heavenReaching the lambda heaven
Reaching the lambda heaven
 
Writing Well-Behaved Unix Utilities
Writing Well-Behaved Unix UtilitiesWriting Well-Behaved Unix Utilities
Writing Well-Behaved Unix Utilities
 
Search-time Parallelism: Presented by Shikhar Bhushan, Etsy
Search-time Parallelism: Presented by Shikhar Bhushan, EtsySearch-time Parallelism: Presented by Shikhar Bhushan, Etsy
Search-time Parallelism: Presented by Shikhar Bhushan, Etsy
 

Andere mochten auch

Pension slides 15 mts.
Pension slides 15 mts.Pension slides 15 mts.
Pension slides 15 mts.Exwel Trust
 
Glasgow University Freshers' Week
Glasgow University Freshers' WeekGlasgow University Freshers' Week
Glasgow University Freshers' WeekNick Wade
 
Eze sg meeting mitke overall progress-poland_30.06.2010
Eze sg meeting mitke overall progress-poland_30.06.2010Eze sg meeting mitke overall progress-poland_30.06.2010
Eze sg meeting mitke overall progress-poland_30.06.2010Eze
 
Hy solution사례(5)저축보험가입고객
Hy solution사례(5)저축보험가입고객Hy solution사례(5)저축보험가입고객
Hy solution사례(5)저축보험가입고객valuasset
 
高崎経済大学の受験..
高崎経済大学の受験..高崎経済大学の受験..
高崎経済大学の受験..kobayashimasakazu
 
Hy solution사례(5)저축보험가입고객
Hy solution사례(5)저축보험가입고객Hy solution사례(5)저축보험가입고객
Hy solution사례(5)저축보험가입고객valuasset
 
Tan sri syed mokhtar shah bin syed nor al
Tan sri syed mokhtar shah bin syed nor alTan sri syed mokhtar shah bin syed nor al
Tan sri syed mokhtar shah bin syed nor almimi
 
Where2.0_presentation_2010
Where2.0_presentation_2010Where2.0_presentation_2010
Where2.0_presentation_2010Roland Shield
 
EPM Live Overview
EPM Live OverviewEPM Live Overview
EPM Live OverviewEPM Live
 
高崎経済大学の受験..
高崎経済大学の受験..高崎経済大学の受験..
高崎経済大学の受験..kobayashimasakazu
 

Andere mochten auch (20)

Pension slides 15 mts.
Pension slides 15 mts.Pension slides 15 mts.
Pension slides 15 mts.
 
fOSSa2012- shruti - mutable instrument - open hardware diy 80s sonic aestheti...
fOSSa2012- shruti - mutable instrument - open hardware diy 80s sonic aestheti...fOSSa2012- shruti - mutable instrument - open hardware diy 80s sonic aestheti...
fOSSa2012- shruti - mutable instrument - open hardware diy 80s sonic aestheti...
 
Power of one
Power of onePower of one
Power of one
 
Glasgow University Freshers' Week
Glasgow University Freshers' WeekGlasgow University Freshers' Week
Glasgow University Freshers' Week
 
Jejak Darah Setelah Berita : Pembunuhan Jurnalis Radar Bali, AA Prabangsa
Jejak Darah Setelah Berita : Pembunuhan Jurnalis Radar Bali, AA PrabangsaJejak Darah Setelah Berita : Pembunuhan Jurnalis Radar Bali, AA Prabangsa
Jejak Darah Setelah Berita : Pembunuhan Jurnalis Radar Bali, AA Prabangsa
 
Eze sg meeting mitke overall progress-poland_30.06.2010
Eze sg meeting mitke overall progress-poland_30.06.2010Eze sg meeting mitke overall progress-poland_30.06.2010
Eze sg meeting mitke overall progress-poland_30.06.2010
 
Fred and Lee
Fred and LeeFred and Lee
Fred and Lee
 
Hy solution사례(5)저축보험가입고객
Hy solution사례(5)저축보험가입고객Hy solution사례(5)저축보험가입고객
Hy solution사례(5)저축보험가입고객
 
Setelah Insiden Jim Foley
Setelah Insiden Jim FoleySetelah Insiden Jim Foley
Setelah Insiden Jim Foley
 
Jeopardy Game
Jeopardy Game Jeopardy Game
Jeopardy Game
 
高崎経済大学の受験..
高崎経済大学の受験..高崎経済大学の受験..
高崎経済大学の受験..
 
Hy solution사례(5)저축보험가입고객
Hy solution사례(5)저축보험가입고객Hy solution사례(5)저축보험가입고객
Hy solution사례(5)저축보험가입고객
 
Tan sri syed mokhtar shah bin syed nor al
Tan sri syed mokhtar shah bin syed nor alTan sri syed mokhtar shah bin syed nor al
Tan sri syed mokhtar shah bin syed nor al
 
fOSSa2012- l grisoni - collbaoration between art and science
fOSSa2012- l grisoni - collbaoration between art and sciencefOSSa2012- l grisoni - collbaoration between art and science
fOSSa2012- l grisoni - collbaoration between art and science
 
Looking at the wetware stakeholders in communities - fossa2011
Looking at the wetware   stakeholders in communities - fossa2011Looking at the wetware   stakeholders in communities - fossa2011
Looking at the wetware stakeholders in communities - fossa2011
 
Where2.0_presentation_2010
Where2.0_presentation_2010Where2.0_presentation_2010
Where2.0_presentation_2010
 
EPM Live Overview
EPM Live OverviewEPM Live Overview
EPM Live Overview
 
Dokumen yang Memanaskan Jakarta
Dokumen yang Memanaskan JakartaDokumen yang Memanaskan Jakarta
Dokumen yang Memanaskan Jakarta
 
高崎経済大学の受験..
高崎経済大学の受験..高崎経済大学の受験..
高崎経済大学の受験..
 
Potret pers jakarta 2013
Potret pers jakarta 2013Potret pers jakarta 2013
Potret pers jakarta 2013
 

Ähnlich wie REST::Neo4p - Talk @ DC Perl Mongers

Kubernetes intro public - kubernetes meetup 4-21-2015
Kubernetes intro   public - kubernetes meetup 4-21-2015Kubernetes intro   public - kubernetes meetup 4-21-2015
Kubernetes intro public - kubernetes meetup 4-21-2015Rohit Jnagal
 
Kubernetes intro public - kubernetes user group 4-21-2015
Kubernetes intro   public - kubernetes user group 4-21-2015Kubernetes intro   public - kubernetes user group 4-21-2015
Kubernetes intro public - kubernetes user group 4-21-2015reallavalamp
 
Graph databases in computational bioloby: case of neo4j and TitanDB
Graph databases in computational bioloby: case of neo4j and TitanDBGraph databases in computational bioloby: case of neo4j and TitanDB
Graph databases in computational bioloby: case of neo4j and TitanDBAndrei KUCHARAVY
 
Dev nexus 2017
Dev nexus 2017Dev nexus 2017
Dev nexus 2017Roy Russo
 
GraphQL-PHP: Dos and don'ts
GraphQL-PHP: Dos and don'tsGraphQL-PHP: Dos and don'ts
GraphQL-PHP: Dos and don'tsVáclav Šír
 
Kubernetes Architecture - beyond a black box - Part 2
Kubernetes Architecture - beyond a black box - Part 2Kubernetes Architecture - beyond a black box - Part 2
Kubernetes Architecture - beyond a black box - Part 2Hao H. Zhang
 
DockerDay2015: Getting started with Google Container Engine
DockerDay2015: Getting started with Google Container EngineDockerDay2015: Getting started with Google Container Engine
DockerDay2015: Getting started with Google Container EngineDocker-Hanoi
 
Impala Architecture presentation
Impala Architecture presentationImpala Architecture presentation
Impala Architecture presentationhadooparchbook
 
Active Cloud DB at CloudComp '10
Active Cloud DB at CloudComp '10Active Cloud DB at CloudComp '10
Active Cloud DB at CloudComp '10Chris Bunch
 
Delivering big content at NBC News with RavenDB
Delivering big content at NBC News with RavenDBDelivering big content at NBC News with RavenDB
Delivering big content at NBC News with RavenDBJohn Bennett
 
Distributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Distributed Tensorflow with Kubernetes - data2day - Jakob KaralusDistributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Distributed Tensorflow with Kubernetes - data2day - Jakob KaralusJakob Karalus
 
Devnexus 2018
Devnexus 2018Devnexus 2018
Devnexus 2018Roy Russo
 
Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...
Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...
Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...Nilesh Panchal
 
Workflow Engines for Hadoop
Workflow Engines for HadoopWorkflow Engines for Hadoop
Workflow Engines for HadoopJoe Crobak
 
Exploring Java Heap Dumps (Oracle Code One 2018)
Exploring Java Heap Dumps (Oracle Code One 2018)Exploring Java Heap Dumps (Oracle Code One 2018)
Exploring Java Heap Dumps (Oracle Code One 2018)Ryan Cuprak
 
Current & Future Use-Cases of OpenDaylight
Current & Future Use-Cases of OpenDaylightCurrent & Future Use-Cases of OpenDaylight
Current & Future Use-Cases of OpenDaylightabhijit2511
 
Gwt and rpc use 2007 1
Gwt and rpc use 2007 1Gwt and rpc use 2007 1
Gwt and rpc use 2007 1Sam Muhanguzi
 
Membase East Coast Meetups
Membase East Coast MeetupsMembase East Coast Meetups
Membase East Coast MeetupsMembase
 
Kubernetes deep dive - - Huawei 2015-10
Kubernetes deep dive - - Huawei 2015-10Kubernetes deep dive - - Huawei 2015-10
Kubernetes deep dive - - Huawei 2015-10Vishnu Kannan
 

Ähnlich wie REST::Neo4p - Talk @ DC Perl Mongers (20)

Kubernetes intro public - kubernetes meetup 4-21-2015
Kubernetes intro   public - kubernetes meetup 4-21-2015Kubernetes intro   public - kubernetes meetup 4-21-2015
Kubernetes intro public - kubernetes meetup 4-21-2015
 
Kubernetes intro public - kubernetes user group 4-21-2015
Kubernetes intro   public - kubernetes user group 4-21-2015Kubernetes intro   public - kubernetes user group 4-21-2015
Kubernetes intro public - kubernetes user group 4-21-2015
 
Graph databases in computational bioloby: case of neo4j and TitanDB
Graph databases in computational bioloby: case of neo4j and TitanDBGraph databases in computational bioloby: case of neo4j and TitanDB
Graph databases in computational bioloby: case of neo4j and TitanDB
 
Dev nexus 2017
Dev nexus 2017Dev nexus 2017
Dev nexus 2017
 
GraphQL-PHP: Dos and don'ts
GraphQL-PHP: Dos and don'tsGraphQL-PHP: Dos and don'ts
GraphQL-PHP: Dos and don'ts
 
Kubernetes Architecture - beyond a black box - Part 2
Kubernetes Architecture - beyond a black box - Part 2Kubernetes Architecture - beyond a black box - Part 2
Kubernetes Architecture - beyond a black box - Part 2
 
DockerDay2015: Getting started with Google Container Engine
DockerDay2015: Getting started with Google Container EngineDockerDay2015: Getting started with Google Container Engine
DockerDay2015: Getting started with Google Container Engine
 
Impala Architecture presentation
Impala Architecture presentationImpala Architecture presentation
Impala Architecture presentation
 
Active Cloud DB at CloudComp '10
Active Cloud DB at CloudComp '10Active Cloud DB at CloudComp '10
Active Cloud DB at CloudComp '10
 
Delivering big content at NBC News with RavenDB
Delivering big content at NBC News with RavenDBDelivering big content at NBC News with RavenDB
Delivering big content at NBC News with RavenDB
 
Distributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Distributed Tensorflow with Kubernetes - data2day - Jakob KaralusDistributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Distributed Tensorflow with Kubernetes - data2day - Jakob Karalus
 
Devnexus 2018
Devnexus 2018Devnexus 2018
Devnexus 2018
 
Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...
Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...
Ruby on-rails-101-presentation-slides-for-a-five-day-introductory-course-1194...
 
Workflow Engines for Hadoop
Workflow Engines for HadoopWorkflow Engines for Hadoop
Workflow Engines for Hadoop
 
Exploring Java Heap Dumps (Oracle Code One 2018)
Exploring Java Heap Dumps (Oracle Code One 2018)Exploring Java Heap Dumps (Oracle Code One 2018)
Exploring Java Heap Dumps (Oracle Code One 2018)
 
Current & Future Use-Cases of OpenDaylight
Current & Future Use-Cases of OpenDaylightCurrent & Future Use-Cases of OpenDaylight
Current & Future Use-Cases of OpenDaylight
 
Intro to CakePHP
Intro to CakePHPIntro to CakePHP
Intro to CakePHP
 
Gwt and rpc use 2007 1
Gwt and rpc use 2007 1Gwt and rpc use 2007 1
Gwt and rpc use 2007 1
 
Membase East Coast Meetups
Membase East Coast MeetupsMembase East Coast Meetups
Membase East Coast Meetups
 
Kubernetes deep dive - - Huawei 2015-10
Kubernetes deep dive - - Huawei 2015-10Kubernetes deep dive - - Huawei 2015-10
Kubernetes deep dive - - Huawei 2015-10
 

Último

openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleopenEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleShane Coughlan
 
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...MyFAA
 
Steps to Successfully Hire Ionic Developers
Steps to Successfully Hire Ionic DevelopersSteps to Successfully Hire Ionic Developers
Steps to Successfully Hire Ionic Developersmichealwillson701
 
Revolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM GridRevolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM GridMathew Thomas
 
Practical Advice for FDA’s 510(k) Requirements.pdf
Practical Advice for FDA’s 510(k) Requirements.pdfPractical Advice for FDA’s 510(k) Requirements.pdf
Practical Advice for FDA’s 510(k) Requirements.pdfICS
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsconfluent
 
Unlocking AI: Navigating Open Source vs. Commercial Frontiers
Unlocking AI:Navigating Open Source vs. Commercial FrontiersUnlocking AI:Navigating Open Source vs. Commercial Frontiers
Unlocking AI: Navigating Open Source vs. Commercial FrontiersRaphaël Semeteys
 
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...telebusocialmarketin
 
8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.Ritesh Kanjee
 
VuNet software organisation powerpoint deck
VuNet software organisation powerpoint deckVuNet software organisation powerpoint deck
VuNet software organisation powerpoint deckNaval Singh
 
Leveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevLeveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevpmgdscunsri
 
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityRandy Shoup
 
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptxCYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptxBarakaMuyengi
 
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...Splashtop Inc
 
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurMinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurPriyadarshini T
 
Mobile App Development process | Expert Tips
Mobile App Development process | Expert TipsMobile App Development process | Expert Tips
Mobile App Development process | Expert Tipsmichealwillson701
 
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdfFlutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdfMind IT Systems
 
BATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data MeshBATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data MeshBATbern
 
Einstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdfEinstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdfCloudMetic
 
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startBuilding Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startMaxim Salnikov
 

Último (20)

openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleopenEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scale
 
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
 
Steps to Successfully Hire Ionic Developers
Steps to Successfully Hire Ionic DevelopersSteps to Successfully Hire Ionic Developers
Steps to Successfully Hire Ionic Developers
 
Revolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM GridRevolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM Grid
 
Practical Advice for FDA’s 510(k) Requirements.pdf
Practical Advice for FDA’s 510(k) Requirements.pdfPractical Advice for FDA’s 510(k) Requirements.pdf
Practical Advice for FDA’s 510(k) Requirements.pdf
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
 
Unlocking AI: Navigating Open Source vs. Commercial Frontiers
Unlocking AI:Navigating Open Source vs. Commercial FrontiersUnlocking AI:Navigating Open Source vs. Commercial Frontiers
Unlocking AI: Navigating Open Source vs. Commercial Frontiers
 
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
 
8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.
 
VuNet software organisation powerpoint deck
VuNet software organisation powerpoint deckVuNet software organisation powerpoint deck
VuNet software organisation powerpoint deck
 
Leveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevLeveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDev
 
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
 
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptxCYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
 
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
 
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurMinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
 
Mobile App Development process | Expert Tips
Mobile App Development process | Expert TipsMobile App Development process | Expert Tips
Mobile App Development process | Expert Tips
 
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdfFlutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
 
BATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data MeshBATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data Mesh
 
Einstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdfEinstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdf
 
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startBuilding Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to start
 

REST::Neo4p - Talk @ DC Perl Mongers

  • 1. REST::Neo4p A Perl Driver for Neo4j Mark A. Jensen SRA International, Inc. 1 https://github.com/majensen/rest-neo4p.git
  • 2. • Perler since 2000 • CPAN contributor (MAJENSEN) since 2009 • BioPerl Core Developer • Scientific Project Director, The Cancer Genome Atlas Data Coordinating Center • @thinkinator, LinkedIn 2
  • 3. Motivation • TCGA: Biospecimen, Clinical, Genomic Data – complex – growing – evolving technologies – evolving policies – need for precise accounting • Customer suggested Neo4j – I wanted to play with it, but in Perl 3
  • 4. Feeping Creaturism • 2012 - Some Perl experiments out there, nothing complete • Got excited • People started using it • Sucked into the open-source attractor MaintenanceGlory 4
  • 8. Design Goals • "OGM" – Perl 5 objects backed by the graph • User should never have to deal with a REST endpoint* *Unless she wants to. • User should never have to deal with a Cypher query† †Unless he wants to. • Robust enough for production code – System should approach complete coverage of the REST service – System should be robust to REST API changes and server backward-compatible (or at least version-aware) • Take advantage of the self-describing features of the API 8
  • 9. REST::Neo4p core objects • Are Node, Relationship, Index – Index objects represent legacy (v1.0) indexes – v2.0 “background” indexes handled in Schema • Are blessed scalar refs : "Inside-out object" pattern – the scalar value is the item ID (or index name) – For any object $obj, $$obj (the ID) is exactly what you need for constructing the API calls • Are subclasses of Entity – Entity does the object table handling, JSON-to-object conversion and HTTP agent calls – Isolates most of the kludges necessary to handle the few API inconsistencies that exist(ed) 9
  • 10. Auto-accessors • You can treat properties as object fields if desired • Caveat: this may not make sense for your application (not every node needs to have the same properties, but every object will possess the accessors currently) 10
  • 11. Batch Calls • Certain situations (database loading, e.g.) make sense to batch : do many things in one API call rather than many single calls • REST API provides this functionality • How to make it "natural" in the context of working with objects? – Use Perl prototyping sugar to create a "batch block" 11
  • 12. Example: Rather than call the server for every line, you can mix in REST::Neo4p::Batch, and then use a batch {} block: 12 Calls within block are collected and deferred
  • 13. 13 You can execute more complex logic within the batch block, and keep the objects beyond it:
  • 14. 14 But miracles are not yet implemented: Object here doesn't really exist yet…
  • 15. How does that work? • Agent module isolates all bona fide calls – very few kludges to core object modules req'd • batch() puts the agent into “batch mode” and executes wrapped code – agent stores incoming calls as JSON in a queue • After wrapped code is executed, batch() switches agent back to normal mode and has it call the batch endpoint with the queue contents • Batch processes the response and creates objects if requested 15
  • 16. Batch Profiling • Used Devel::NYTProf, nytprofhtml – Flame graph: 16 Vertical : unique call stack (call on top is on CPU) Horizontal : relative time spent in that call stack configuration Color : makes it look like a flame
  • 18. Batch Profiling 18 batch keep : 1.1 of 1.2s batch discard: 1.0 of 1.1s no batch: 13.0 of 13.9 s
  • 21. Agent • Is transparent – But can always see it with REST::Neo4p->agent – Agent module alone meant to be useful and independent • Elicits and uses the API self-discovery feature on connect() • Isolates all HTTP requests and responses • Captures and distinguishes API and HTTP errors – emits REST::Neo4p::Exceptions objects • [Instance] Is a subclass of a "real" user agent: – LWP::UserAgent – Mojo::UserAgent, or – HTTP::Thin 21
  • 22. Working within API Self-Description 22 Get first level of actions Register actions Get ‘data’ level of actionsRegister more actions Kludge around missing actions
  • 23. Working within API Self-Description 23 • Get the list of actions with – $agent->available_actions • And AUTOLOAD will provide (see pod for args): – $agent->get_<action>() – $agent->put_<action>() – $agent->post_<action>() – $agent->delete_<action>() • Other accessors, e.g. node(), return the appropriate URL for your server
  • 24. Agent Profiling 24 lwp: 2.5 of 2.7 s mojo: 3.3 of 3.6s thin: 2.4 of 2.6s
  • 26. Use Case You start out with a set of well categorized things, that have some well defined relationships. Each thing will be represented as a node, that's fine. But, You want to guarantee (to your client, for example) that 1. You can classify every node you add or read unambiguously into a well-defined group; 2. You never relate two nodes belonging to particular groups in a way that doesn't make sense according to your well-defined relationships. 26
  • 27. Constrain/Constraint • Now, v2.0 allows integrated Labels and unique constraints and prevents deletion of connected nodes, but… • REST::Neo4p::Constrain - An add-in for constraining (or validating) – property values – connections (relationships) based on node properties – relationship types according to flexible specifications 27
  • 28. Constrain/Constraint • Multiple modes: – Automatic (throws exception if constraint violated) – Manual (validation function returns false if constraint violated) – Suspended (lift constraint processing when desired) • Freeze/Thaw (in JSON) constraint specifications for reuse 28
  • 29. 29 Open the POD now, HAL.
  • 30. Cypher Queries • REST::Neo4p::Query takes a familiar, DBI-like approach – Prepare, execute, fetch – "rows" returned are arrays containing scalars, Node objects, and/or Relationship objects – If a query returns a path, a Path object (a simple container) is returned 30
  • 31. 31
  • 32. Cypher Queries • Prepare and execute with parameter substitutions 32 Do This! Not This!
  • 33. Cypher Queries • Transactions are supported when you have v2.0.1 server or greater – started with REST::Neo4p->begin_work() – committed with REST::Neo4p->commit() – canceled with REST::Neo4p->rollback() (here, the class looks like the database handle in DBI, in fact…) 33
  • 34. DBI – DBD::Neo4p • Yes, you can really do this: 34
  • 36. Future Directions/Contribution Ideas • Get it onto GitHub https://github.com/majensen/rest-neo4p.git • Make batch response parsing more efficient – e.g., don't stream if response is not huge • Beautify and deodorize • Completely touch-free testing • Add traversal functionality • Could Neo4p play together with DBIx::Class? (i.e., could it be a real OGM?) 36

Hinweis der Redaktion

  1. People are using it!