Hadoop User Group Ireland (HUG) Ireland - Eddie Baggot Presentation April 2016

BAE SYSTEMS PROPRIETARY1 Unpublished Work Copyright 2015 BAE Systems. All Rights Reserved.
(See final slide for restrictions on use.)
|
BAE SYSTEMS PROPRIETARY
BAE Systems Apache Spark GraphX and
GraphFrames
April 11th 2016
 Eddie Baggott

|
• Functional and Data Architect
• BAE Systems, Norkom
• Anti Fraud, AML, Compliance, Watch lists, Cyber Security
• Disclaimer
• All my own opinion
Introduction

|
• Graph databases are databases that use graph structures for semantic
queries with nodes, edges and properties to represent and store data.
• Storing and showing Networks
What are graph databases

|
• Finding networks
• Analyse Relationships
• What to see how customers and accounts are connected
• See the transactions between them
• Credit Card
• Comprised Devices
• AML Rings
• Insurance
• Unauthorized Trading
• Social Networks
• Uber – Lyft Cancel Wars
• Panama Papers
What are they used for

|
Customer behaviour
Relationships
Showing direction of payments , co-ownerships
Use different type of lines and shapes to give extra meanings
Width of lines can show bigger amounts

|
offshoreleaks.icij.org/nodes/262484
Start search with “mossack fonseca”
Panama Papers

|
Spider out one level
Panama Papers

|
Show more connections
Panama Papers

|
• Graph Databases
•  Neo4j, Titan ,OrientDB
•  Can Store and manage data
•  Transversal queries
• Processing Engine
• Spark , Giraph
•  GraphX
•  GraphFrames
• Can be complementary and used together e.g. MazeRunner
• Elastic Search Graph
•  New , uses search and term relevancy
Graph Databases : different approaches

|
Apache Spark
DataFrames
GraphFrames

|
• GraphX is a graph computation engine built on top of Spark that enables
users to interactively build, transform and reason about graph structured
data at scale. It comes complete with a library of common algorithms.
• Spark , based on RDDs
• Num Vertices, Num Edges ,Degrees
•  Algorithms
•  PageRank
•  Connected Components
•  Triangle Counting
GraphX

|
•  In Big Data “Hello World” is usually a “Word Count”, of Wikipedia
• So lets graph wiki
•  Clean the Data
•  Making a Vertex RDD
val vertices = articles.map(a => (pageHash(a.title), a.title))
•  Making the Edge RDD
val edges: RDD[Edge[Double]] = articles.flatMap { a =>
Edge(srcVid, dstVid, 1.0) }
•  Making the Graph
val graph = Graph(vertices, edges, "")
•  Run PageRank on Wikipedia
val dublinGraph = graph.subgraph(vpred = (v, t) =>
t.toLowerCase contains “dublin")
val prDublin = dublinGraph.staticPageRank(5)
titleAndPrGraph.vertices.top(10).print
GraphX Example

|
• GraphFrames support general graph processing, similar to Apache Spark’s
GraphX library. However, GraphFrames are built on top of Spark
DataFrames, resulting in some key advantages:
• Python, Java & Scala APIs: GraphFrames provide uniform APIs for all 3
languages. For the first time, all algorithms in GraphX are available from
Python & Java.
• Powerful queries: GraphFrames allow users to phrase queries in the
familiar, powerful APIs of Spark SQL and DataFrames.
• Saving & loading graphs: GraphFrames fully support DataFrame data
sources , allowing writing and reading graphs using many formats like
Parquet, JSON, and CSV.
• In GraphFrames, vertices and edges are represented as DataFrames,
allowing us to store arbitrary data with each vertex and edge
• http://spark-packages.org/package/graphframes/graphframes
Spark Graph Frames

|
Spark Graph Frames Example
Customer ID
Eddie 1
Alan 2
Matt 3
Deirdre 4
Bob 5
Sue 6
John 7
// Create Vertices ( customer ) and Edges payments )
Vertices = customers.select("Customer", "id").distinct()
Edges = payments.select("Sender","Receiver","Amount", "Country")
Graph = GraphFrame(Vertices, Edges)
Sender Receiver Amount Country
Eddie Matt 10,000 UK
Eddie Deirdre 15,000 Irl
Eddie Bob 25,000 USA
Alan Sue 32,000 USA
Alan John 43,000 USA
Matt Alan 50,000 Irl
Matt Deirdre 60,000 Irl
Matt Bob 120,000 USA

|
• Who sent more than 100k?
graph.vertices.filter(“amount> 100000").show
Matt
• Who sent to more than 2 people?
graph.inDegrees.filter("inDegree > 2").show
Eddie,Matt
• Who sent to most to Ireland?
graph.edges.filter(“country =‘Irl’” "). groupBy(”sender”).sum
•  Who are most connected?
results = graph.pageRank(resetProbability=0.15, maxIter=10)
display(results.vertices)
Spark Graph Frames Example

|
• Another way to see who is sending money to who
Chord Diagram

|
www.elastic.co/products/graph
• Find connections based on relevance
• 
Elastic Search : Graph

|
• Graph good for Finding networks and Analysing Relationships
• Different approaches
• Lots of visualization options
• Get the benefits of using Spark
• We’re hiring!
• http://www.baesystems.com/en/cybersecurity/careers
•  Any Questions?
Recap

FREEDOM OF INFORMATION ACT
This document (<projectreference><documentnumber>) contains confidential and commercially sensitive material
which is provided for the Authority’s internal use only and is not intended for general dissemination.
The information contained herein pertains to bodies dealing with security, national security and/or defence matters
that would be exempt under Sections 23, 24 and 26 of the Freedom of Information Act 2000 (FOIA). It also consists of
information which describes our methodologies, processes and commercial arrangements all of which would be exempt
from disclosure under Sections 41 and 43 of the Act.
Should the Authority receive any request for disclosure of the information provided in this document, the Authority is
requested to notify BAE Systems Applied Intelligence. BAE Systems Applied Intelligence shall provide every assistance
to the Authority in complying with its obligations under the Act.
BAE Systems Applied Intelligence’s point of contact for FOIA requests is:
Chief Counsel
Legal Department
BAE Systems Applied Intelligence
Surrey Research Park
Guildford Gu2 7YP
Telephone 01483 816082
|

BAE SYSTEMS
Surrey Research Park
Guildford
Surrey
GU2 7YP
United Kingdom
T: +44 (0)1483 816000
F: +44 (0)1483 816144
Copyright © 2015 BAE Systems. All Rights Reserved.
BAE SYSTEMS, the BAE SYSTEMS Logo and the product names referenced herein are trademarks of BAE Systems plc.
No part of this document may be copied, reproduced, adapted or redistributed in any form or by any means without
the express prior written consent of BAE Systems Applied Intelligence.
BAE Systems Applied Intelligence Limited registered in England and Wales Company No. 1337451 with its registered
office at Surrey Research Park, Guildford, England, GU2 7YP.
|

Hadoop User Group Ireland (HUG) Ireland - Eddie Baggot Presentation April 2016

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Hadoop User Group Ireland (HUG) Ireland - Eddie Baggot Presentation April 2016

Ähnlich wie Hadoop User Group Ireland (HUG) Ireland - Eddie Baggot Presentation April 2016 (20)

Mehr von John Mulhall

Mehr von John Mulhall (13)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Hadoop User Group Ireland (HUG) Ireland - Eddie Baggot Presentation April 2016