Fiducia & GAD IT AG: From Fraud Detection to Big Data Platform: Bringing Hadoop to the Enterprise

1. From Fraud Detection to Big Data Platform:
Bringing Hadoop to the Enterprise at
Fiducia & GAD IT AG
Daniel Schmitt & Florian Herrmann
October 25th 2016

2. 2
About us
9/15/2016World of Watson 2016
Daniel Schmitt (1985)
• Karlsruhe, Germany
• Business Intelligence Dep. at Fiducia & GAD IT AG
since 2009
Topics
Business Analytics Design and Implementation, Reporting, Planning
and all topics related to Analytics
Experience
Apache Hadoop, Cognos BI, Cognos TM1, GeoInformation Systems,
Cognos Enterprise Planning etc

3. 3
About us
9/15/2016World of Watson 2016
Florian Herrmann (1988)
• Karlsruhe, Germany
• Database Development Dep. at Fiducia & GAD IT AG
since 2013
Topics
Data Modelling and Database Design for core banking system,
Performance Optimization and in-house consulting for all topics related
to DBs
Experience
Apache Hadoop, Database Systems (DB2-Family, Oracle) etc

4. 1. The Challenge 2. The Solution
3. The Lessons Learned4. The Blueprint

5. 1. The Challenge
Make the fraudsters shiver

6. 6
Fiducia & GAD IT AG at a glance
9/15/2016World of Watson 2016
Computer Center
Services
Integration Platform Competence Center
Leading
Banking System

7. 7
Fiducia & GAD IT AG at a glance
9/15/2016World of Watson 2016
167,000 workstations in banks
6,300m accounting entries per year
79m active accounts
36,000 self-service terminals
550m ATM cash withdrawals per year

8. Requirements
• Evaluation of all user initiated online transactions on fraud suspicion
• Integration in core banking system and existing banking processes
(Fiducia & GAD is just the service provider not the owner!)
• Model based on customer behavior
• Flexible system design for a fast reaction on new fraud patterns
8
Fraud Detection for online banking
9/15/2016World of Watson 2016

9. 9
Fraud Detection for online banking
9/15/2016World of Watson 2016
Millions of transactions per day
Up to 100 transactions per second
Evaluation in less than 100 milliseconds
System adjustment in minutes
Be prepared for new datasources or -formats

10. 10
Fraud Detection for online banking
9/15/2016World of Watson 2016
Transaction handling
Fraud Detection System
Development of
evaluation models
Storage of all
transactions
Evaluation
in milliseconds
Flexible adjustment
Evaluate Transaction
Accounting and Forwarding

11. 2. The Solution
One elephant to rule them all

12. The Solution
9/15/2016World of Watson 201612
Velocity
Realtime evaluation
of incoming data.
Access on large data
volume within
milliseconds
Variety
Transactional data
won’t be enough in
foreseeable future
Volume
Store millions of
transaction details
each day over years
Flexibility
Quick response on
changing fraud patterns.
Integration of complex
data structures.
Model development
based on current events

13. The Solution
13 9/15/2016World of Watson 2016
Pig
Spark
Hive
Data Access
Storm
Phoenix
HBase
Governance
Sqoop
Kafka
Flume
Hadoop & YARN
Operations
Security
RangerKnox
Oozie
Zoo-
keeper
Ambari

14. The Solution
14 9/15/2016World of Watson 2016
Cognos Bi
Fidoop Gateway
Big SQL
Kafka
Core Banking System
Storm
Realtime
Processing
Datasources
R-Studio
HiveHBase
Spark
Jobs
…Java App
Ambari
Knox
Ranger
…
Hadoop (IOP)

15. The Solution
15 9/15/2016World of Watson 2016
Potential Use Cases
Master Worker Big Data
Plattform
Fraud
Detection
Usecase 2 Usecase 3

16. 3. The Lessons Learned
What a year with the elephant taught us

17. The Lessons Learned
17 9/15/2016World of Watson 2016
- one has to manage things like
hardware configuration, network
architecture, disksizes, security and
more
- getting even the development skills
can take much time (not to mention
the understanding of a distributed
system)
- there is a bunch of components to
get used to
Hadoop is complex

18. The Lessons Learned
18 9/15/2016World of Watson 2016
Support means
- vendor support
- external support
- (and maybe) internal support
Support is a
key to success

19. The Lessons Learned
19 9/15/2016World of Watson 2016
Even standard tasks can
generate big effort or cause
a deadlock
- the advantage of fast feature
availability comes with a price
- some features are theoretically
available but not enterprise ready
- Hadoop is not an “out-of-the-box”
tool

20. The Lessons Learned
20 9/15/2016World of Watson 2016
Open source within a
distribution comes
with a price
Advantages:
stability, component interoperability,
easy installation, support …
The price:
Seeing fixed issues to be available in
a project but not in your distribution
can be frustrating
Bugs and feature requests are
difficult to handle as there is a
distributor and a open source project

21. The Lessons Learned
21 9/15/2016World of Watson 2016
New technologies
require a change of
thinking
- a distribution of open source
projects isn’t a single vendor tool
- establishing a distributed platform
can require new processes or
procedures
- sometimes building up a new thing
can help to get rid of old junk

22. The Lessons Learned
22 9/15/2016World of Watson 2016
Costs:
hardware as for a cluster you have to
buy servers
software support as open source is
free but not “for free”
external support if you don’t have all
skills (and you’ll need a lot)
integration as a new platform has to
be integrated decently
Establish Hadoop
as a plattform generates
relevant inital costs

23. 4. The Blueprint
How to get the elephant started

24. The Blueprint
24 9/15/2016World of Watson 2016
Take a simple use case (if possible)Hadoop is complex

25. The Blueprint
25 9/15/2016World of Watson 2016
Use as few components as possible
Support is a
key to success
Hadoop is complex

26. The Blueprint
26 9/15/2016World of Watson 2016
In the beginning start with a security
that is as simple as possible
Hadoop is complex

27. The Blueprint
27 9/15/2016World of Watson 2016
Try to be agile in development as
building up a plattform will be
sophisticated
Even standard tasks can
generate big effort or cause
a deadlock

28. The Blueprint
28 9/15/2016World of Watson 2016
Be sure to have good management support
for budget decisions and escalations
Establish Hadoop
as a plattform generates
relevant inital costs
New technologies
require a change of
thinking

29. The Blueprint
29 9/15/2016World of Watson 2016
Concentrate on relevant parts and
avoid to much additional effort where
possible (buildtools etc)
Establish Hadoop
as a plattform generates
relevant inital costs

30. The Blueprint
30 9/15/2016World of Watson 2016
Calculate with training time and bugfixing
Even standard tasks can
generate big effort or cause
a deadlock
Hadoop is complex

31. The Blueprint

32. Thank You