Capgemini - Project industrialization with apache spark

Apache Spark and Bluemix Meetup
Jean-Baptiste Martin
July 6, 2016
Project industrialization with Apache Spark

2Copyright © Capgemini 2015. All Rights Reserved
Who am I
 Jean-Baptiste Martin
 Managing Consultant at Capgemini
 Background: technical
 Big Data Analytics for 2 years
 Product manager People Analytics
 Founder at Top Notch

Project industrialization with Apache Spark
1. Spark in People Analytics
2. Team Organization
3. Issue #1: Text Replace
4. Issue #2: Non-Serializable Objects
5. Issue #3: Unit Testing
6. Issue #4: Wall of Code
Code available at:
https://github.com/jeanbmar/meetup-spark

Spark in People Analytics
 What is People Analytics?

Spark in People Analytics
Unstructured
WEX
AppBuilder
Watson Explorer
WEX Engine
Data Indexing
Visualization
HDFS
Store
Analytics Engine
Data Reconciliation
ODPi
HDFS Access
Structured
SGBD
CSV Files
Employees
Candidates
Jobs
1
2
3
4

Team Organization
1. Prototyping:
• Technologies: Hadoop, Java, R, Watson Explorer
• Team Profiles: 4 big data dev (Java), 1 data scientist, 1 data analyst
2. Industrialization:
• Technologies: Hadoop, Java and Scala, Spark, Watson Explorer
• Team Profiles:
– 2 data scientists
– 2 software developers
– 1 sys admin
– 2 web developers
3. All along:
• Strong support from IBM (expertise, implementation, go-to-market)

Issues we faced
 Issue #1: Text Replace
 Issue #2: Non-Serializable Objects
 Issue #3: Unit Testing
 Issue #4: Wall of Code

Issue #1 : Text Replace
 Browse and replace text is common when performing natural language
processing
« I work with WEX at Cap Gemini »
« I work with Watson Explorer at Capgemini »
Cap Gemini Capgemini
WEX Watson Explorer
+
=

 Issues when:
• There’s a lot of documents to process
• Dictionaries (synonyms, stopwords, protected words, …) contain 1000+ entries
 Traditional implementations:
• Loop over dictionary entries  LOW PERF AND/OR INCORRECT
• Regular Expressions  LOW PERF
 We want: read text 1x and perform transformations on the fly

 Solution
1. Expand dictionaries in HashMap objects, e.g.
2. Read text character by character and perform lookups over HashMap objects
– X  combination of characters is a part of an existing word
– null  no match
– Other  match
W X
WE X
WEX Watson Explorer

 Case 1:
• Have: “Engineer. English. Fluent en.”
• Want: “Engineer. English. Fluent english.”
 Case 2:
• Have: “Cap Gemini consultant and Big Data developer with strong xp on Hadoop,
mostly Hadoop FS. BI background (DataStage, Cognos, Oracle, DB2). Worked on
multiple Watson technologies, including Watson API and WEX.”
• Dictionary, 875 entries including:
Cap Gemini Capgemini
Hadoop FS HDFS
DataStage IBM DataStage
Cognos IBM Cognos
DB2 IBM DB2
WEX Watson Explorer

Issue #2: Non-Serializable Objects
 Sometimes, people need to use external libraries to perform specific
transformations on objects
 Example: perform NLP transformations with Apache OpenNLP
 Problem:
• OpenNLP objects are not serializable  No broadcast
• OpenNLP objects take time to initialize  Never-ending closures
• We don’t want to convert OpenNLP source code (actually we tried)

 Solution: Initiliaze singletons and bind them to Spark tasks using Java
ThreadLocal
Singleton class
Bind singleton to task thread
Will be called in closure

 Then call transformation in closures:
 Benefits: objects are initialized only 1x per task instead of 1x per RDD element
Retrieve holder from current task
Get singleton object
Get SimpleClass object

Issue #3: Unit Testing
 One major step when moving from prototype to production is to define a proper
testing strategy
 Way people do their tests (non-exhaustive):
1. They run everything on cluster o/
2. They use a local context
 What we did:
• Use a local context
 Problem: jobs grab content from HDFS using Oozie job.properties
 Solution: setup a flexible configuration to operate seamlessly on cluster and locally

 How it looks like:
Class applying a set of transformations
This grabs files on HDFS, can’t use locally

 How can seamlessly operate with remote or local job.properties?
Using this
ConfigHelper class

Call conf
Grab on FS

 Finally, our test:

Issue #4: Wall of Code
 Object-oriented programming modeling doesn’t apply well in Spark
 As a result, we tend to write huge functions with tons of transformations
 People Analytics V0.01alpha : 1 class
 How we managed this:
 We regrouped consistant sets of transformations into functional classes
Functional class
Class consecutive
operations in run method

Thank You
Credits:
jean-baptiste.martin@capgemini.com
jerome.delvigne@capgemini.com
Code available at:
https://github.com/jeanbmar/meetup-spark

The information contained in this presentation is proprietary.
Copyright © 2015 Capgemini. All rights reserved.
Rightshore® is a trademark belonging to Capgemini.
www.capgemini.com
About Capgemini
With 180,000 people in over 40 countries, Capgemini is one of
the world's foremost providers of consulting, technology and
outsourcing services. The Group reported 2014 global revenues
of EUR 10.573 billion.
Together with its clients, Capgemini creates and delivers
business, technology and digital solutions that fit their needs,
enabling them to achieve innovation and competitiveness. A
deeply multicultural organization, Capgemini has developed its
own way of working, the Collaborative Business Experience™,
and draws on Rightshore®, its worldwide delivery model.
Learn more about us at www.capgemini.com.

Capgemini - Project industrialization with apache spark

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Capgemini - Project industrialization with apache spark

Ähnlich wie Capgemini - Project industrialization with apache spark (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Capgemini - Project industrialization with apache spark