At Bouygues Telecom, we rely heavily on our user data to relay the best possible services. We had been using a proprietary solution for our directory of customer services. This system is used by severals other teams, such as voicemail, customer identification, online services and the mobile marketplace, for online payments, OAuth and network authentication. We faced several problems with scalability and the frequent changes in our data model as services changed on a daily basis, and decided to build our own software to meet our needs. MongoDB's sharding and schema free document model made it a perfect fit for our in-house solution. The system is now designed to manage 12 million customers, 3000 request per second and availability of 99.9%. In six months we were able to build the new system that effectively connects the our various systems and partners, leading to a simpler and more streamlines system for access. In this talk we will discuss how we build this system from the ground up with MongoDB, Storm and Tomcat REST Services.

1. Adding Flexibility and
Scalability to
Bouygues Telecom
with MongoDB
Pierre-Alban DEWITTE
@__pad__
Bouygues Telecom
MongoDB World –
06/2x/2014

2. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Bouygues Telecom
13 M
Customers

3. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Who is speaking ?
@__pad__

4. Thank
s !

5. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Agenda
Adding Flexibility and
Scalability to Bouygues
Telecom with MongoDB
1. Existing problems
2. Design choices
3. Build
4. A few steps back

6. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Agenda
1. Existing problems

7. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Existing problems
Consum
er
Custom
ers
Existing
Sytem
Compag
ny
Custom
ers
XML
Files
XML
Files
Legacy
App
&
Protocol
LDAP
Direct
access
Micropaym
ent system
Identity and
Oauth
management
system
Network
Connexion
system
ETL

8. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Existing problems
Two years
delay
Schema
customization very
hard to design and
implement
Performance issues

9. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Existing problems

10. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Agenda
1. Existing problems
2. Design choices
A. MongoDB
B. Tomcat
C. Storm

11. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices -
MongoDB
Schema less data integration
High availability
Sharding
Less expensive data storage
allowing duplication

12. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices -
MongoDB
Team with development and
architecture skills
Schema design experience
Capability to train new actors

13. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
POC between Tomcat and
NodeJS
We choose our battle,
MongoDB !
Design choices - Tomcat

14. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld

15. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - Storm
Distributed real-time computation
system
It is scalable, fault-tolerant, guarantees
your data will be processed, and is
easy to set up and operate.
Suits for real-time data acquisition
treatment but also for batch

16. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - Storm
Consum
er
Custom
er
MongoDB
database
Compag
ny
Custom
ers
FilesFiles
Legacy
App
REST
Micropaym
ent system
Identity and
Oauth
management
system
Network
Connexion
system
API
STORM

17. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Agenda
1. Existing problems
2. Design choices
3. Build
A. Storm topology
B. Schema design and refactor
C. DevOps

18. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - Storm
Customer
files are
pushed
every night
Customer
files are
pushed
every night
Read file line
by line
Read file line
by line
1
… DetectFil
e
Spout
DetectFil
e
Spout
ProcessLi
ne
Bolt
ProcessLi
ne
Bolt
2
lineline

19. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - Storm

20. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - Storm
Emitting tuple for
each set of
functional
collections
Emitting tuple for
each set of
functional
collections
ProcessLi
ne
Bolt
ProcessLi
ne
Bolt
4
Oauth
UpdaterB
olt
Oauth
UpdaterB
olt
Identity
UpdaterB
olt
Identity
UpdaterB
olt
Transform a line
into a document &
check delta
Transform a line
into a document &
check delta
3

21. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - Storm

22. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - Storm

23. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - Storm
Message failure handle
ack(Object msgId)
failed(Object msgId)
Take care of system stability

24. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Design choices - Storm
We scaled and
easily !

25. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and
refactor
Conception User
- Name
- Compag
ny
- Billcycle
- Payment
Mode
Acces
Point- Name
- Compag
ny
- Billcycle
- Payment
1
1..
n
Conception

26. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and
refactor
Development

27. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and
refactor
MAJOR ISSUE 1 : « Access
points should be returned
with user while reading it »
Qualificati
on
MAJOR ISSUE 2 : « User
should be returned with
access point while reading
it »

28. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and
refactor
Development
« Wait next build ! »

29. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and
refactor
« That works »
Qualificati
on
Conception

30. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and
refactor
« User read response times are
over the limit and it is even
worse with access point ones»
Stress
Test

31. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and
refactor
ConceptionConception

32. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and
refactor
ConceptionConception

33. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and
refactor
Development

34. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - Schema design and
refactor
PERFECT !
Development
Qualificati
on
Conception
Stress
Test
We were prepared to fail
That makes rework
painless

35. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - DevOps
Our tools !

36. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - DevOps
Supervisio
n
Monitorin
g
Automatic
deployment
Our tools !

37. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Build - DevOps
Moving the line between Dev and Ops
Internal training by Dev
Redaction of an Ops PlayBook
Pre production tests played by both team
Early write of stress tests played during
acceptance tests

38. 6 months later

39. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
6 months later
3000 req/s
95% req < 100ms
500 000 customers
loaded in less than 2
hours

40. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
Agenda
1. Existing problems
2. Design choices
3. Iterations during build
4. A few steps back

41. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
A few steps back
Apache Tomcat Apache STORM
Flexibilty and
Scale with a 100%
Open Source

42. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
A few steps back
1. Choose your battle
2. Use the right tool for
the right job
3. Be prepare to fail and
to rework
4. To move organization,
training is a key

43. Thank you for
listening
Questions
?
Further talks @__pad__