Realtime OLTP on Conventional Oracle Database: 4 Years of Tušmobil Experience

1. !
Real&me!OLTP!on!Conven&onal!Oracle!
Database:!
4!Years!of!Tušmobil!Experience!
!
www.tusmobil.si!
!
!
Ognjen!Antonič,!Tušmobil!
ognjen.antonic@tusmobil.si!
hIp://www.tusmobil.si!

2. About!Tušmobil!
!
www.tusmobil.si! 2!
L !3rd!mobile!operator!on!Slovene!market,!holding!
2G!and!3G!licenses!
L !Launched!live!in!Q4/2007!!
L !More!than!200.000!ac&ve!mobile!subscribers!
as!of!September!2011!(more!than!300.000!
provisioned!on!the!network!and!billing!system)!
!
!
!
!
!
!
!

3. The!need!for!realL&me!convergent!billing!
system!
www.tusmobil.si! 3!
L !Addressing!both!Prepaid!and!Postpaid!
market!segments!!
L prepaid!entails!realL&me!processing!and!
charging!
L !Voice!&!videoLtelephony,!CSD!data!&!fax,!
short!messages!(SMS),!mul&media!messages!
(MMS),!GPRS!/!3G!data,!content,!…,!
whatever!future!service!
L !From!consumers!to!business,!from!retail!to!
wholesale!
!
!

4. Tradi&onal!Way!
!
www.tusmobil.si! 4!
L !Separated!postpaid!and!prepaid!billing!/!
charging!systems!
L !Some&mes!even!separate!per!service!/!
technology!billing!/!charging!systems!
L !Prepaid!services!tradi&onally!developed!in!
the!IN!(intelligent!network)!as!a!part!of!
network!domain!
L !Postpaid!services!resided!in!an!offline!
billing!system!entailing!separate!charging!
data!records!and!was!part!of!the!IT!domain!
!
!

5. Tušmobil!went!real&me!covergent!with!
Oracle!Billing!&!Revenue!Management!
www.tusmobil.si! 5!
L !Implemented!Oracle!BRM!7.3!in!2007!
L !Digital!Route!Media&on!Zone!as!real&me!
Media&on!Solu&on!for!VAS!services!
L !RealL&me!voice,!data!(GPRS),!SMS,!MMS!&!
content!authoriza&on,!charging,!billing!
L !account!management,!servce!management,!
provisioning!for!for!prepaid!&!postpaid!users!
!
!
!
!
! Oracle!BRM!
Prepaid!
Postpaid!

6. Extended!core!BRM!with!inLhouse!
solu&ons!
www.tusmobil.si! 6!
L !RealL&me!Online!Credit!Control!and!Fraud!
preven&on!for!postpaid!(alarming!users!and!
limi&ng!usage!when!thresholds!are!reached)!
L !RealL&me!Policy!Control!Func&on!(limi&ng!
network!data!speed!ager!user!consumes!
allocated!quota!preven&ng!network!
conges&ons)!
!
!
!
!
! Oracle!BRM!
Credit!
Control!
Policy!
Control!

7. Conven&onal!!telecom!way!…!
www.tusmobil.si!
Network!
Element!
Offline!
Media&on!
Batch!
Ra&ng!
Database!
L !Call!/!event!data!is!being!read!from!CDR/
EDR!files!ager!calls!/!events!are!already!
finished!
L !Event!data!is!being!processed,!rated!and!
charged!
L !Simple!basic!logic!and!event!/!data!flow!
!
!
!
!
!
7!

8. …!versus!realL&me!way!
www.tusmobil.si! 8!
L !Event!/!calls!are!being!authorized,!
reauthorized!and!then!rated!
L !Data!is!being!processed!when!event!/!call!
is!being!aIempted!and!/!or!s&ll!being!in!
progress!
!
!
!
!
! Authorize!
ReL
Authorize!
Charge!

9. RealL&me!is!all!about:!Latency!
www.tusmobil.si! 9!

10. Latency!/!Response!Time!
www.tusmobil.si! 10!
L !!Time!taken!to!Authorise/Reauthorise/
Charge!an!event!in!telco!network!
L !Target!value!is!300!ms!or!less!for!
Authorize/Reauthorize,!!
L !1000!ms!for!Charge!(ager!call/event!is!
finished!by!the!user)!
!
!
!
!
!
Authorize!
ReL
Authorize!
Charge!

11. Real&me!charging!at!Tušmobil!
!
www.tusmobil.si! 11!
!
!
!
!
!
!
Oracle!
BRM!
AAA!GW!
RealL&me!
Media&on!(MZ)!
SMSC! MMSC! GGSN!
HP!
OCSAC!
SCP!
2G/3G!
network
s!
2G/3G!networks!
SS7!/!
CAP2!
IPv4!/!
MBI!
IPv4!/!
Diameter!
CONTENT!
provider!
Service!&!
Billing!GW!
IPv4!/!
Diameter!

12. Database!is!the!final!back!end!
www.tusmobil.si! 12!
!
!
!
!
!
!
2G/3G!
network
s! Oracle!
BRM!
!
Middleware!
RealL&me!
Media&on!(MZ)!
SMSC! MMSC! GGSN!
HP!
OCSAC!
SCP!
2G/3G!networks!
Oracle!RAC!
instance!
Deep!
Oracle!RAC!
instance!
Thought!
BRM!
database!
Answer!

13. DB!Back!End!@!Tušmobil!
www.tusmobil.si! 13!
!
!
!
!
!
!
!
!
L !Oracle!10.2!Enterprise!Edi&on!Real!Applica&on!
Clusters!Database!
L !Red!Hat!Enterprise!Linux!
L !2!node!Intel!Xeon!based!server!cluster!
L !more!than!5!TB!user!data!!
L !Live!since!July!2007!

14. Database!Servers!
!
www.tusmobil.si! 14!
2!x!!IBM!x3650!
!2!x!Intel!Xeon!Dual!Core!3.16!GHz!
!48!GB!memory!
!Max!SGA!Size!set!to!24!GB!
!Automa&c!Shared!Memory!Management!
!Red!Hat!Enterprise!Linux!4!(Nahant!Update!8)!
!mul&ple!Gbit!ETH!and!4Gb!FC!
!Automa&c!Storage!Management!used!
!
!
!
!
!

15. Storage!used!
!
www.tusmobil.si! 15!
1!x!!HP!XP!24000!
!ASM!managed!dedicated!disks!
!64!GB!cache!
!4!gbit!FC!bandwidth!
!(otherwise!shared!with!other!company’s!apps)!
!
!
!
!
!

16. Integrated!Database!Concept!!
www.tusmobil.si! 16!
L Single!database!for!billing,!media&on,!
middleware,!repor&ng/DWH!
L Reduced!&me!&!expenses!on!database!
administra&on!
L Data!centraliza&on!allows!for!reduced!
CAPEX!and!OPEX!on!server!infrastructure!
L More!Centralized!security!and!audi&ng!of!
users!
L Gerng!rid!off!data!integra&on!pains!(just!
run!a!query!running!against!both!billing!
and!network!data!in!same!database)!
!
!
!
!

17. Applica&ons!Running!on!top!of!DB!
!
www.tusmobil.si! 17!
L !Oracle!Communica&ons!BRM!!
L !99%!OLTP!with!RealL&me!Requirements!
L !Small!amount!of!batch!loads!&!scripts!(like!
monthly!billLrun)!
L !Middleware!(JBoss)!
L !Classical!OLTP!
L !Media&on!(Media&on!Zone)!
L !95%!batch!loads!(impor&ng!CDR!data)!
L !5%!OLTP!&!Real&me!loads!
L !DWH!&!Repor&ng!Applica&ons!
L !Classical!DWH/repor&ng!workloads!
!
!
!

18. Applica&on!Figures!
www.tusmobil.si! 18!
L BRM!
L More!than!50!M!charged!events!per!
month!
L More!than!250!M!records!per!month!
inserted!/!wriIen!into!various!BRM!
tables!
L More!than!99%!of!them!processed!
realL&me!as!they!happened!on!the!
network!
!
!
!
!
!
!
!

19. Figures!Con&nued!!...!
www.tusmobil.si! 19!
L Media&on!
L More!than!300!M!events!processed!
and!loaded!into!database!monthy!
L DWH/repor&ng!
L More!than!200!M!records!ending!up!
in!data!mart!tables!in!various!deL
normalized!and!aggregated!forms!
!
!
!
!
!
!
!

20. RAC!Logical!Par&&oning!
www.tusmobil.si! 20!
Ac&ve/Standby!usage!of!nodes!(from!apps!
perspec&ve)!
Instance!Deep!(primary!realL&me!/!online!
workloads)!
BRM!Primary!
Middleware!Primary!
Media&on!Secondary!
Repor&ng!&!DWH!Secondary!
Instance!Thought!(primary!batch!/!offline!
workloads)!
Media&on!Primary!
Repor&ng!&!DWH!Primary!
BRM!Secondary!
Middleware!Secondary!
!
!

21. Why!RAC!Logical!Par&&oning!
!
www.tusmobil.si! 21!
L Ac&ve/Ac&ve!not!supported!by!BRM!
(RAC!HA!only)!
L Distribu&ng!workloads!between!nodes!
evenly!and!reuse!of!their!capacity!
L Avoiding!DB!Cache!used!for!realL&me!
online!applica&ons!being!washed!out!by!
batch!offline!workloads!
L In!case!of!one!node!failure,!batch/offline!
usage!can!be!degraded!to!minimum!
needed!for!ongoing!opera&ons!keeping!
realL&me/online!systems!running!
!
!
!

22. Some!Mesurements!Taken!
www.tusmobil.si! 22!
L 1st!of!September!2011!
L Back!to!School!Day!L!with!busy!mobile!
subscribers!
L BRM!Monthly!Bill!run!started!for!
processing!over!500k!BRM!accounts!
on!the!eve!from!1st!to!2nd!of!
September!
!
!
!
!
!
!
!

23. Some!metrics!!
www.tusmobil.si! 23!
AVG_LATENCY !BRM!Authorize/Reauthorize/Charge!Average!Latency!ms!
AAA_AVGOPSS! !BRM!Average!Authorize/Reauthorize/Charge!Opera&ons!per!
! !second!
AVG_CPUPCT !Database!Average!CPU!Load!%!
AVG_BUFF_HR! !Database!Average!Buffer!Cache!Hit!Rate!Ra&o!%!
0!
20!
40!
60!
80!
100!
120!
140!
160!
2011.09.01!16!
2011.09.01!17!
2011.09.01!18!
2011.09.01!19!
2011.09.01!20!
2011.09.01!21!
2011.09.01!22!
2011.09.01!23!
2011.09.02!00!
2011.09.02!01!
2011.09.02!02!
2011.09.02!03!
2011.09.02!04!
2011.09.02!05!
2011.09.02!06!
2011.09.02!07!
2011.09.02!08!
2011.09.02!09!
2011.09.02!10!
2011.09.02!11!
2011.09.02!12!
2011.09.02!13!
2011.09.02!14!
2011.09.02!15!
AVG_LATENCY!
AAA_AVGOPS!
AVG_CPUPCT!
AVG_BUFF_HR!

24. Some!metrics!!
www.tusmobil.si! 24!
AVG_UTPS ! !Database!Average!User!Transac&ons!per!Second!
AAA_AVGOPSS! !BRM!Average!Authorize/Reauthorize/Charge!Opera&ons!per!
! !second!
AVG_DBCPS_MB !Database!Average!DB!Block!Changes!per!Second!in!MBytes!
AVG_DB_CPUTIME_RATIO!!Database!Average!CPU!Time!Ra&o!%!(rest!is!Wait!Time)!
0!
20!
40!
60!
80!
100!
120!
140!
160!
180!
200!
2011.09.01!16!
2011.09.01!17!
2011.09.01!18!
2011.09.01!19!
2011.09.01!20!
2011.09.01!21!
2011.09.01!22!
2011.09.01!23!
2011.09.02!00!
2011.09.02!01!
2011.09.02!02!
2011.09.02!03!
2011.09.02!04!
2011.09.02!05!
2011.09.02!06!
2011.09.02!07!
2011.09.02!08!
2011.09.02!09!
2011.09.02!10!
2011.09.02!11!
2011.09.02!12!
2011.09.02!13!
2011.09.02!14!
2011.09.02!15!
AVG_UTPS!
AAA_AVGOPS!
AVG_DBCPS_MB!
AVG_DB_CPUTIME_RATIO!

25. Some!metrics!!
www.tusmobil.si! 25!
BRM_CACHED_GB !Average!BRM!Data!Objects!Cached!inside!DB!Cache!–!GBytes!
AVG_LATENCY !BRM!Authorize/Reauthorize/Charge!Average!Latency!ms!
AAA_AVGOPSS! !BRM!Average!Authorize/Reauthorize/Charge!Opera&ons!per!
! !second!
AVG_BUFF_HR !Database!Average!Buffer!Cache!Hit!Ra&o!%!
AVG_DB_CPUTIME_RATIO!!Database!Average!CPU!Time!Ra&o!%!(rest!is!Wait!Time)!
0,0!
20,0!
40,0!
60,0!
80,0!
100,0!
120,0!
140,0!
160,0!
2011.09.01!16!
2011.09.01!17!
2011.09.01!18!
2011.09.01!19!
2011.09.01!20!
2011.09.01!21!
2011.09.01!22!
2011.09.01!23!
2011.09.02!00!
2011.09.02!01!
2011.09.02!02!
2011.09.02!03!
2011.09.02!04!
2011.09.02!05!
2011.09.02!06!
2011.09.02!07!
2011.09.02!08!
2011.09.02!09!
2011.09.02!10!
2011.09.02!11!
2011.09.02!12!
2011.09.02!13!
2011.09.02!14!
2011.09.02!15!
BRM_CACHED_GB!
AVG_LATENCY!
AAA_AVGOPS!
AVG_BUFF_HR!
AVG_DB_CPUTIME_RATIO!

26. Keeping!Up!RealLTime!Preformance!
www.tusmobil.si! 26!
L Do!not!be!cheap!on!memory!!!
L Plenty!of!db!cache!reduces!SQL!response!
&mes!significantly!as!it!keeps!buffer!
cache!hit!ra&os!high!
L More!memory!for!DB!Cache!usually!
yields!much!beIer!results!and!delays!
requirement!for!faster!disks!in!real&me!
OLTP!systems!
L We!Started!with!4!GB!per!node,!learned!the!
hard!way!that!48!GB!of!memory!for!Intel!
Xeon!is!cheap!compared!with!revenue!
losses!due!to!degraded!opera&on!
!
!
!

27. Op&mize!you!Applica&ons!
www.tusmobil.si! 27!
L Op&mize!your!apps!for!rela&onal!
database!access!
L Bad!SQL!code!
L Missing!primary/foreign!keys,!missing!
indexes!
L Some&mes!even!Oracle!does!not!
know!Oracle!(developers!like!to!live!in!
object!oriented!world,!while!their!
data!lives!in!rela&onal!world)!"!
L Out!of!the!box!BRM!was!missing!many!
indexes!and!even!primary!keys!
!
!
!
!

28. Take!Care!of!your!network!infrastructure!
www.tusmobil.si! 28!
L Do!not!be!cheap!on!your!network!
L Gbit!ETH!network!infrastructure!is!
cheap!
L Use!bonding!of!mul&ple!Gbit!ETH!
ports!for!increased!throughput!and!
response!&mes!
L Started!!with!“smart”!Network!admins!
connec&ng!our!Gbit!ETH!IC!ports!to!100!
Mbps!switch!ports!resul&ng!in!reduced!
response!&mes!"!
!
!
!
!
!

29. Those!damn!DBMS!Stats!...!
www.tusmobil.si! 29!
L Beware!of!SQL!execu&on!plan!devia&ons!
against!par&&oned!tables!with!
inadequate!sta&s&cs,!when!new!
par&&ons!are!being!“opened”!
L For!classical!&me!par&&oned!tables,!just!
in!advance!copying!of!gathered!sta&s&cs!
from!old!to!new!empty!table!par&&ons!is!
good!enough!to!save!you!from!
headaches!
!
!
!
!
!
!
!

30. And!there!are!strange!bugs!
www.tusmobil.si! 30!
L Beware!of!strange!applica&on!bugs!
L Like!applica&on!making!new!table!
entry!(INSERT!…)!instead!!of!making!
an!UPDATE!of!one!exis&ng!record,!
suffoca&ng!itself!with!its!own!data!
L If!some!things!do!not!make!sense,!they!
usually!do!not!make!sense,!so!try!digging!
deeper!into!you!apps!
!
!
!
!
!
!

31. Coexistence!of!real&meLOLTP!and!DWH!
www.tusmobil.si! 31!
L RealL&me!OLTP!apps!and!DWH!/!batch!
apps!can!coexist!on!same!RAC!database!
L Use!logical!RAC!node!par&&oning!for!max!
performance!
L Do!not!overuse!indexes!for!BI,!use!
par&&oned!compressed!tables!and!full!
table!scans!(direct!path!reads)!to!avoid!
you!database!cache!being!washed!away!
L BoIleneck!can!become!shared!disks!and!
RAC!Interconnect!!
!
!
!
!
!

32. www.tusmobil.si! 32!
Ognjen!Antonič!
Tušmobil,!IT!department!
ELmail:!ognjen.antonic@tusmobil.si!
hIp://si.linkedin.com/in/oantonic/!
!
!
!
!
!

33. www.tusmobil.si!
! 33!