1. Managing&Exadata&and&the&
Human&Factor+
Mar4n&Bach&
Enkitec&Europe&
+
h.p://www.enkitec.com+
h.p://mar4ncarstenbach.wordpress.com++
www.enkitec.com++
1+++

2. About+myself+
• Oracle+pla?orm+enthusiast++
•
•
•
•
Linux+
Engineered+Systems+
Storage+
Database+
• A+li.le+bit+of+background+
• Pro+Oracle+Database+11g+RAC+on+Linux+
(with+Steve+Shaw+and+Julian+Dyke)+
• Wordpress+blog+
+++h.p://mar4ncarstenbach.wordpress.com+
• @Mar4nDBA+on+Twi.er+
+
www.enkitec.com++
2+++

3. Who+We+Are+
• Oracle-centric Consulting Partner focusing on the Oracle
Technology Stack
• Exadata Specialized Partner status (one of a handful globally)
• 200+ Successful Exadata Implementations
• Dedicated, In-house Exadata Lab (POV, Patch Validation)
• All of our Consultants have 12+ years of Oracle Experience
• Additional practices include
-
-
-
-
Oracle RAC Implementation
Oracle Golden Gate
Oracle Applications Administration
On-Call Database Administration
www.enkitec.com++
3+++

4. What+you+should+expect+to+hear+in+this+session+
• Some+of+the+current+trends+in+IT+
+
• Possible+situa4on+before+the+arrival+of+the+Engineered+System+
• Who+should+take+responsibility+of+the+Engineered+System+
• What+to+do+closer+to+the+goUlive+and+rolling+out+
www.enkitec.com++
4+++

5. Prelude+and+some+context+
+
What+are+the+current+trends+
(not%that%I%would%necessarily%endorse%those)%
www.enkitec.com++
5+++

6. What+is+the+industry+discussing?+
• Companies+are+driven+by+requirement+to+make+money+
• Only+some+might+now+be+capable+of+intrinsic+growth+
• Usual+result:+cost+cu[ng+
• “Management+by+accoun4ng”+
• Reducing+the+number+of+staff+in+opera4onal+teams+
• Offshoring+
• And+everyone+is+trying+to+increase+efficiency+
•
•
•
•
•
•
Automa4on+
Standardisa4on+
SelfUservice+portals/private+and+public+clouds/DBaaS+
Virtualisa4on+(inside+the+database+of+course)+
…+we+all+know+these+
What’s+the+success+rate?+
www.enkitec.com++
6+++

7. What+this+poten4ally+leads+to+
• Reduc4on+of+workforce+in+higherUcost+countries+
• Staff+there+=+expensive+
• Reduced+numbers,+moving+up+to+level+2+and+3+support+
• Two+models+for+level+1+support+
• Many+tasks+can+be+replaced+by+scripts+
• Alterna4vely+move+L1+support+to+lowerUcost+areas+
• Huge+workloads+for+an+individual+on+site+in+higher+cost+
country+
• Stressful+atmosphere+
• DBAs+looking+ager+hundreds+of+databases+not+uncommon+
• “Know+your+data”+most+likely+impossible+
+
www.enkitec.com++
7+++

8. +
So+what+does+this+have+to+do+with+
this+presenta4on+or+Exadata?+
%
www.enkitec.com++
8+++

9. Exadata+implementa4on+goals+
• Common+Engineered+System+project+goals+are:+
• Cost+savings+
• Consolida4on+
• Hardware+refresh+
• Engineered+System+is+cheaper+than+compe4tor+etc.+
• High+performance++
• Collateral+benefits+
•
•
•
•
Common+pla?orm+
Be.er+visibility+of+performance+metrics+
More+control+over+the+stack+
…+
www.enkitec.com++
9+++

10. Can+this+be+done+
?+
www.enkitec.com++
10+++

11. +
Common+ini4al+situa4on+
(before%the%Engineered%System%arrived)%
www.enkitec.com++
11+++

12. www.enkitec.com++
…+
Database+opsUproduct+
Database+opsUglobal+
Storage+Engineering+
Database+Engineering+
One+problem+in+IT+
…+as+I+see+it+
12+++

13. And+another+one+
www.enkitec.com++
13+++

14. But+why?+
• Lots+of+factors+playing+in+
•
•
•
•
Historical+development+
Job+security+concerns+
Mergers+and+Acquisi4ons+
…+
• Engineered+Systems+
• Fill+a+gap+
Current+failures+in+delivering+suitable+and+sustainable+solu4ons+in+IT+helped+
towards+the+success+of+Engineered+Systems+
• Consolida4on!+
• Remove+old/subUop4mally+performing+hardware+
• Less+floor+space,+cooling,+power,+…+
www.enkitec.com++
14+++

15. One+of+the+biggest+problems+on+the+way:+Silos+
• Two+different+types+of+silos+
• ProductUrelated+
• TaskUrelated+
• ProductUrelated+silo+
• Product+managers+build+their+own+team+and+infrastructure+
• Which+is+not+necessarily+aligned+with+the+global+IT+strategy+
• Leading+to+higher+cost+for+training+and+maintenance+
• TaskUrelated+silo+
• Worse!+
• Leads+to+the+crea4on+of+the+“silo+DBA”+
• The+“silo+DBA”+is+a+specialist+in+his+field+
www.enkitec.com++
15+++

16. ProductUaligned+DBAs+
Product+A+
Product+B+
www.enkitec.com++
Product+C+
16+++

17. ProductUaligned+DBAs+become+one+team+
global+
Product+A+
Product+B+
www.enkitec.com++
Product+C+
17+++

18. Departments+can+also+form+their+own+silos+
Database+
Storage+
Network+
Unix+
Love?+
management+
Engineering+
Backup+
Database+
Storage+
Opera4ons+
Network+
UNIX+
Backup+
www.enkitec.com++
18+++

19. The+historical+dimension+
Core+network+
switches+
SMP+database+
server+
The+world+of+the+network+administrator+
&
&
Home+for+many+
• Database+administrator+
• System+administrator+
This+worked+well+
• Storage+administrator+*+
actually.+Then.+ Backup+*+
•
• IT+Security+*+
Fibre+Channel+
switch+
The+world+of+the+storage+administrator+
Fibre+Channel+
storage+array+
=+Many+teams+responsible+for+the+stack+
www.enkitec.com++
19+++

20. Exadata+uses+More+Modern+Technology+™+
Oracle+Real+Applica4on+Clusters+
• Up+to+8+smallUish+nodes+
• Or+2+rather+large+servers+
Oracle+Clusterware+
Oracle+Automa4c+Storage+Management+
Infiniband+switches+
• RAC+Interconnect+
• Storage+
…+and+yet+many+sites+decide+to+use+the+same+opera4onal++
“standards”+for+Exadata+as+for+their+other+systems+
www.enkitec.com++
20+++

21. Oracle+takes+over+more+and+more+of+the+stack+
Features&introduced&past&9i&
• Oracle+Clusterware+
• HAIP+resource+
• Automa4c+Storage+
Management+
• Cluster+LVM+for+data+files+
• “Cloud+File+System”+
• GNS+or+IP+Address+
Management+
As&opposed&to&the&9i&world&
• 3rdUparty+clusterware+for+
node+membership+
• NIC+aggrega4on+ogen+
requires+addi4onally+
licensed+sogware+
• Cluster+LVM+sogware+
usually+needs+addi4onal+
licenses+as+well+
This+causes+unease+outside+the+DBA+camp+
www.enkitec.com++
21+++

22. With+that+said+
•
•
•
•
Who+takes+ownership+of+an+Engineered+System?+
What+about+coordina4on+with+surrounding+infrastructure?+
And+what+if+you+are+migra4ng+off+another+system?+
Difficult+ques4on+to+answer!+
www.enkitec.com++
22+++

23. Thinking+about+ownership+of+the+stack+
+
What+can+be+considered?+
(poten<al%solu<ons)%
www.enkitec.com++
23+++

24. Time+spent+on+typical+administra4on+tasks+
Database+administra4on+
System+administra4on+
Storage+administra4on+
Network+administra4on+
www.enkitec.com++
24+++

25. Possible+Exadata+management+models+
• Commonly+used+approaches+
• Keep+silo+model+
• Use+the+AUTeam+
• Allow+DBAs++to+take+ownership+of+the+full+stack+
• Each+model+has+its+advantages+and+disadvantages+
• It+very+much+depends+on++
• Desired+efficiency+
• Corporate+structures,+procedures+and+hierarchies+
• How+well+teams+communicate+
www.enkitec.com++
25+++

26. Keeping+the+silo+model+
• Why+
•
•
•
•
“Separa4on+of+du4es+worked+before”+
“Why+shouldn’t+it+work+in+11.2/12.1?”+
“We+have+always+done+it+that+way”+
“The+standards+say+so”+
• Why+not?+
•
•
•
•
•
A+lot+of+coordina4on+between+teams+needed+
One+problem:+root+account+
My+experience+shows+storage+team+not+too+keen+on+ASM+
You+also+ought+to+separate+grid+owner+from+RDBMS+owner+
It+can+actually+be+frustra4ng+for+the+DBAs+
www.enkitec.com++
26+++

27. Ye+olde+silo+
• The+idea:+
• Train+exis4ng+SAN/network/UNIX+admin+in+the+use+of+Exadata+
• Keep+individual+responsibili4es+in+the+respec4ve+team+
• Cons+
• Pro+
• Least+intrusive+for+the+teams+
• Exis4ng+standards+don’t+need+
changing…much+
• Less+responsibility+for+the+
individual+
www.enkitec.com++
• Contradicts+the+idea+of+
“Engineered+System”+
• Exadata+parts+have+
dependencies+
• Risk+of+overUcustomising+the+
system+
• Might+delay+roll+out+(training,+
haggling+over+competence,+
documenta4on…)+
27+++

28. The+silo+model+can+work+
• Under+certain+circumstances+
•
•
•
•
Willingness+to+learn+and+embrace+the+Exadata+model+
(Primarily+addressed+to+nonUDBAs)+
Good+communica4on+between+teams+
Maintenance+windows+defined+well+in+advance+
• But+you+are+s4ll+giving+up+on+agility++
• Problem+solving+may+take+longer+
• Main+ques4ons+IMO+
• Who+patches?+
• Who+owns+the+root+account?+
www.enkitec.com++
28+++

29. Stepping+it+up+
• Now+what+if+the+silo+model+turned+out+to+be+less+op4mal?+
• Apart+from+consterna4on+
• And+swearing+that+Exadata+is+too+complex+
• Imagine+you’d+give+the+DBAs+more+control+
• Or+the+root+password+for+that+ma.er+
• The+DBAs+do+70U80%+of+daily+administra4on+anyway+
• Trust+them+
It’s+4me+for+a+special+team+
www.enkitec.com++
29+++

30. Introducing+the+AUTeam+(1)+
• (More)+Power+the+peop^H^H+DBAs+
• What+changes?+
• The+DBA+has+a+first+shot+at+the+problem+
• Normally+that’s+all+there+is+to+it.+
• If+the+DBAs+don’t+know,+get+dedicated+help+from+storage/sys+admin+
• Why+the+reluctance?+
• Nobody+worried+when+the+DBAs+had+to+look+ager+OEM!+
• It+has+the+Oracle+label+so+it’s+the+natural+habitat+of+the+DBA+;)+
• And+there+is+backup+if+needed+
www.enkitec.com++
30+++

31. The+AUTeam+
• The+idea:+
• Create+a+new+team+with+exis4ng++members+from+storage,+network,+
database+teams+
• Use+only+the+most+skillful+and+keenest+
• Pro+
• Cons+
• Could+be+on+a+projectUbasis,+
i.e.+non+full4me+ini4ally+
• Is+an+intermediate+step+most+
likely+
• Team+members+can+act+as+
knowledge+replicators+
• Conveys+warm+and+fuzzy+
feeling+
www.enkitec.com++
• S4ll+risk+that+Exadata+is+overU
engineered+or+too+heavily+
customised+
• Needs+enough+team+members+
for+a+24/7+support+
• Removes+capacity+from+nonU
Exadata+related+work+
31+++

32. The+ideal+Exadata+administrator+
• Final+evolu4on+stage+
• The+DBA+evolves+
• Arup+Nanda+coined+the+term+Database+Machine+Administrator+
• Broad+range+of+skills+desired+
• Exadata+truly+benefits+from+an+incarna4on++of+a+“DBA+2.0”+
•
•
•
•
•
•
•
Automa4c+Storage+Management+
Infiniband++
10Gbit+Ethernet+
Real+Applica4on+Clusters++
Disaster+Recovery,+Backup+
ZFS+appliance?+Exaly4cs?+
Not+to+forget+SQL+skills!+
www.enkitec.com++
32+++

33. The+Database+Machine+Admin+
• The+idea:+
• Allow+Oracle+database+administrators+to+evolve+
• Grant+them+the+appropriate+access+privileges+
• Pro+
• Cons+
• Build+on+exis4ng+experience+
• Single+group+manages+all+
Exadata+systems+
• Be.er+understanding+of+
impact+of+changes+
• Most+effec4ve+problem+solving+
www.enkitec.com++
• Could+be+more+expensive+
• Training+needs+
• Backfill+or+new+hires+to+replace+
those+in+the+new+team+
• Red+tape+
• Might+not+be+possible+due+to+
regulatory+requirements+
33+++

34. (Necessary?)+changes+with+Exadata+
• A+DBA+siloUenvironment+seems+a+subop4mal+approach+
• Pick+one+
•
•
•
•
Con4nue+as+it+is+…+(or+be.er+not)+
Create+an+AUTeam+
Allow+DBAs+to+progress+to+Database+Machine+Administrators+
See+Arup+Nanda’s+blog+and+presenta4ons+as+well+
• Need+to+move+carefully+
•
•
•
•
Fear+factor:+job+security+in+the+other+teams+
Don't+let+the+Red+Stack+be+perceived+as+enemy+
Access+to+the+holy+grail:+root+password+
Other+security+constraints+
www.enkitec.com++
34+++

35. No+one+gets+leg+behind+
• Well,+this+is+an+interes4ng+aspect+
• Evolving+to+an+Exadata+DBA+is+not+for+everyone+
• Ideally+you+are+hungry+to+learn+
• What+if+not?+
•
•
•
•
Exadata+rarely+replaces+every+Oracle+database+
So+there+you+go!++
Keep+the+DBAs+who+prefer+the+status+quo+on+those+
(but+feed+them+cool+stuff+so+they+get+interested+in+Exadata)+
www.enkitec.com++
35+++

36. Migra4on+aspects+
+
Migra4ng+off+a+different+pla?orm+
This+is+probably+the+most+challenging+aspect+
www.enkitec.com++
36+++

37. Why+is+a+migra4on+from+a+different+pla?orm+difficult?+
• Imagine+how+you’d+feel+
• If+someone+told+me+to+support+<arbitrary+compe44on+to+Oracle>+
• …+I+wouldn’t+be+too+enthusias4c!+
• The+migra4on+is+not+always+well+communicated+
• Project+team+ogen+external+
• Might+be+perceived+as+alien+
• So+it’s+even+more+important+to++
• Make+the+team+support+the+migra4on+
• Get+them+interested+into+the+migra4on+
• Involve+them+as+much+as+possible+
www.enkitec.com++
37+++

38. Post+“Proof+of+Value”+
+
Now+that+it’s+going+live+
Engineering+and+opera4onal+support+
www.enkitec.com++
38+++

39. What+does+usually+happen+
• The+value+of+the+Engineered+System+has+been+proven+
• Prototype+solu4ons+need+extending+
• Monitoring+of+the+Engineered+System+(OEM)+
• No4fica4ons+need+to+be+set+up+
• Staff+management+
• Those+who+take+care+of+Exadata+might+need+training+
Not+necessarily+
• A+lot+of+documenta4on+is+wri.en+
internal+staff!+
• The+project+team+will+need+to+pass+its+experience+on+
• Procedures+
• Procedures+need+upda4ng+if+they+are+relevant+to+Exadata+
• Test,+valida4on,+and+more+tes4ng+
• But+don’t+delay+for+too+long!+
www.enkitec.com++
39+++

40. Rollout+
• The+Avant>Garde%
• Engineering+leading+the+efforts+towards+a+rollout+
• Opera4onal+DBAs+not+trained;+con4nuing+as+operators+
• Pro+
• Cons+
• Assuming+that+Engineering+is+
really+competent+
• Centralised+team+ac4ng+as+
competence+center+
• Complexity+hiding+via+scripts+
• Standardisa4on+of+procedures+
www.enkitec.com++
• Team+must+be+able+to+run+with+
problem+un4l+resolu4on,+but:+
• Engineering+ogen+
understaffed+
• Might+not+meet+the+
requirements+of+the+
opera4onal+team+
• Large+effort+
40+++

41. Rollout+
• Coordina4on+and+CoUopera4on%
• All+teams+perform+their+specific+tasks+towards+the+rollout+
• Everyone+is+empowered,+informa4on+is+shared+between+all+involved+
• Pro+
• Cons+
• More+involvement+from+the+
team+
• The+effort+is+distributed+on+all+
shoulders+
• Poten4al+to+get+more+work+
done+in+less+4me+
www.enkitec.com++
• Exis4ng+skillset+makes+this+
difficult+
• Resistance+to+change+from+
nonUDBA+departments+
41+++

42. Poten4al+difficul4es+
Try+to+write+a+script+for+
• Even+in+Engineering+there+are+many+subUdivisions+ deploying+bundle+
• Risk+of+overUengineering+a+solu4on+
• FireUandUforget+scripts+are+difficult+to+write+
patches+!+
• Dependent+systems+can’t+be+upgraded+in+4me+
• How+many+of+you+are+on+Enterprise+Manager+<+12c+
• Or+the+Trouble+Ticket+System+can’t+be+integrated+
• Backup,+recovery,+disaster+recovery+
• Budget+constraints+
• The+security+team+
• Human+factors+
www.enkitec.com++
42+++

43. You+might+want+to+avoid+these+
• OverUengineering+
• Complexity+is+the+enemy+of+availability+
• Some4mes+it+is+be.er+to+follow+Oracle’s+recommenda4on+than+
corporate+standards+
• Do+not+try+these+at+home:+
•
•
•
•
Or+be.er:+review+and+“know+what+you+are+doing”+
Subdivide+half+rack+into+2+separate+clusters+(including+Clusterware!)+
Try+to+fit+42+applica4ons+into+single,+consolidated+database+
Create+an+Oracle+home+per+consolidated+applica4on+
www.enkitec.com++
43+++

44. I+like+it+when+a+plan+comes+together+
• Don’t+forget+the+master+plan!+
•
•
•
•
•
•
Keep+it+simple+
Runbook+for+operators?+
Monitoring?+
Rotas+for+outUofUhours+and+weekends?+
Migra4ng+to+Exadata:+tried+and+tested?+
Do+you+have+a+curious+IT+security+team?+
• NonUtangible+
• Maintain+a+“canUdo”+a[tude+
• Don’t+get+bogged+down+in+too+much+detail+
• Include+the+applica4on+stake+holders+and+bring+them+in+line+with+the+
overall+strategy+
• Challenge+“we+can't+do+it+this+way+we+always+did+it+like+x.”+
www.enkitec.com++
44+++

45. IT&ALL&DEPENDS&of&course&
&
Just&because&it’s&wriHen&&
doesn’t&mean&it&is&true&
www.enkitec.com++
45+++

46. Time+for+ques4ons+and+comments!+
www.enkitec.com++
46+++