VMworld 2013 Sheldon Brown, SRP Girish Manmadkar, VMware Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare

1. How SRP Delivers More Than Power to
Their Customers
Sheldon Brown, SRP
Girish Manmadkar, VMware
VAPP5105
#VAPP5105

2. 2
Agenda
 Introduction
 Before We Start - Homework
 Figuring It All Out
 What It All Looks Like Now
 After the Dust Settled
 Summary
 Questions

3. 3
Introduction

4. 4
Girish Manmadkar – Consulting Architect
With VMware since 2007 in various roles, focusing on providing
VMware solutions around Business Critical Application with major
focus around SAP
More than 22+ years experience starting with SAP R4.6
Regular speaker at VMworld since 2007
Recent VMware Partner Exchange, EMCWorld events

5. 5
Sheldon Brown – Manager SAP Technology
 Responsible for SAP Technology (BASIS,Development/Integration)
 21 years with SRP
 10 years with Server and Storage Team
 2 years with SAP implementation team
 9 of 10 years at VMWorld
 sheldon.brown@srpnet.com

6. 6
Who We Are – Salt River Project Agricultural and Power District (SRP)
 Power and Water supplier to the 1,000,000+ customers in the
Phoenix metro area
 Third largest public power company in the country
 100+ years old
 5000+ employees
 Slow moving, risk adverse
 70% virtualized
 http://www.srpnet.com

7. 7
Home Work
VMware Best Practices

8. 8
Best Practices
 Use the latest processor generations, especially with EPT (Intel
XEON 55xx, 65xx, 75xx and newer) or RVI (AMD Opteron 83xx,
84xx, 61xx and newer) which bring performance benefits
 Assigning vCPUs to virtual machines
• ESXi 5.1: shows scaling for virtual machines > 8 vCPUs
• Understand the v/NUMA architecture for larger VMs
 You can overcommit CPUs, but reasonably
• Start with no CPU overcommitment. When the average utilization of SAP
instances and the average idle time on the host are known, accordingly raise
the number of vCPUs on the host (for example, adding virtual machines). Use
VMware vSphere Distributed Resource Scheduler (DRS) capabilities to control
and balance overall workload.
• If performance problems are seen, first reserve CPU to see if the performance
problems are a result of resource contention

9. 9
Best Practices (cont.)
• Do not overcommit memory. This is not recommended at all with SAP.
Reserve 100% of the assigned memory.
• Installation of VMware Tools is mandatory to avoid time conflicts and have all
necessary drivers (for example, new network cards)
• Timekeeping: Use NTP (Network Time Protocol) on the ESX host and guest
as per Timekeeping best practices for Windows, including NTP and
SAP Note 989963
http://kb.vmware.com/kb/1318
• Use multiple virtual SCSI controllers for the database virtual machine and
virtual machines with high I/O load. The use of multiple virtual SCSI controllers
allows the execution of several parallel I/O operations inside the guest OS.
• SAP requires the implementation of the new virtualization aware monitoring.
See SAP Note 1409604.
• When using VMware snapshots, follow Best practices for virtual machine
snapshots in the VMware environment. Make sure that the virtual machine has
no active snapshot before reporting performance problems.
http://kb.vmware.com/kb/1025279

10. 10
Best Practices for Linux
 Disabling the Linux I/O scheduler
• Set kernel parameter "elevator=noop"
• Red Hat KB 5428 (applies to RHEL and SLES)
 Choose the optimal SAP memory model
• MAP memory model (SAP Note 386605) for CPUs without EPT/RVI
• STD memory model (SAP Note 941735) for CPUs with EPT/RVI

11. 11
vSphere – ESX CPU Scheduling
 Determine size of hardware NUMA nodes on the server
 Size the virtual machine as small as possible
• Too many vCPUs creates scheduling overhead
• Too much RAM results in accessing RAM from a far NUMA node
 Enable simultaneous multithreading
• More hardware execution contexts are available
 Generally CPU affinity not required

12. 12
Best Practices for Databases
 Oracle
• SAP Note 1173954 states support with minor restrictions
• Oracle on VMware vSphere Essential Database Deployment Tips
http://www.vmware.com/resources/techresources/10101
• VMware Alliances statement
http://www.vmware.com/solutions/partners/alliances/oracle-vmware-
support.html

13. 13
Best Practices for Databases – I/O Subsystem
 Store data files and log files on separate physical devices and distribute
data files as per SAN Volume Controller as defined by the storage
vendor’s best practices
 Quote from SAP on SQL Server best practices guide
• “The number of disks is determined by the throughput in I/O operations per
seconds (IOPS). Therefore discussions around storage investments or
configurations should be driven by IOPS”
• http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/4ab89e84-0d01-0010-
cda2-82ddc3548c65
 IOPS used by storage and OEM vendors to size SAP storage on
VMware – Similar to physical
 The standard SAP rules for databases apply!
• SAP Note 806554 for general I/O intensive DB tasks
• SAP Note 592393 and 793113 for Oracle

14. 15
Best Practices for vStorage APIs (cont.)
 VMware vSphere® Storage APIs – Array Integration (VAAI)
• Has nothing to do with backup – it is a better way of integrating storage
capabilities into vSphere
• vStorage APIs for Array Integration FAQ
http://kb.vmware.com/kb/1021976
• Follows the vSphere lifecycle
• Offloads specific storage operations to compliant storage hardware
• Write Same/Zero
• Eliminating redundant and repetitive write commands
• Fast/Full Copy
• Leveraging array ability to mass copy, snapshot, and move blocks
• Hardware Offloaded Locking (ATS)
• Stop locking LUNs and start locking only blocks

15. 16
Best Practices for Network
 Standard vSphere networking guidelines apply
• Separate infrastructure traffic from virtual machine traffic: use physical NICs or VLANs
• Use NIC teaming for availability and load balancing (group physical NICs connected to
the same physical network)
• Take advantage of network I/O controls to converge network and storage I/O onto
10GbE
• Take advantage of the dvSwitch and/or Cisco Nexus 1000v to simplfy the architecture
and support
 Use VMXNET3 adapter in guest OS
• Improved performance
• Helps with network traffic in three-tier setup between application and DB VM
• SAP Batch Job Performance on vSphere
http://blogs.vmware.com/performance/2010/02/sap-batch-job-performance-on-
vsphere.html

16. 17
Best Practices – Further Information
 SDN Forum SAP on VMware
• http://forums.sdn.sap.com/forum.jspa?forumID=471
• Most complete link collection around SAP on VMware
 SAP Notes
• Following the SAP Notes is mandatory!
 VMworld Sessions
• Various documents – from high level overview to technical deep dive
 Performance Tests
• Continuous performance measurement for databases and SAP software

17. 18
Figuring It All Out

18. 19
Figuring It All Out – SAP Architecture and Components

19. 20
Figuring It All Out – the Team
 Team Effort
 Consultants
• Accenture
• SAP
• HP
• VMware
• NetApp
• Cisco
 SRP Employees
• Server
• Storage
• Network
• BASIS
• Strategy and Planning
• Executive Management

20. 21
Figuring It All Out – What we needed to decide
 Datacenters
 Host Servers
 Virtual Servers
 Storage
 Network
 Database
 Operating System
 Infrastructure Security

21. 22
Figuring It All Out – SAP Environment Description
ECC, SRM, PI, BW, SolMan, BOBJ, GRC, Portal, Data Services
Bolt Ons
• Vertex
• OpenText
• BPC
Landscapes (Production, Quality, Test, Development, Sandbox, Training)
• Production
• Quality
• Test
• Development
• Sandbox (Technical and Functional)
• Training
5000+ users
850+ unique per day

22. 23
Figuring It All Out – Quick Sizer Results (nothing “quick about it”)
User Based SAPs
Memory and Disk Sizing

23. 24
What It Looks Like Now

24. 25
What It All Looks Like Now – Datacenters and Network
 Two datacenters
• 30 miles by the wire apart
• Connected by 20Gbps (Layer 2) 16 Gbps (Layer 3)
 Network
• Cisco 7k series core
• Cisco 5k series edge
• 10Gbps network

25. 26
What It All Looks Like Now – Physical Servers
 24 ESX Hosts (2 clusters of 12 at each data center)
• HP DL580 G7
• vSphere 5.1 U1
• 4 proc 10 core
• 512 GB RAM
• 2 10GBps ports
 6 ESX Host Stretch Cluster (3 at each data center)
• HP DL580 G7
• vSphere 5.1 U1
• 4 proc 10 core
• 256GB RAM
• 2 10GBps ports

26. 27
What It All Looks Like Now – Virtual Servers
 Production – 76 servers
 Quality – 59 servers
 Test – 27 servers
 Development – 32 servers
 Sandbox (Technical and Functional) – 40 servers
 Training – 3 servers
 Administration – 3 servers
 Total – 240
 Operating Systems – Red Hat Enterprise Linux (RHEL) 5.x to 6.2
and Windows 2008 R2

27. 28
What It All Looks Like Now – Storage
 4 NetApp 6280’s (2 at each data center)
• NetApp Data ONTAP Release 8.1.2P3 7-Mode
• Theoretical capacity 77,000 IOPS (440 disk * 175 IOPS)
• Highest recorded capacity 8212 IOPS
• Raw Capacity:154,493GB or 150TB
• Consumed Capacity: 61519GB or 60TB
• Extensive use of Snap Shot Manager for SAP

28. 29
What It All Looks Like Now – Network Architecture
Datacenter 2Datacenter 1

29. 30
What It All Looks Like Now – SAP vSphere Host Configuration
Datacenter 1 Datacenter 2

30. 31
What It All Looks Like Now – Storage Architecture

31. 32
What It All Looks Like Now – Monitoring
 SAP Solution Manager – Application
 Quest vFoglight – Virtual Server Performance
 Top, Satellite – Operating System
 CA Wily Interscope – Java
 NetApp ONCommand Unified Manager – Storage
 HP SIM – Server Hardware
 Q1, Cascade Shark, eHealth – Network
 CA Spectrum – Operations

32. 33
What It All Looks Like Now – Database
 Oracle 11g (Corporate Standard)
 DataGuard for replication
 One virtual server per database
 Unlimited Oracle for SAP

33. 34
What It All Looks Like Now – Disaster Recovery/High Availability
 Completed a Business Impact Analysis
 12 hour RTO
 Close to zero RPO
 Load-balanced dialog instances
 Central Messaging instance at primary datacenter
 Database instance at primary datacenter
 Database replication happening real time

34. 35
Datacenter 1
What It All Looks Like Now – Disaster Recovery/High Availability
Replication
Log Shipping
Datacenter 1 Datacenter 2

35. 36
What It All Looks Like Now – Why We Virtualized SAP on VMware
 Cost Effective
 Flexibility
 Higher Availability
 Faster Deployment
 Easier to recover in disaster
 Higher Server Density = Lower Data Center Costs
 Virtualization is the SRP corporate standard
 VMware is the leader
 Favorable license agreement with SAP

36. 37
What It All Looks Like Now – Refresh Methodology
 Prepare the source and target systems
 Snap Clone production using Snapshot Manager for SAP
• Mount cloned copies to targeted servers
• Changes the SID
• Minutes instead of hours
 Post clone cleanup
 Validate target systems
 Scrub production data
 Enjoy

37. 38
After the Dust Settled

38. 39
After the Dust Settled - Lessons Learned
 Overbuild
 Plan for disaster now
 Plan out storage configuration early
 Pay attention to security early
• Infrastructure
• OS and Application hardening

39. 40
After the Dust Settled - Successes
 Flexibility
 Speed in reaction
 Environment Refresh rate
 Spin up environments quickly
 Saved $1,000,000 in hardware by virtualizing Oracle

40. 41
After the Dust Settled - Things to Come
 Release 2
• Work Management?
• HR
• More Supply Chain
• More Finance
• Customer Information?
 Mobility?
 HANA?

41. 42
Summary
 Now that we are done – don’t forget your home work
 Figure it all out
 What It All Looks Like Now
 After the Dust Settled

42. 43
Questions

43. 44
Other VMware Activities Related to This Session
 HOL:
HOL-SDC-1301
Applied Cloud Operations

44. THANK YOU

46. How SRP Delivers More Than Power to
Their Customers
Sheldon Brown, SRP
Girish Manmadkar, VMware
VAPP5105
#VAPP5105