For those who have SAP on Itanium and Windows, or wish virtualization on Itanium without Unix ...

1. SAP, Linux,
Virtualization and ...
... Itanium
Solution Proposal Popović Zoran, MSc
Senior System Engineer
07.07.2009.
1. Introduction
2. Environment in this study with Itanium platform
3. Business Requirements
4. Solution Proposal
5. Realization options
6. Added value and benefits for here descibed business
environment
7. References
SAP, Linux, Virtualization and ... Itanium

2. Introduction
First version of proposal based on a prototype system were available last summer. Many
things happened in the mean time, one of which is the Red Hat's decision, by the end of
last December, to discontinue support for Itanium (or IPF, Intel's Itanium Processor
Family) in the announced future release RHEL 6. This was not so unexpected, and RH
states that this decision is made because of the small number of Itanium systems sold in
the recent period, and which doesn't tend to grow. But this story is not likely to end that
fast (RH is fully supporting Itanium on RHEL 5 to the end of it's life in March 2014, while
giving extended support until year 2017 through OEMs), Oracle bought Sun who has
SPARC architecture and other hardware (there was a failed attempt to port Solaris to
Itanium), enters the Xen advisory board (http://xenbits.xensource.com) with people
involved in Open Source contributions, Oracle Enterprise Linux and Xen IA64 (Itanium)
ports. RH6 is to be supported on IBMs POWER architecture, and so are other "traditional"
RISC/CISC CPU architectures, x86 and x86_64 (http://kbase.redhat.com/faq/docs/DOC-
23403). There are no clear indications from other software and hardware vendors about
Itanium's future (at least not the ones I deal with here having their support: HP, SAP and
Microsoft), but there are current benchmarking results and research studies with
predictions.
Back in 2003 my company bought HP EVA 5000 storage and OpenVMS cluster with two
Alpha ES47 nodes to support Oracle 9i RAC database for our legacy application, along
with a rack of Xeon based Proliants (it's a kind of tradition, before them we had Alpha
and VAX systems). A year after or more, it was decided that we should migrate to SAP. It
was an ambitious project: we had 9-12 months from start to going live, implementing
BW3.5, ERP 2004 with MM, WM (having MFC automated warehouse), FI/CO, SD, PP, QM,
PM, TM, logistics and procurement, etc (maybe I have omitted something), everything
that we had in the legacy Oracle application but HR, CRM and some remote locations
(which were covered in later roll-out and optimization projects). We also implemented
Solution Manager as project base and maintenance base, having all the necessary strict
Q&A and GMP procedures. All this had to be supported with appropriate hardware and
infrastructure. The first choice was (among others) was HP's offer with 10 Integrity
(Itanium based) servers and HP-UX. Somewhere in the early beginnings, our project
management decided to change OS platform to - Windows !!! Today, five years later, we
have at least twice larger system (1TB database in size in production, number of users -
up to 800 with expected growth, additional affiliate production plants, automated
warehouses, end users, etc) working quite good with only minor changes (added RAM up
to 16-24G in production, same number of CPUS: 2x mx2 1.1GHz per node), and some
infrastructure improvements. We never had any serious unplanned downtime or system
failures, performance, reliability, availability and stability was predictable (apart from
some OS problems with Microsoft MSCS and one short storage outage).
But, in the past year we had a large roll-out and optimization project involving ERP
upgrade from ECC5.0 to ECC6.0, and it showed that project members, developers, IT
staff and key users always needed (and still need) additional sandbox systems, IDES
demo systems, system copies and similar for different purposes (from experimenting
and unofficial testing, to training and Q&A, project preparation and operation, etc). This
is showing the need for new level of flexibility which could be only obtained with
virtualization. There were many options, from buying new physcial machines (very
expensive even for less expensive architectures), manual consolidation (MCOS and
MCOD, with or without Microsoft WSRM - Windows System Resource Manager), thin OS
SAP, Linux, Virtualization and ... Itanium Popović Zoran

3. (single kernel) virtualization (like VZ, Parallels Virtuozzo, but as we have strict security
standards and procedures which conflict with OS patch level for VZ, this wasn't
convenient even for testing), to full virtualization platforms. It turns out that with Itanium
you can only have HP-UX VSE / Itanium VM or RHEL if you want official SAP support (as
explained later). There is also a very interesting and sophisticated SAP solution -
Adaptive Controller, but for now, Xen on RHEL is doing a very good job. I have two HP
BL870c hosts with 6 active sandbox systems (homogeneous copies of ERP development,
Q&A and production, BI/SEM IDES with Java instance, ERP IDES, etc) and they all work
properly and very stable. From pefromance standpoint, HP-UX with Windows guests
doesn't offer more at all (without AVIO drivers, similar to PV guest drivers on Xen which
don't exist only for Itanium), the only thing I miss at the moment is Live Migration on
RHEL/Xen (which Integrity VM supports, as it also has some other nice-to-have features).
But I am able to move virtual machines to different hosts manually without a problem
(multipathing and shared storage does it's work).
SAP supports virtual guests, without responsibility about their performance, and
database on production systems should be on physical servers if maximum performance
is needed - otherwise, there aren't important reasons not to do it even with database.
We use Oracle 10g at the moment (10.2.0.4) on Windows, which works fine on guest
systems - but because of Windows, it has to be HVM, full virtualization. For optimum
performance (and many other good reasons), the best option would be migration to
RHEL, including production systems (all the tests show that it would be a smooth
transition). There is a general trend about migration of SAP systems from Unix to Linux
(for all the good reasons), while migration from Windows is less popular. Thing is that
60% of all recently sold Itanium servers are Unix (read: HP-UX), 35% are Linux (RHEL
and SLES), and only 5% are Windows (http://www.infoworld.com/d/open-source/red-hat-
will-drop-itanium-support-in-enterprise-linux-6-083). Microsoft has no intention to
introduce Hyper-V on Itanium (as Citrix also doesn't currently, because the Xen code
branch they bought didn't cover ia64), but it is supported in all important flavours on
Windows Server 2008. Migration to a different hardware platform is not an option, just as
migration to HP-UX at the moment (current number of Itanium servers does not justify
that risk and expense). But, the old aging hardware should be either upgraded (but it
costs – additional CPU costs almost as one BL860c, and though it's partly consequenceee
of local vendor's clumsiness, it is not very reasonable), or replaced with new servers – or,
used for non-production (until dead), which is aligned very good with virtualization
(RHEL/Xen paravirtualization is available for old Madison cores, but HVM isn't). There are
better reasons to migrate to RHEL beside this (money saving driver) – decision makers
can decide to change CPU architecture at one point, but with RHEL we can already have
the level of flexibility we need – far better than in any other option: HP-UX is too
expensive, and Windows doesn't offer it. One important aspect is coming from GMP and
other compliance issues, so we need same platform both on test (Q&A) systems and in
production, and this justifies even more migration to RHEL5.
So, the starting point was: SAP ERP and BI systems on Windows Server 2003 on HP
Integrity servers with HP EVA virtualization solution, and the current stop is: some of
those systems working on Xen fully virtualized guest machines on Windows Server 2003
and on HP Integrity blade servers, and some working on paravirtualized guest machines
on RHEL5. Next stop would be more systems involved, and finally – everything migrated
to RHEL, either on Xen virtual machines or bare metal. Also, I find that Integrity blade
servers are far more affordable (comparable to Xeon based Proliants) than expensive
mid-range cell based and other Integrity flavours.
SAP, Linux, Virtualization and ... Itanium Popović Zoran

4. Environment in this study with Itanium platform
In an enterprise I am refering to here, IPF (Itanium Processor Family) architecture is
being used exclusively the needs of SAP systems, there are 18 such servers at the
moment (10х HP rx4640 + 2х rx2620 with Madison cores, 2х rx640 and 2х BL860c with
Itanium dual-core Montecito cores, 2х BL870c with Montvale cores), beside additional
x86 HP Proliant DL360 G4/5 servers (4x without additional servers for support SAP
routers and Web Dispatchers which are in Microsoft Cluster for production just as SAP
central services on production and test systems, or on VMWare platform for support and
other purposes). Beside VMWare as main VTI for x86/x86_64 processor families, HP EVA
Storage nodes are used as form of FC SAN Storage virtualization (also using MSA2012 for
VMWare, additional EVA 8400 obtained, but not yet ready for production).
SAP landscape currently consists of:
− ERP systems (development, test (Q&A) and production), ECC6.0 Ehp3
− BI systems (development, test (Q&A) and production), BI7.0 SP18
− CEN system (transport domain controller, CUA, partly central monitoring), NW04
− Solution Manager (MOPZ, EWA, monitoring, project roadmaps, ticketing, and other)
− SAP routers (for end users, and for support and external access)
− SAP Web Dispatchers (BI portal, WebGUI)
− network printing servers with SAPSprint, SAP Web Console for some RF terminals
− different sandbox and other systems: homogeneous system copies, traning
systems, IDES systems, etc. - all working as Xen guests at the moment
Equipment has life time (it is aging)
Among these 18 servers, 12 servers are entering 5th year of usage: rx4640/2620
servers on which lies the main SAP landscape (it consists of ERD/Q/P, BWD/Q/P
systems). All these servers are very reliable and stable – there never was any serious
hardware failure or issue on production servers (or even any incident as a consequence)
until this day ! But, with hardware aging, support becomes more expensive, additional
components or spare parts also become very expensive, and also vendor desupport
dates are getting closer and closer for equipment and different functionalities (usual
practice is to have maintenance renewal periods and IT equipment amortization within
3-5 years, but sometimes it makes sense to extend it). There are at least two possible
roads: continued use and planning about Itanium platform, or migration to another
processor architecture (as mentioned). If such migration is planned, then all other
technological aspects and existing options must be taken into account (some less
demanding and ”painful”, some are not, but offer different advantages and overall total
cost – just as changing the OS platform, changing the server vendor, storage vendor,
etc. should be considered as well). Some kind of trade-in model or amortization were not
practice up to now as they are not offered by all available vendors currently (used or
vendor-supported and refurbished equipment is usually not an option in production
systems in general) and probably it also will not be in respective future. There is an
option to stay on Itanium platform by sole migration to Itanium blade servers which has
for consequence mandatory need for additional servers based on current requirements
for the main landscape, but without all the other additional systems (this is not justified
and will be explained further on).
SAP, Linux, Virtualization and ... Itanium Popović Zoran

5. Itanium servers – unused brute force
Main argument for Itanium system application is usage within OLTP systems and
databases – an example: for CPU patch 19 for Oracle 10.2.0.4 based on standard
README instructions, downtime lasts about half an hour (given for some internal Oracle
tests for a database with 2000 views and 4000 objects, while our production system has
at least twice as more), on a production system with old Itanium Madison 1.1GHz CPUs
(2x mx2 CPU) it lasts about 5 minutes. But, except for biggest loads during the day
(peak is usually 13:00-15:00 on work days and in the end of the month), this power
mostly remains unused during the rest of the day (average CPU load on our production
does not go over 10% on central instances with database and central service nodes, and
up to 35% on dialog instances, having at least 400 active users among 800 users) and
this is expected state. Consolidated CPU usage and other server resources usage can be
realized by installing additional SAP or DB instances on the same OS instance (operating
system instance, or in general OE, Operating Environment) on the same server, allowed
by SAP MCOD or MCOS installations – but, all there many possible problems of
coexistence of such instances during their work, usage and maintenance life-cycle. That
is one of the reasons why consolidated OS environments should be isolated and that is
usually done through some form of virtualization. It can be justified above all simply by
using non-productive and productive systems and their total cost – but, completely
virtualized production environments have became a standard experience today and also
a need for many business environments (if it is possible to scale resources and estimate
bottom lines, then it is just necessary to take into account the “price” for virtualization).
Existing solutions for virtualization which were considered in this text:
• Citrix XenServer
• Hyper-V / Win2k8
• HPUX VSE/VM, vPars, nPars
• Parallels Virtuozzo / OpenVZ
• Novell Suse / Xen
• Oracle VM
• Fedora / Xen
• FreeBSD / Xen
• Red Hat Enterprise Linux (RHEL), and so is Centos / EPEL / xenbits
• Sun Virtual Box
• QEMU/KVM
• VMWare
(there were also other solutions which were not considered for obvious reasons as
described bellow)
Some solutions were eliminated from the start because those are not available for IPF
and are not even planned to be supported and working on Itanium any time near (Hyper-
V, VMWare, XenServer, Virtual Box, and Oracle VM up to some point) but only
x86/x86_64 (the environment I am refering here to is mostly based on these platforms
and Windows OS, apart from some remaining OpenVMS and Alpha systems). Following
criteria are applied on remaining solutions:
• only full virtualization (hardware based, HVM) or paravirtualization (PVM, more
SAP, Linux, Virtualization and ... Itanium Popović Zoran

6. favoured) with hypervisor is supported (solutions which provide full isolation) – HP
vPars, Virtuozzo, OpenVZ represend OS / “One Kernel“ virtualization which is not
supported (by SAP and other vendors) on production systems
• all other official requirements given by SAP (as a software vendor for our
production systems), HP (as the main hardware vendor for the business
environment I am refering to) and Oracle (as the main software and database
(RDBMS) vendor here) about platform support
• fully working installation prototype which confirms solution feasibility in our system
environment, e.g. boot from SAN
After this all these constraints it has turned out that there only two possible usable
solutions for Itanium: RHEL/Xen and HP-UX Integrity VM (or nPars and vPars but only on
HP-UX). Their comparison with bare metal native systems follows bellow (prices
represent only estimations, in EUR):
license / price Support
Solution options** production
(EUR) 1 year
1000/
RHEL / Xen 0 partially yes
socket
yes, but no
HP-UX Integrity VM 10000* / CPU 1000* yes
PV drivers
HP-UX nPars 10000* / CPU 1000* yes yes
HP rx3600 (4G RAM) 2 core 14000 - yes
HP rx6600 (8G RAM) 4 core 24000 - yes
HP BL860c 4 core (4G RAM) 5000 500 - yes
HP BL870c 8 core (8G RAM) 13000 1000 - yes
HP rx7640 16 core + VSE 115000 - yes
Table 1
Prices given here were extracted from pre-sales and based on old proposals (* - brute
estimation, mostly low bound). One must have in mind (based on proposals from HP by
the end of year 2008) that memory prices for Itanium blade servers is about 240EUR per
4G RAM quad (officially refurbished and similar options are available and have 2-3 times
lower price, good enough for non-production purposes), and this is a very critical
resource for virtualization – while the memory price for older (non-blade) Integrity
servers is even up to 5-10 times bigger. For HP-UX it is also mandatory to have
additional expenses for HP installation and maintenance support including additional
preparations, training and similar activities which are not needed for RHEL (though, HP-
UX systems offer high level of availability, manageability and more, they represent top
of the business and industrial standard, but it is very questionable if this is really
needed). RHEL Xen still does not support many advanced features** (like Live Migration,
PCI passthrough, paravirtualized Windows drivers – HP-UX doesn't support those drivers
too, etc) which are easily enabled on other platforms (x86, x86_64) or which HP-UX
supports, not likely that all of the will be, but some might be.
Furthermore, HP BL870c can support almost 4 completely isolated active guest systems
(even on physically different processors, kind of CPU partitions – HP nPars can not go in
granularity under the physically available resources, including CPU, Integrity VM can),
and without a significant performance fall it can support up to 7-15 such guest systems
SAP, Linux, Virtualization and ... Itanium Popović Zoran

7. (having 16 HT cores) compared to 1-core based bare metal systems, and more if high-
end performance and load is not expected. Of course, number of inactive guest systems
is almost unlimited – ready to be started and used when needed at any time if resources
are available, without additional reinstallation, side-effects and server setup.
Business requirements
Business requirements which emerge during development, testing and usage of all SAP
systems are putting high demands about fast changes and mechanisms which enable
such flexibility having all the existing business standards and SAP usage standards
preserved (GMP and validation practice, QA, availability/reliability SLAs above all, etc).
Specific requirements which arise from these are:
• one or more system copies in a given time (usually ASAP), homogeneous or
heterogeneous – as part of the strategy for testing and/or development (but not
for Q&A needs), and by given requirement or expressed need (testing of new or
risky features, system types/usages and functionalities)
• change of the existing architecture / SAP Landscape – for instance, the problem of
the peak loads on the Q system during testing periods (systems were never sized
for these needs), Support Pipeline (to circumvent transport change and validation
requirements in order to speed up cut-over after system upgrade, roll-out and
similar), training and sandbox systems – last two examples show in our practice
that system copies are far more efficient and usable than any other solution
(aligned with storage split-mirror technologies like snapshots and snapclones)
compared to unjustified client copies or exports on systems with already more
than 1 TB in DB size, which can not support frequent requests for Q system data
refreshments. One temporary solution is also to combine those two procedures (as
used in R/3 system upgrade to ECC6.0 during year 2009), and even more efficient
would be to slightly change validation practice (which in essence remains same,
but using far more efficient system copies). This is aligned with SAP note 432027 -
Strategy for using SAP Support Packages, which describes additional sandbox and test
system usage (e.g. sandbox in Upgrade Master Guide) and as part of the official
landscape.
A step further (a solution and a problem) would be involvement of additional
development and Q&A systems, but also using more efficient change management
and automated test procedures using SAP Solution Manager and eCATT tools
which we already have available at hand without additional investments (but there
also many, many other useful tools) except for additional planned effort by
validation and different SAP teams. Every testing of changes during any of the SAP
projects is a convenient opportunity for preparation and implementation of such
tools and solutions.
• Changes during system patching or system upgrade which require alternate
system landscapes (as during upgrade in year 2009), etc.
SAP, Linux, Virtualization and ... Itanium Popović Zoran

8. Solution proposal
Pool of Servers
The strategy which enables the fastest possible response on large number of requests:
certain number of servers is prepared like an “on hold“ template (firmware / hardware,
FC / network, SAN storage and other needed resources, then the OS installation,
Windows domain name reservation and AD infrastructure settings, antivirus / firewall /
proxy, account specifications and all other OS settings, Oracle RDBMS and SAP s/w
installation), and then by a well defined procedure a copy is made using HP
Storageworks Business Copy technology in a shortest time according to SAP standards
(homogeneous copy based on database copy, with additional post-installation activities
which last not more than an hour). Server in the pool can be activated in a very short
time, deactivated or renewed with fresh data (just by making database copy and post-
installation tasks again).
Many use both physical (native, bare metal) and virtual guest systems equally, based
on specific needs and requested system types. Also, this can be mixed, for instance, as a
phase in the implementation of virtual guests for productive systems – database is left
on physical platform, while dialog instances (application servers) work on virtual guests,
which is a well known and well used practice. In the meantime, hardware and
virtualization platforms grew enough to support completely virtualized systems which
can be used without special constraints (apart from the aforementioned performance
factor). Oracle gives also additional opportunities with RAC (Real Application Cluster)
option in terms of scalability and HA (which is additionally licensed). This option is useful
also with physical servers, and with MCOD (Multiple Components on One Database - one
database for many SAP systems) gives also a solution with optimal performance (it has
also same downturns as for consolidated systems, where patching and maintenance
introduces risks for many systems).
Virtualization
Xen originates from the 90-ies as a Cambridge University project, and also as VMWare
today it represents one of the best known virtualization platforms. The idea of
virtualization exists for decades (starting from 60-ties and IBM which introduces it as
part of OS for mainframe systems). In the meantime Xen code branch was acquired by
Citrix, and similar technology is also used by Microsoft Hyper-V VTI platform. Beside
those, Open Source / GPL code branch of the Xen development remains (currently
version 3.4 is available) which is used by many Unix/Linux distributions, among which
also does RHEL (RHEL5.3/5.4 is packed with and using Xen 3.1/3.2). It is announced that
RHEL 5.4 and following updates are also going to support KVM (Kernel based Virtual
Machine) as the hypervisor (but not on Itanium so far), with possible improvements for
paravirtualized drivers for Windows guest systems under IA64, PCI Passthrough is
already available with VT-d on other Xen platforms, and other improvements.
Xen can offer a very good solution with minimal investments in memory hardware on
some servers, and in support subscriptions for RHEL OS (more details follow in next
section). RHEL as a base platform support all important requirements which are
implemented and expected from servers on Windows Server 2003 currently (e.g. SAN FC
redundant multipathing for HA standards, which is also useful for a form of migration of
virtual server – GFS/Cluster options are not needed so far but are available (specially for
SAP, Linux, Virtualization and ... Itanium Popović Zoran

9. PV systems), having guest HA mechanism yet to be tested). SAP note 962334 shows that
SAP supports this platform (RHEL, Xen), but additional checks and tests would be needed
for production systems. In any consolidation option this virtualization solution on Itanium
platform is more optimal than any other, and it is also complementary to existing
VMWare infrastructure as a proven solution for other platforms. Xen and VMWare can
support all possible given requirements in the environment I am discussing here. With
these technologies (together with Charon/SIMH emulators for Alpha/VMS systems, which
are already tested on some VMWare guests) it is possible to support virtualization
infrastructure for all platforms and systems in I was mentioning. In the following ROI
analysis available in the text bellow, it was not taken for whole infrastructure with
complete TCO structure where virtualization can potentially bring only bigger index
(which has to be validated in each special case – for Oracle above all):
• the advances of licensing – many vendors give special licensing policies for
guests, all somehow licensing expenses can be lower or somehow tend to favour
virtualization trends, for HP VSE is is known that it can be based on partitions (but
has to be checked with vendor or partners)
• simplified system maintenance, etc.
First prototype – testing has already started
Currently, we have a system working non-stop and being actively used for second
month without complaints (after the upgrade of the production system), the SAP-TEST
guest system (as a replacement for the previous ERT system in the alternate landscape,
8 VCPUs on 8 virtual cores or 2 dedicated CPUs, and with 6G RAM so far) on BL870c host
(SAP-ERQ6) and no interruptions – above all, very stable and reliable, in the basic
RHEL5.3 version (host was not updated or patched so far). System was successfully
presented with copy of the production OLTP database, and the complete installation of
the guest Windows 2003 EE SP2 system was done by the standard operational
procedures. It is even possible to use some of the standard EFI tools on the guest, a
successful manual migration was tested, and a copy template system was also made (on
SAP5), by which all important functional requirements were fulfilled for using
virtualization platform in our environment. The only problem with this solution (as
mentioned about the missing features for Xen on Itanium) is a very poor disk I/O. This is
the consequence of the needed full virtualization (HVM) for Windows guests (3-6 times
slower than on bare metal, while network I/O is less deteriorated). This makes critical
impact on database performance in some cases (though parsing is done faster than on
older bare metal systems), which makes the whole ERT system in average slower (e.g.
closing the financial period in SAP takes 4 hours in batch instead of 2.5 hours as in
production system).
There are many workarounds and solutions for this problem - using paravirtualized I/O
drivers if it was possible, iSCSI/NFS storage approach (not top performance, again), etc -
but the best thing would be moving database to physical machine, using MCOD
wherever possible. Database nodes wouldn't be easily reallocated, but one of the
possible solutions in future would also involve adaptive infrastructure (SAP ACC, with
Solution Manager - Adaptive Computing Controller) as part of the SAP strategy for
system usage and consolidation:
• all SAP systems can adapt to business needs very fast and in a flexible manner
SAP, Linux, Virtualization and ... Itanium Popović Zoran

10. • dynamic load balancing according to system load (physical and virtual)
• easier SAP landscape maintenance
• HP-UX prototype is being prepared (the goal is to make sample comparison
with Xen)
All these goals can be accomplished without SAP Solution Manager ACC plug-in, of
course, as proposed in this paper with some additional manual procedures and way of
system maintenance. Optimal solution would involve ACC because not only that it avoids
better possible human errors, but it also follows vendor recommendations and standards
better given with SAP Solution Manager as the base technical tool which is really useful
in all system maintenance and management tasks according to GMP / ITIL / ITSM, and
which is free of charge (unless additional Service Desk Extended component and
functionalities are needed for non-SAP systems and Service Desk / Help Desk, and
similar). Therefore it makes sense to make additional effort and improve this system and
it's usage in our environment.
SAP, Linux, Virtualization and ... Itanium Popović Zoran

11. Realization options
These are the only existing usable IPF consolidation platforms and their perspectives:
• acquisition of new physical servers (without hypervisor and virtualization)
Pros: most efficient (but not optimal) use of existing knowledge and resources
Cons: price of 1 Itanium server is between $5К and $20К in average, and more
(without support agreements and other hidden maintenance costs, human labor)
• HP-UX Integrity VM hypervisor:
o HP Integrity VM (software virtualization) as part of VSE (Virtual Server
Environment) offers also full isolation, as opposed to vPars (similar to OpenVZ
and appropriate Parallels Virtuozzo solution)
o HP nPars as part of VSE (cell based virtualization, similar to IBM LPARs, HP-UX as
hypervisor is only controlling them), demanding specific hardware (rx7640 or
stronger, like Superdome)
Pros: a highly stable platform which represents on of the leading industry standards
Cons: the licenses only for MC OE (DCOE, VSEOE now) are about $10000 per CPU
(TCOE as a possible minumum is more than $2000 per CPU), performance factor
• Xen with RHEL5 as hypervisor
Good side: supported by SAP and HP, Open Source, quite stable and tested, in-
house knowledge available.
Cons: does not support all the features other options have (at least some of the
features like live migration), performance factor
All these solution are stable and robust, and some details about maintenance and
support for RHEL / Xen infrastructure are given in the table bellow (prices are given in
EUR, per host) – RHEL Support is excellent and reliable at least as any other vendor's
support, with some advanced maintenance and provisioning features including standard
ticketing and case management:
Virtualized 1 year 3 year
Type options
Guests support support
1 x Red Hat Enterprise Linux
1. RHEL Advanced 1 x Red Hat Cluster Suite
Platform, Standard 13x5 unlimited 1 x Red Hat Global File System 1199 3057
(unlimited sockets) 1 x Red Hat Network Update
Module
1 x Red Hat Enterprise Linux
2. RHEL Advanced 1 x Red Hat Cluster Suite
Platform, Premium 24x7 unlimited 1 x Red Hat Global File System 1999 5097
(unlimited sockets) 1 x Red Hat Network Update
Module
SAP, Linux, Virtualization and ... Itanium Popović Zoran

12. Virtualized 1 year 3 year
Type options
Guests support support
3. RHEL Standard 1 x Red Hat Enterprise Linux
Platform, Standard 13x5 4 1 x Red Hat Network Update 639 1629
(2 sockets) Module
4. RHEL Standard 1 x Red Hat Enterprise Linux
Platform, Premium 24x7 4 1 x Red Hat Network Update 1039 2649
(2 sockets) Module
* DVD prices for Red Hat Enterprise Linux 5 (for Intel Itanium) Media Kit: around 30 eur
Table 2
For our testing purposes the 3. type of subscription would be sufficient, specially in first
phases of realization (HA features are not needed yet, and so Advanced supported
options for GFS / Cluster Suite, we can rely on FC SAN / EVA Storage, and we are not
using RHEL PV guests at first), but as we advance and increase number of systems, and
as we plan to test RHEL OS as guest platform, (on stronger servers) it makes sense to
take the type 1. support subscription. Prices are estimated for one server – minimally 1-6
subscriptions can be considered (that is the number of servers available for HVM for
Windows guests, not all should be covered from the start during testing, old servers
support only RHEL PV guests). The smallest expense is 640 euros, some optimal expense
is 1920 or 3840 a year at the moment (for HP-UX TCOE licenses themselves are 6-7
times more expensive, or up to 30 times for MCOE, without support – for more exact
information can be obtained from the vendor or his partners). It always remains possible
to change or move to a bigger number of subscriptions. RHEL or his partners might also
provide us with experienced consultants in order to get help with some phases of
implementation.
There is a possibility to obtain RHEL support from HP as hardware vendor indirectly, but
I would not recommend this (as it is not justified). I do not have prices for this option, but
having our environment and level of support in the nearby call center (proactive support
for SAP systems hardware) I recommend obtaining subscriptions directly from RHEL. We
might get some additional suggestions from HP in the meantime, though.
Realization phases
In general, following implementation gradual phases and steps are proposed in a short
overview:
1. Prototype preparation and testing (in process).
2. Virtualization of all IA64 sandbox (and test) systems and preparation of pool of
servers (for system copies), Solution Manager (SOD), dialog instances of
development systems.
2.1. In parallel, all 32-bit SAP servers on VMWare platform should be virtualized (old
Solution Manager, IDES, CEN; even SAP routers, but not those for end users, for a
start).
3. Complete virtualization of development systems.
4. Test drive virtualization for Q&A and production systems (they must be aligned,
dialog instances first).
5. Complete virtualization of the whole SAP landscape.
SAP, Linux, Virtualization and ... Itanium Popović Zoran

13. Taking Q&A servers together with productive servers is necessary because of the nature
of their intended usage. The last phase is not yet planned (if it ever will be), at least not
with a final date or specific requirements at the moment (higher reliability (disaster
recovery), greater security and scalability). This might justify HP-UX as an option as it's
implementation has references nearby, with full implementation and integration support
from HP in a critical environment (with given dates).
For the successful implementation of these phases, good preparation is crucial, specially
about planning needed resources:
• storage space
• available network and optical ports and switches, settings on firewall, etc.
• needed total number of servers in the landscape (categorized according to usage
types and associated with possible hosts)
• this provides information needed to estimate people-days needed for
implementation of the pool of servers in each of the phases, and also physical
resources needed for the requested service levels according to user requirements.
Much of storage space is saved using HP BC Snapshot functionalities (as during
upgrade), but these also have some level of impact on EVA performance and puts
constraints on original VLUN, and makes storage maintenance more complex. Price of
the EVA FC disks is about 15-20 eur/G which also has to be noted (snapshot grows up
to about 10% of the original disk size in a normal usage period, which gives around
1.5-2 eur/G) – there are other storage options, but there also other parts of the TCO
structure:
• backup (about 7 eur/G in average, but varies from 1 to 15eur)
• network equipment and security (antivirus, firewall, LAN, FC), licenses
• guest OS licenses and support agreements
• infrastructure in the data center room (space, air conditioning, power supply)
• system maintenance (human resources, licenses for some monitoring solutions)
• other hidden expenses ...
Therefore, it is of the utmost importance to have business requirements carefully
prepared and estimated, because all real technical requirements (like additional
hardware or licenses, memory, servers, storage). For instance, if large enough number
of servers is hosted (which grows very easily with virtualization), so grows the I/O
bandwidth on SAN or the number of storage spindles.
Validation practice and documentation is very important and the mandatory
prerequisite in order to make everything work, among all the other standard
operational procedures and working instructions – SOPs for system copies, virtual
hosts and guests installation, pool of servers qualification above all, their
maintenance, and all other standards that we follow give a good base for equal
and smooth usage of virtual systems among the usage of all our systems and their
maintenance.
SAP, Linux, Virtualization and ... Itanium Popović Zoran

14. Added value and benefits for
here described business environment
The total cost of one server is certainly larger if seen through it's physical partitions
(e.g. two BL860c are approximately physically same as one BL870c), not taking into
account the support agreement price, and the return of investment grows with each
added virtual guest system. For instance, if one BL870c server hosts 4 virtual guest
systems, with about 1500 euros per year for support for RHEL and about 1500 euros for
16G RAM (giving 8G of RAM for each virtual system given the current state, which is
enough for every standard SAP installations), we get 4 virtual guest systems with power
similar to 4 physical BL860c servers with 2 CPUs given their level of usage (price of one
such is in the order of 5000 euros, while the price of BL870c is almost three times as
that). Then, the 5th server gives around 100% ROI (or, respectively, 3rd server if the
price of host is already taken as what we have), under condition that the usage level of
each guest does not go over the current usual usage level of BL870c (which is the case
we have) and so the price of one virtual server equals with one physical partition in CPU
power (about 90% of 1 BL860c).
If we add to one such host additional 3 active guest systems with the investment in
total 32G RAM (2000-3000 euros), ROI goes over 200%, and so on. In that manner we
can go up to added or return value of at least 100000 euros (depending on the given ROI
model) with existing 2-6 host servers and up to 20-28 virtual guest systems (with
minimal total investment of 5000 euros, having not quite optimal but well balanced
usage of physical resources). This goes, once again, only under condition that the total
required active guests load is not greater than the available host resources (CPU can not
be added into BL servers, but can be added to Integrity rx servers). If this solution is
seen as part of the pool of servers, value can be far more bigger (not all systems are
needed active) ! With HP-UX option, this ROI decreases drastically, or it almost
diminishes. So, the number of all inactive servers is also very important having no
significant limitation in their number (only about their storage space). Virtualization
enables also dynamic resource allocation and balancing physical resources, but not
more than that – host can get overloaded if this is not carefully planned, and sometimes
only the physical consolidation is possible. This can be illustrated by following: two
BL870c have the same number of SAPS (SAP's standard benchmarking system
measurement, above about CPU power, but also about all other physical resources) as
all 6 productive Integrity rx4640 servers have, but in such a physical consolidation it is
not possible to have two MSCS clusters (not even with two additional BL860c), and there
is no isolation between database and dioalog instances, and HA (High Availability) is
then not justified – while these servers can work excellent as virtualization platform for
the same purpose. What do we get, beside the obvious added value:
• Flexible system maintenance and usage according to business needs (which is not
possible otherwise, due to expenses, human resources or for other reasons).
• SAP landscape remains platform homogeneous in the as much as possible extent
(all guest systems and physical systems remain on the same IA64 platform, but
some h/w differences remain – though, SAP and Oracle mostly rely on OS which is
not that dependent)
• Optimally used existing Itanium servers (both old ones and planned new ones,
according to usage – no easy “throwing away”), better consolidation later on.
SAP, Linux, Virtualization and ... Itanium Popović Zoran

15. Details about implementation
Installation of current environment is divided in three groups:
• installation of hosts (Dom0s)
• installation of HVM (fully virtualized) Windows guests
• installation of PV (paravirtualized) RHEL guests
While the installation of hosts and PV guests has many similar steps (network setup – I
am using trunk utility as local NTLM proxy for http/ftp, RHN registration, ntfs-3g uitilities
installation, etc), there are things to be set only on hosts (xend service and boot
parameters, HP support pack installation, etc).
For guests I am generally proposing using copies of the template systems for many
different reasons, one of which is using good features of EVA storage (snapshots and
snapclones) or LVM on local disks. If making guest from the scratch (or new template), it
is always best to use physical (phy:/) drives instead of files – they have better
performance, enabling moving to another host (with shared storage), and always use
multipathd.
I am giving the following hints for people who have experience with Windows
installations and SAP administration on Itanium (and who have some basic knowledge
and awareness of things about Linux) – many things are similar, but there are also many
misleading details.
SAP, Linux, Virtualization and ... Itanium Popović Zoran

16. HOST INSTALLATION
- RHEL installation is quite simple and straightforward – no EFI boot driver is needed
(Windows installation needs it), usual server preparation, firmware update if needed,
boot device configuration in EBSU setup disk – boot from SAN is used, so HBA adapater
should be configured there and a VLUN prepared, after booting from installation disk,
installer should be started manually by entering “linux mpath vnc=1” (vnc is optional,
GUI is then available later through vnc client on screen 1, or http://that-host:5901)
- installation number can be entered after the installation (using rhn_register),
Virtualization group should be chosen if available (beside Development, Web Server and
others) - or later, packages to be installed are: xen, kernel-xen, xen-ia64-guest-firmware,
virt-manager, libvirt and virt-viewer, or just “yum groupinstall Virtualization” (if doing it
manually with rpm -ivh in VT folder of the installation disk: libvirt-python, python-virtinst,
Virtualization-en-US, perl-Sys-Virt, xen-devel ../Server/bridge-utils ../Server/xen-libs
../Server/gnome-python2-gnomekeyring ../Server/iscsi-initiator-utils ../Server/gtk-vnc
../Server/gtk-vns-python ../Server/cyrus-sasl-md5)
... installation lasts up to 1 hour – all setup parameters (like network address, gateway,
etc) should be prepared
... check if /boot/efi/efi/redhat/elilo.conf is set correctly before restarting
... /etc/resolv.conf should contain at least one line "nameserver DNS_IP_address" and
one "domain group.company.com", enabling and disabling network interface is done
with ifup eth0 / ifdown eth0
- on some bad terminal emulation/clients after restart, firstboot gets stuck (additional
setup wizard), which shuts down automatically after about 10 minutes, and after logging
in it can be disabled with “chkconfig firstboot off”
- trunk (local NTLM proxy, http://ntlmaps.sourceforge.net) after unpacking with tar -xzf
has to be configured (NT_DOMAIN:HEMONET, USER:proxyuser, PASSWORD:somepass,
PARENT_PROXY:proxy_server) and started manually with “scripts/ntlmaps &” (or put
into /etc/rc.local to start automatically after boot), then it is possible to start
“rhn_register –proxy=http://127.0.0.1:5865”
- also, for other utilities accessing internet, like yum (package installer), edit
~/.bash_profile (using gedit in GUI console, or vi editor):
export http_proxy=http://127.0.0.1:5865
export HTTP_PROXY=http://127.0.0.1:5865
export ftp_proxy=http://127.0.0.1:5865
export FTP_PROXY=http://127.0.0.1:5865
... update can be started with “yum update” after setting up software channels and
other additional settings on RHN side (if needed)
NOTE: LVM/multipath is not good is not good with FAT partitions, so before each kernel
update/installation it is needed to mount manually /boot/efi (e.g. with “pvs” one can get
sytsem disk device for system VG LogVol00, like /dev/sd..., for instance /dev/sdh2, and
then EFI partition is mounted with: “mount -t vfat /dev/sdh1 /boot/efi”)
- using Windows (cifs) shares: e.g. serverx$ can be mounted on /mnt/tmp with:
mount -t cifs -o dom=HEMONET,username=someuser //server/x$ /mnt/tmp
... or for simple copying smbclient is useful (like ftp)
SAP, Linux, Virtualization and ... Itanium Popović Zoran

17. - ntfs-3g installation (http://www.ntfs-3g.org):
cp /mnt/tmp/rhel/ntfs­3g*.tar
tar xvf ntfs­3g....
cd ntfs­3g...
./configure
make
make install
yum install ntfsprogs
- firewall should be configured (opening needed ports, system­config­firewall) and
selinux (for a start and troubleshooting, “setenforce 0” and SELINUX=permissive in
/etc/selinux/config)
- Xen Dom0 configuration for several network adapters, create /etc/xen/scritps/network-
multi-bridge:
#!/bin/sh
dir=$(dirname "$0")
"$dir/network­bridge" "$@" vifnum=0 bridge=xenbr0 netdev=eth0
"$dir/network­bridge" "$@" vifnum=1 bridge=xenbr1 netdev=eth1
"$dir/network­bridge" "$@" vifnum=2 bridge=xenbr2 netdev=eth2
"$dir/network­bridge" "$@" vifnum=3 bridge=xenbr3 netdev=eth3
... and then: “chmod +x /etc/xen/scritps/network­multi­bridge” and edit
/etc/xen/xend-config.sxp, putting “(network­script network­multi­bridge)” instead
of “(network­script network­bridge)”
... also, put there (dom0-min-mem 2048) ... and dom0_mem=2G into elilo.conf to
reserve 2G for Dom0 and avoid memory ballooning (2G should be enough) – elilo.conf
example:
...
image=vmlinuz­2.6.18­164.10.1.el5xen
vmm=xen.gz­2.6.18­164.10.1.el5
label=2.6.18­164.10.1.el5xen
initrd=initrd­2.6.18­164.10.1.el5xen.img
read­only
root=/dev/VolGroup00/LogVol00
append="dom0_mem=4224M ­­ palhalt rhgb quiet"
...
- vncserver setup (/etc/sysconfig/vncservers), vncpasswd should be set for the running
account, and initial vncserver start (“chkconfig vncserver on” for the service, too)
- hp support pack installation:
yum install net­snmp net­snmp­libs
./install.sh (options: 1,2,3,8)
yum install tog­pegasus
service tog­pegasus start
domainname=group.company.com is needed in /etc/sysconfig/network - in case of
additional problems with hpsmhd start, in /etc/hosts should be put:
host-IP-address host.group.company.com host
... initial configuration “/etc/init.d/hpima sample [agent]” (or better,
reconfiguration, especially after each kernel update):
SAP, Linux, Virtualization and ... Itanium Popović Zoran

18. /etc/init.d/hpmgmtbase reconfigure
... (and additional restart of services: tog-pegasus, hpmgmtbase, hpima, hplip, hmsmhd)
/etc/init.d/hpima reconfigure
/etc/init.d/hpima start
... kernel parameters and related settings and recommendations (by Oracle, SAP):
/etc/rc.local:
ulimit ­u 16384 ­n 65536
modprobe hangcheck­timer hangcheck_tick=30 hangcheck_margin=18
/etc/sysctl.conf:
kernel.msgmni=1024
kernel.sem=1250 256000 100 1024
vm.max_map_count=300000
net.core.rmem_max=1048576
net.core.rmem_default=1048576
net.core.wmem_max=262144
net.core.wmem_default=262144
- old A/P storages should have excluesively Failover (not failback, not None) in the
prefered path in the presentation options
- multipath.conf example (starting the service and setting chkconfig also is needed):
/etc/multipath.conf:
blacklist {
devnode "^(ram|raw|loop|fd|md|dm­|sr|scd|st)[0­9]*"
devnode "^(hd|xvd|vd)[a­z]*"
wwid "*"
}
# Make sure our multipath devices are enabled.
defaults {
udev_dir /dev
polling_interval 10
selector "round­robin 0"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id ­g ­u ­s /block/%n"
prio_callout "/sbin/mpath_prio_alua /dev/%n"
path_checker tur
rr_min_io 100
rr_weight uniform
failback immediate
max_fds 65536
no_path_retry 12
user_friendly_name yes
}
blacklist_exceptions {
wwid "3600508b4000129f70002900002030000"
wwid "36005*"
wwid "36001*"
}
multipaths {
multipath { /* for newer A/A storages, from EVA VCS4.0 */
wwid 3600508b4000e302d0000a00001200000
SAP, Linux, Virtualization and ... Itanium Popović Zoran

19. alias sap­trn­sys
}
multipath { /* for A/P storage, up to EVA VCS4.0 */
wwid 3600508b4000129f70002900000cb0000
path_grouping_policy multibus
path_checker readsector0
prio_callout /bin/true
no_path_retry queue
rr_weight priorities
alias sap­bisem­sys
}
...
}
devices {
device {
/* EVA 3000/5000 with new firmware, EVA 4000/6000/8000, EVA 4400 */
vendor "(COMPAQ|HP)"
product "HSV1[01][01]|HSV2[01]0|HSV300|HSV450"
getuid_callout "/sbin/scsi_id ­g ­u ­s /block/%n"
prio_callout "/sbin/mpath_prio_alua /dev/%n"
features "0"
hardware_handler "0"
path_selector "round­robin 0"
path_grouping_policy group_by_prio
failback immediate
rr_weight uniform
no_path_retry 12
rr_min_io 100
path_checker tur
}
/* MSA */
device
{
vendor "HP"
product "MSA2[02]12fc|MSA2012i"
getuid_callout "/sbin/scsi_id ­g ­u ­s /block/%n"
features "0"
hardware_handler "0"
path_selector "round­robin 0"
path_grouping_policy multibus
failback immediate
rr_weight uniform
no_path_retry 18
rr_min_io 100
path_checker tur
}
}
- Data Protector installation (MUST be started from the PA-RISC disk, e.g. B6960-15008-
01.tar.gz or downloaded from ITRC SUM):
yum install xinetd (if not installed)
... add port 5555 in iptables firewall (/etc/sysconfig/iptables "-A INPUT -m state --state
NEW -m tcp -p tcp --dport 5555 -j ACCEPT")
./omnisetup ­source /root/hp/B6960­15008­01/ ­server
dprotector.group.company.com ­install da,ma,cc
chkconfig ­­list | grep omni
... if no service found, edit manually /etc/xinted.d/omni:
SAP, Linux, Virtualization and ... Itanium Popović Zoran

20. service omni
{
socket_type = stream
protocol = tcp
wait = no
user = root
server = /opt/omni/lbin/inet
server_args = inet ­log /var/opt/omni/log/inet.log
disable = no
}
... INFO: linux rescue is a method of disaster recovery, by booting installation disk with
“linux rescue” or better, “linux rescue mpath”, and it is also possilbe to do a
reinstallation or upgrade over the existing OS file system (as a repair method) by
choosing any of these options, guided by the installer dialigues
!!!!! NOTE: always, always make backup of everything that is important ...
SAP, Linux, Virtualization and ... Itanium Popović Zoran

21. PV GUESTS
- GUI installation is also available (vnc) as with Dom0, the only important difference is
that NFS/FTP/HTTP can only be used as the installation source
- installation procedure is as easy as with Dom0, Virtualization group should be select if
given (installation number can be entered afterwards with rhn_register) and in
/boot/efi/efi/redhat/elilo.conf must be set el5xen (xen kernel) option - Xen supports
different kernel versions on host and guest, but SAP recommends they should be same
- Oracle JDK/JRE home can be used for sapinst GUI, but it is recommended to use Sun
JDK
- naming conventions differ from Windows significantly; Oracle is using additional OS
account ora<sid>, oracle home must is in format /oracle/<SID>/102_64 (this _64 is
mandatory), even if you put something else, sapinst refuses to work normally later
- after DBCA database creation (if needed):
cd $ORACLE_HOME
su
<rootpassword>
./root.sh
exit
- starting / stopping SAP:
- Oracle:
(vi ~/.dbenv_<server_name>.csh and there can be changed SID, ORACLE_HOME,
SCHEMA or whatever needed in oraerc's profile, where oraert is the template Oracle
user):
su ­ oraert
lsnrcstl start/stop
sqlplus / as sysdba ...
- SAP:
su ­ ertadm
startsap r3
stopsap r3
(using all instead of r3 if Java stack is present, as it is with BI systems)
- shutting down:
shutdown ­h 0 (as with other guest or bare metal systems, there is always a risk with
forced shutting down of damaging system disk or something more – linux rescue boot
with prepared sap-rescue configuration with affected system disk would help, or restore
from a backup ...) or init 0
!!!!! NOTE: always, always make backup of everything that is important ...
SAP, Linux, Virtualization and ... Itanium Popović Zoran

22. HVM GUESTS
- It is enough to copy existing template configuration (e.g.. /etc/xen/sap-test) for a new
guest, having at least changed name=new_machine (instead of name=sap-test), and
also /usr/lib/xen/boot/nvram_sap-test into /usr/lib/xen/boot/nvram_new_machine
(this is instead of nvrboot import from the backup on the EFI partition – HVM guests have
their own EFI environment, just as any bare metal machine)
- for new guest disks created either as snaphots / snapclones or CA replicas of the
template disks, check the parted /dev/disk/by-id/... print before proceeding – if any I/O
errors occur, try changing in EVA Command View the Presentation / Preferred path/mode
on old EVA 5000 (A or B, when upgraded to new A/A VCS4.xx firmware this will no longer
be a problem).
It recommended to use for each VLUN it's wwid set accordingly in /etc/multipath.conf
with appropriate /dev/mapper/alias ...
With ntfsls utility one can even check the M$ partition without any mounting
- for each new disk (which is not a copy/replication, not having gpt label) initialization
must be done: parted /dev/disk/by-id/... mklabel gpt (it is also possible as usual in EFI
shell) ...
- after Windows boot and administrator login, first the IP address should be set in the
GUI console (vnc) and remote desktop enabled (for a newly installed machine, it should
be already set in the template) and then continue working with RDP (rdesktop)
- the next step is changing the name to the machine and adding it to Winows domain:
newsid /a novoime (restart should be done only from virt-manager or with xm create,
anything other will fail to bring up network correctly)
- machine preparation is the same as with any other Windows machine (Oracle and SAP
installation, mind installing Montecito Oracle patches if needed)
- SAP ERT 30 instance in the template with preinstalled ERT instance can be used only
for the same machine, which comes down to same MAC address, having same database
schema and licenses preserved with (otherwise, additional setup is needed):
... on the old system:
exp file=lic.dmp userid='/ as sysdba' tables=(SAPERP.SAPLIKEY)
... on the newly copied system:
imp file=lic.dmp userid='/ as sysdba' fromuser=SAPERP touser=SAPERP
- SAPDATA_HOME and oraarch folders (or whatever set in sapinst for the template) are
best to be kept on C: - HVM guests have that unfortunate constraint to have maximum
four drives
!!!!! NOTE: always, always make backup of everything that is important ...
SAP, Linux, Virtualization and ... Itanium Popović Zoran

23. KNOWN PROBLEMS
I had several problems during during all these tests, and some of them are significant
issues:
• EFI environment can slow down vnc console and virt-manager refresh terribly for
unknown reasons (xm commands are usable, but also a bit slower, while the guest
domains and Dom0 work perfectly normal) – a RH service request is open on this
one
• ORA-07445 occurs intermittently on Dom0s and DomUs (but does not on normal
kernel) – a metalink SR is open on this one (filed bug 8868468) - the only viable
workarounds are not to use Xen for Oracle, or to somehow prepare database
without Xen, and then use it under Xen (this is not causing serious issues, but it
would be unacceptable in producion)
SAP, Linux, Virtualization and ... Itanium Popović Zoran

24. References
Note 962334 - Linux: SAP on Xen virtual machine
Note 1048303 - Red Hat Enterprise Linux 5.x: Installation and upgrade
Note 958253 - SUSE LINUX Enterprise Server 10: Installation notes
Note 941735 - SAP memory management for 64-bit Linux systems
Note 964705 - Linux IPF: "floating-point assist fault" in Linux syslog
Note 784391 - SAP support terms and 3rd-party Linux kernel drivers
Note 527843 - Oracle RAC support in the SAP environment
Note 592393 - FAQ: Oracle
http://en.wikipedia.org/wiki/Xen
http://www.linux-kvm.org/page/Small_look_inside
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Virtualization_Guide/index.html
http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Virtualization-en-US/task-virt-lab1.html
http://www.redhat.com/legal/open_source_assurance_agreement.html
http://kbase.redhat.com/faq/docs/DOC-10113
http://kbase.redhat.com/faq/docs/DOC-17364
http://kbase.redhat.com/faq/docs/DOC-23403
http://www.scribd.com/doc/3210098/xen-install-windows
http://forums.citrix.com/thread.jspa?threadID=241177
https://www.redhat.com/apps/store/server/
http://www.redhat.com/rhel/virtualization/
presales 1-888-REDHAT1 ext 45031.
http://kbase.redhat.com/faq/docs/DOC-23403
http://www.encyclopedia.com/doc/1G1-168820045.html
http://www.eweek.com/c/a/IT-Infrastructure/Intel-Allies-Look-to-Linux-for-Itanium-Growth/
http://www.itaniumsolutions.org/resources/red_hat_enterprise_linux_virtualization_and_intel_itani
um_linux_open_source_virtualization_with_enterpriseclass_features
http://whitepapers.techrepublic.com.com/abstract.aspx?docid=301970
http://blog.xen.org/index.php/2009/11/18/fujitsu-and-va-linux-systems-japan-k-k-join-xen-advisory-board
http://www.oracle.com/us/technologies/linux/025994.htm
http://community.citrix.com/display/ocb/2008/12/18/The+Oracle+comes+to+Xen
http://www.hp.com/go/lxintegritycert
http://www.hp.com/go/integrityrhel
http://h18000.www1.hp.com/products/servers/linux/redhat/howtobuy.html
https://www.redhatrenewals.com/rhel/compare/
http://www.itjungle.com/tlb/tlb021908-story02.html
http://www.centos.org/docs/5/html/5.2/Virtualization/chap-Virtualization-Configuring_ELILO.html
http://docs.hp.com/en/5992-3193/apcs05.html
Oracle VM http://download.oracle.com/docs/cd/E11081_01/doc/doc.21/e10898/createvm.htm
http://ia64.koji.fedoraproject.org/fedora-ia64/rawhide-latest/development/ia64/os/Packages/
http://fedoraproject.org/wiki/Architectures/IA64
https://www.scientificlinux.org/news/archive/sl41.ia64
http://www.linux-kvm.com/content/using-windows-installer-paravirtual-network-drivers
http://xenbits.xensource.com/ext/efi-vfirmware.hg
SAP, Linux, Virtualization and ... Itanium Popović Zoran

25. http://xenbits.xensource.com/ext/ia64, http://www.selenic.com/mercurial/wiki/
http://www.selenic.com/mercurial/wiki/index.cgi/UnixInstall
http://www.kernel.org/pub/software/scm/git
http://www.mail-archive.com/virtualization@lists.linux-foundation.org/msg05115.html
http://logout.sh/computers/linux/netboot/
http://www.opensourcehowto.org/how-to/tftp/tftp--pxe--syslinux--fedora-install.html
http://www.tpc.org/tpcc/results/tpcc_perf_results.asp
http://www.tpc.org/tpcc/results/tpcc_price_perf_results.asp
http://ts.fujitsu.com/ps2/press/read/news_details.aspx?id=2202
http://www.oracle.com/corporate/press/1524171.html
http://www.nec.co.jp/press/en/0802/2101.html
http://en.wikipedia.org/wiki/Rate_of_return
http://en.wikipedia.org/wiki/Total_cost_of_ownership
benchmarks:
SAPS http://www.sap.com/solutions/benchmark/measuring/index.epx
http://www.sap.com/solutions/benchmark/index.epx
http://service.sap.com/benchmark
http://www.sap.com/solutions/benchmark/sd3tier.epx?num=200
http://www.sap.com/solutions/benchmark/sd2tier.epx
Solaris http://www.c0t0d0s0.org/archives/4962-SAP-Benchmarks-revisited.html
http://www.sapdb.org/7.4/sap_db_benchmarks.htm
http://www.redhat.com/f/pdf/Virt_SAP_SD_DB2_Reference_Architecture_V1_01-08-2009.pdf&ei=t3tbSrOSCsjI-
QaQy5SBCQ&rct=j&q=SAP+benchmark+SAPS&usg=AFQjCNHLcq9bYTOPSXUPGzXDmtQIBwD6xA
http://download.sap.com/download.epd?
context=40E2D9D5E00EEF7C6D06E4404E078BAD5C01554ECD2027EFC759D35F83100EEE&ei=t3tbSrOSCsjI-
QaQy5SBCQ&rct=j&q=SAP+benchmark+SAPS&usg=AFQjCNGNyIZqCbzYY5-HsKxeXCY8I9yK-A
http://www.oracle.com/corporate/pricing/pricelists.html
http://docs.hp.com/en/T2767-90141/index.html
http://forums11.itrc.hp.com/service/forums/questionanswer.do?
admit=109447626+1247514013348+28353475&threadId=1302823
http://h71028.www7.hp.com/enterprise/cache/258348-0-0-0-121.html
adaptive infrastructure http://h71028.www7.hp.com/enterprise/cache/483409-0-0-0-121.html
licensing http://h71028.www7.hp.com/enterprise/cache/262372-0-0-0-121.html
tco http://h20338.www2.hp.com/hpux11i/cache/324574-0-0-0-121.html
HPUX vs. linux http://h20338.www2.hp.com/hpux11i/cache/615059-0-0-0-121.html
http://www.microsoft.com/windowsserver2008/en/us/hyperv-faq.aspx
http://www.microsoft.com/presspass/press/2007/nov07/11-12HyperVPR.mspx
http://www.microsoft.com/windowsserver2008/en/us/licensing-itanium.asp
OpenVZ vs, Xen performance report:
http://slashdot.org/article.pl?sid=07/05/15/019251
www.parallels.com/virtuozzo, hp osip/osms (open source),
http://www.parallels.com/solutions/vdi/
http://www.parallels.com/partnernet/technology/company,11925
http://www.parallels.com/solutions/virtualization/itanium
OpenVZ: http://wiki.openvz.org/Main_Page
http://wiki.openvz.org/Download
http://download.openvz.org/openvz.repo http://download.openvz.org/current/
http://service.sap.com/solutionmanager
http://www.solutions2projects.com/pdfs/Good_Business_Practices_October%202006.pdf
SAP, Linux, Virtualization and ... Itanium Popović Zoran

26. http://h10076.www1.hp.com/education/curr-itsm.htm
http://technet.microsoft.com/en-us/library/cc305136(TechNet.10).aspx
http://www.itil.org.uk/ http://www.itil-officialsite.com/home/home.asp http://www.itil.co.uk/
http://en.wikipedia.org/wiki/Information_Technology_Infrastructure_Library
http://itsm.the-hamster.com/
SAP, Linux, Virtualization and ... Itanium Popović Zoran