21. Q & A
Post questions to:
https://vdocs.community.veritas.com/sites/kls/Lists/ TrainingQuestions
By using the link “⊕ new item”
Hinweis der Redaktion
Your systems hold the information and data that drives your businesses. Question to the audience - Without those systems how many people in the audience would be able to sustain business? Get the audience to start to talk about how important systems are in their organization.
Without adequate planning and preparation, system downtime can compromise a return to operations and cause financial damage as the ability to generate revenue is lost. Most organizations anticipate that unplanned downtime can — and likely will — occur at some future time. But, in today's world, the need to ensure Web-enabled 24 x 7 x 365 access to key applications is paramount. Thus planning for operational continuity and recovery from outages is rapidly becoming an urgent priority for commercial and government entities alike.
Manual server recovery is too complex. Manual server recovery can be a time-consuming and tedious process. Typically, manual recovery includes rebuilding a server by reinstalling the operating system, rebooting several times throughout the recovery process, reconfiguring the system, loading the tape backup software, and hoping that no errors have occurred along the way. This process, which can take hours or even days, generally exceeds the capabilities of the average small business.
For most midsize businesses with Windows-based environments and multiple locations, backups take place at each individual site with separate servers, tape systems, and dedicated staff. If one of the individual locations experiences a system failure and thus data loss, the backup tapes are manually returned and the data restored to the local servers. Returning those tapes takes time.
Simply put, manual disaster recovery methods is any size of organization are time consuming and error prone. It also hinders the ability of an organization to meet recovery time objectives set by senior management. Why is this important? As mentioned above, downtime can compromise a return to operations, cause disruption and financial losses.
Changing nature of IT environment. Traditionally, disaster recovery was aimed primarily at the central-site datacenter; today, it is becoming necessary to preserve business continuity for computing resources that may be located at a variety of locations in a networked computing environment. Organizations need the ability to recover systems and data quickly and efficiently regardless of where those systems reside. In addition, the IT landscape has dramatically transformed with the adoption of virtualization and cloud technology. A single host now contains multiple virtual servers. If a single host fails, the disruption is more severe as multiple servers fail at the same time.
The cost of downtime includes tangible/direct costs such as lost transaction revenue, lost wages, lost inventory, remedial labour costs, marketing costs, bank fees and legal penalties from not delivering on service level agreements, and intangible/indirect costs including lost business opportunities, loss of employees and/or employee morale, decrease in stock value, loss of customer/partner goodwill, brand damage, driving business to competitors or even bad publicity/press.
Interesting Article for supplementary information:
How Much Does Downtime Really Cost?
InfoManagement Direct, August 6, 2009
Henry Martinez
Why is it difficult to justify an investment in business resilience when the risk and costs of downtime are so high? One reason is because the current business environment insists on clearly demonstrated financial returns on investment. The other reason is because the risks and costs are not well understood.
Despite advances in infrastructure robustness, occasional hardware, software and database downtime is unavoidable. In fact, Dunn & Bradstreet reports that 49 percent of Fortune 500 companies experience at least 1.6 hours of downtime per week. That translates into more than 80 hours annually. How would your customers react to an 80-hour outage?
Understanding Downtime Costs
There are options for reducing total downtime, in some cases almost to zero, but none of them are free. Hence, organizations must determine whether these options are worth the cost given the financial impact of downtime. Put differently, each organization needs to decide what losses it is willing to accept due to outages and design a solution to avoid this worst case. Average downtime costs vary considerably across industries, from approximately $90,000 per hour in the media sector to about $6.48 million per hour for large online brokerages. Downtime costs also vary significantly within industries. Business size is the most obvious factor, but it is not the only one. For instance, companies that can revert to manual processing can continue to function when their systems are unavailable, although usually at an appreciably lower level of activity. In contrast, some companies, such as online retailers, cannot conduct any business during system downtime. Similarly, certain manufacturing businesses must destroy all of the work in progress (such as food and pharmaceuticals).
To gain a firm understanding of the financial impact of downtime, examine the various subcomponents of downtime costs: revenue, human resources, regulatory and compliance, remedial and reputation impact.
Revenue Impact
Almost all companies now use IT to record sales. Some organizations, such as low-volume, high-value manufacturers, may still be able to take and fulfill orders manually. However, most retailers and high-volume manufacturers can no longer complete the sales process when their systems are unavailable. This is especially true for Web-only stores whose entire order-taking capability stops when their systems are offline.
A company’s average sales per hour, adjusted to account for the percentage of sales that can and likely will be completed manually during a system outage, can be used to estimate the revenue that will be lost per hour of downtime. However, this yields only a very rough approximation.
Not all of the sales that would have otherwise occurred during a downtime event will be permanently lost. Some persistent customers will wait for their preferred vendor’s systems to come back online rather than switching to a competitor.
If an outage creates a disruption in a supply chain with a high level of expectation in responsiveness (i.e., medical services or overnight delivery), the business may be exposed to damages. Often, damages stem from the inability to deliver (i.e., loss in delivery fees due to arriving late or lawsuits due to collateral damages). These highly publicized situations can impact shareholder value.
On the other hand, the value lost when loyal (or potentially loyal) customers do switch to a competitor is more than just the value of the purchases that would have been made at that time. It amounts to the lifetime purchases those customers might have made.
Human Resource Impact
Another significant downtime cost is lost employee productivity, which can be measured in terms of the salaries, wages and benefits of idled people. If, for example, the work of 1,000 people earning an average of $60 per hour, including benefits, depends on a system that becomes unavailable for one hour, the value of the lost productivity can be estimated as $60,000 for that hour. The estimate might be somewhat lower if some employees are still able to do a portion of their work manually or switch to tasks that require only systems that are still available. Nonetheless, many organizations have become so dependent on IT that an unavailable system totally idles a significant portion of the enterprise, when the old manual processes often no longer exist. For instance, What happens when email goes down? What happens to that new cost-saving voice-over Internet protocol system when the LAN goes down?
Subject to the above caveats, the following is a quick formula for estimating hourly downtime labor costs:
Hourly Labor Cost = P x A x C
Where:
P = number of people affected
A = average percentage they are affected
C = average employee cost (salaries or wages + benefits)
Regulatory and Contract Compliance Impact
Increasingly stringent regulations require organizations to safeguard the availability, reliability and privacy of financial, human resources and customer data. In addition to government regulations, many companies now include service level agreements in the contracts they sign with their customers and business partners. Failure to fulfill the conditions of an SLA can lead to the imposition of substantial penalties. In addition, the regulatory and contractual fines that may result from an inability to perform business functions vary widely, but can be substantial.
Goal – communicate the average cost of system downtime – it’s different for every organization.
According to Dunn & Bradstreet, 59% of Fortune 500 companies experience a minimum of 1.6 hours of downtime per week. To put this in perspective, assume that an average Fortune 500 company has 10,000 employees who are paid an average of $56 per hour, including benefits ($40 per hour salary + $16 per hour in benefits). Just the labour component of downtime costs for such a company would be $896,000 weekly, which translates into more than $46 million per year.
$3,000 per day:
It costs small businesses a median of $3,000 per day
$84,000 - $108,000 per hour:
On average, businesses lose between $84,000 and $108,000 (US) for every hour of IT system downtime, according to estimates from studies and surveys performed by IT industry analyst firms. In addition, financial services, telecommunications, manufacturing and energy lead the list of industries with a high rate of revenue loss during IT downtime. Downtime costs vary not only by industry, but by the scale of business operations. For a medium-sized business, the exact hourly cost may be lower, but the impact on the business may be proportionally much larger. While idled labour and lower productivity costs may seem to be the most substantial cost of downtime, any true cost of downtime estimate should include the value of the opportunities that were lost when the applications were not available. For example, consider a company that averages a gross profit margin of $100,000 per hour from Web and telemarketing sales. If its order-processing systems crash for an hour, making it impossible to take orders, what is the cost of the outage? The easy, but erroneous, answer would be $100,000. Some customers will be persistent and call or click back at another time. Sales to them are not lost; cash flow is simply delayed. However, some prospects and customers will give up and go to a competitor.
Let me give you an overview of how this product works.
First, with Veritas System Recovery, you can protect any Windows system – a server, desktop, or laptop.
With VSR, you can take a full backup of the system that includes the OS, applications, configurations, settings and data and store that in a Recovery Point (also known as a backup). The Recovery Point can be taken while the system is online and active or “hot.” You do not need to take the server, desktop, or laptop offline. And you can even schedule when Recovery Points are taken. VSR also lets you schedule full or incremental recovery points.
This recovery point can be stored on any disk storage device such as NAS or SAN, iSCSI, SATA, a USB or Firewire external drive, flash memory or even burn the Recovery Point to a CD or DVD. (does not support tape)
Then once you have that recovery point, VSR also enables you to copy that recovery point to a secondary location including public cloud, an FTP server for enhanced disaster recovery capabilities.
Then when you experience a system failure, recovery is simple. You take the recovery point and restore system back to that point in time. Via Recovery Disk. You insert the Recovery DVD in the system you want to restore. And this recovery, it auto detects the hardware attached to the system, and automatically loads the critical system drivers that are needed for recovery. Then all you have to do is select which recovery point you want to restore to, and VSR will restore that system back to that exact point in time – without the need to manually reinstall and reconfigure anything. (only restore entire system or data -files and folders or granular items – not applications)
But also with VSR, if you lose specific data or information, you also have the ability to recover individual files or folder, or other business critical information.
But that is not all. VSR also provide that same lightning fast recovery when recovering to dissimilar hardware – a different server, desktop, or laptop. It’s the exact same process whether you are recovering to same hardware or dissimilar hardware.
In addition, VSR also lets you convert a physical Recovery Point to a virtual environment. This is especially useful if a physical server is not available to restore to.
And you can also convert that virtual environment back to physical again using System Recovery.
Flexible offsite protection helps your disaster recovery planning. Let’s talk about how it works.
Take a recovery point or backup of the protected system and store that at the primary destination. After it’s stored. In the wizard you can select at up to two offsite destinations. Take a copy and store it to.
Benefit – in addition to primary storage destination you now have a copy for enhanced disaster recovery should a disaster happen at the primary destination. Scenario – power outage. Use secondary copy to get backup and running quickly.
The System Recovery LightsOut Restore feature enables administrators to restore a computer from a remote location, regardless of the state the computer is currently in, so long as its file system is intact. Note: The LightsOut feature requires at least 1 gigabyte of memory to run.
Depending on your hardware configuration, you can use LightsOut Restore to complete a system restoration on a remote server via a Web browser, using your server's remote connection capabilities, and the Veritas recovery environment. By using this option, you save the time it takes to physically visit the computer to perform the restore.
When LightsOut Restore is set up, it installs a recovery environment directly to the file system on the system partition, and places a recovery environment boot option within the Windows boot menu. Whenever this boot menu option is selected, the system will boot directly to the recovery environment using the files installed on the system partition.
Once LightsOut Restore has been set up and the boot menu option has been added, you can use a hardware device to remotely connect to the system. When you are connected, you can power on or reboot the system and select the recovery
environment from the boot menu. The system then boots to the recovery environment.
In this example we have the IT department based at the head office. If the server goes out a remote location we can perform a recovery without it staff at that site. The great thing about VSR if that the IT admin can perform that restore from any location as long as he has access that remote site. It could be that office or it could be at home.
Not only do P2V and V2P conversions dramatically reduce system downtime they also reduce management time and set-up when you schedule them in advance.
Need to recover to dissimilar hardware and to virtual environments. Recovering to dissimilar hardware and to new virtual environments is essential to effective system protection. The ability to virtually store and restore server images allows companies to reduce the total numbers of servers in their IT environment. With the average cost of servers ranging from $15,000 to $30,000 for the hardware, software, and maintenance, this capability can generate substantial savings.
Overall with System Recovery’s Granular Recovery Technology, you have the ability to recover granular items from a single pass backup including:
Files
Folders
Exchange email, folders, mailboxes or attachments
Pg 6, item 2.7: Introduced custom port functionality to change port dynamically from manager. Client port number can be managed by port.txt file in client machine – NEED PATH
Same launch point as previous Management Solution
Need to verify 64-bit
Installs a named instance: VSR Manager
Can configure hourly refresh interval
System Requirements from SR 21.x Admin Guide: https://www.veritas.com/content/support/en_US/doc/38007533-139298167-0/v35813850-139298167