Applications and the information they hold are increasingly the lifeblood of many organisations. In my experience, many businesses which have encountered major loss of data are never able to reopen; some attempt to but do not succeed and only a handful survive. This just goes to show the long-term importance of a business aligned DR strategy today.
For an enterprise-wide DR programme to be truly effective the IT requirements need to reflect business needs and the value of the assets that are being protected. Once an organisation can capture what assets they have, the external and internal risks they need to be protected from, and how to maintain the accuracy of those assets, an analysis of the impact of loss needs to be performed.
The understanding of ‘actual business needs’ and impact of loss can be determined through a Business Impact Analysis (BIA). This identifies how much downtime you can actually afford and therefore, what level of protection you can ‘get away with’. The amount of downtime an organisation can ‘really afford’ is directly related to the financial, legal or public relations impact an application has on the organisation in the event of its unavailability. During the BIA a pain point is established whereby the impact of the application unavailability significantly spikes. This will provide the time key which identifies the amount of downtime your organisation can afford.
These well known metrics are the Recovery Point Objective (RPO) and Recovery Time Objective (RTO), both of which have a direct impact on the amount of investment it takes to protect the application. The more aggressive these metrics are, the more expensive the infrastructure required to protect them. Getting this impact versus cost balance right is one of the biggest challenges facing organisations when implementing and maintaining DR capability.
Another challenge for businesses creating a DR strategy is that they have to decipher the tangible and intangible impacting factors and deal with emotive and unrealistic views on the impact of applications. For example, human nature dictates that ‘my’ application is the most important and so requires the highest level of protection. The consequence of that is huge expenditure required to ensure minimal downtime for applications that do not have the business impact to warrant such investment.
Until recently the complexity of determining the impacting factors has played a significant role in DR being reserved for only the most critical applications. Now, technological enhancements have meant the cost of delivering DR and recovery objectives has become cheaper and more commonplace, but effective understanding of application unavailability remains, at best, confusing.
A DR strategy is paramount for any organisation that operationally or legally relies on its applications and the information they hold. Without one the interruption of applications could spell disaster for you personally and or your organisation. Getting the balance right for downtime versus the level of protection you could get away with will result in an effective DR strategy that matches the value of the application to the cost of the infrastructure to protect it.
-Simon Johnson, GlassHouse Technologies (UK) Disaster Recovery Practice Lead
Disaster recovery (DR) is, by its very nature, difficult to plan for. But we’re all well aware of the problems associated with insufficient DR processes, policies and procedures. If a business’ IT infrastructure cannot recover from a ‘disaster’ quickly the implications can be extremely costly.
Recovery Time Objective (RTO) and Recovery Point Objective (RPO) are key measurements an IT manager needs to make the business aware of and provision for downtime accordingly. Both are recovery metrics that are calculated in time which provide quantifiable figures used to understand the tolerance levels of the business for application downtime and data loss.
RTO measures the maximum amount of time that is needed to recover from disruption and for the business to be operational again. The more aggressive your RTO, the shorter the critical time period to restore the system to normal functioning. This, inevitably means more financial investment is required in high availability infrastructure, but perhaps a small price to pay in the long run if something does go wrong. There are many technology options to consider including various clusters or complete redundant infrastructure and data replication on or offsite.
The RPO looks at the maximum amount of data loss acceptable in the event of a disruption. A business will ask itself “how much can we afford to lose”. For example, if there is a nightly backup at 21:00 and the system fails at 07:30 the following day, the system will have lost all data modifications since the backup at 21:00 the previous night. The question is – is that loss acceptable to the business?
Like RTO, the more aggressive the RPO, the greater the financial investment in infrastructure is required to meet the objective in a shorter period of time.
Some businesses - or areas within a business - may not be able to tolerate RTOs and RPOs of any longer than a few hours, while others may be able survive downtime for periods of, say a week with minimal impact. These requirements can normally be determined by the Service Level Agreements (SLAs).
For years businesses and their IT departments have struggled to understand and communicate effectively with each other, resulting in either significant under or over investment in both operational and disaster recovery application protection. Accurate RPO and RTO metrics have helped bridge this gap and, combined with business impact analysis, facilitate the alignment of applications to correct data protection levels and generate the accurate levels of investment to protect data.
SLAs are unachievable unless a business has the capabilities to deliver them. Organisations need to understand how and where data protection is delivered in order to optimise operations and meet the SLAs. Although they typically play significant roles, backups, snap shots and mirrors do not solely deliver RTOs and RPOs. Many levels of resilience throughout the IT supply chain combine to deliver recovery capabilities. These must all be accurately measured to generate the RPOs and RTOs. These quantifiable objectives translate requirements into tangible metrics which facilitate the selection of infrastructure to enable effective achievement of the SLAs, even in an unforeseen disaster situation.
-Simon Johnson, GlassHouse Technologies (UK) Disaster Recovery Practice Lead
Build an excel table and enter each of the business units along with the revenue and contribution (ie the contribution to the organization’s bottom line). Enter projected revenue and contribution as well. Now you have an empirical basis for ranking business units based on their revenue and/or contribution to the organization. BIA calculations often include not only lost sales and revenue, but costs of operating during the down time. These costs often include a per hour loss based on what it costs to maintain head count and other internal carrying costs during the disaster period. Enter head count and average carrying cost to your spreadsheet.
Costs based on lost customers also need to be taken into account. Add columns to calculate impact such as 5% loss of customers permanently, or perhaps 2% of the biggest customers are lost and 10% of the remaining customers too. The revenue loss for the major and minor customers can now be calculated into your model.
You now have the ability to run some calculations that indicate the impact of a disaster on each of these business units. The prudent DR planner will seek Finance Office assistance and guidance during this process. If you can get Financial Office buy in it makes life much easier for the prudent DR planner. Often times the calculated Business Impact will grow exponentially over time. The longer the outage, the more lost budiness.
Now you have the key elements to help you understand the cost to the organization for each business unit in the event of a disaster. This information allows you to align your DR plans with business impact using business numbers that have been blessed by the finance office.
By Dick Benton, Principal Consultant with GlassHouse Technologies Inc.
In previous blogs we covered the identification of policies and assumptions as DR planning foundations. The third foundation for effective DR planning is the empirically based Business Impact Analysis or BIA. The BIA refers to a process that determines the tangible cost to the organization should a business unit be unable to operate. This assessment provides the ROI for investing in DR mitigation. The relativity of the mitigation costs to an organization’s financial goals guides us in developing appropriate recovery objectives for the applications used by this business unit. The BIA is critical to the DR plan because it tells us how to determine what applications must be recovered first and how quickly those applications need to be recovered.
But what do you do if there is no budget and/or little interest in developing a formal BIA?. A good start is to visit the finance team and have them help in getting a handle on the revenue generated by each business unit last year. It is also important to get a handle on projected revenue for the coming year. Sometimes a business unit can come into being and generate an astonishing percentage of revenue from a new business line. Understanding the past and looking at how business units project the future is an essential part of the picture.
By Dick Benton, Principal Consultant with GlassHouse Technologies Inc.
to be continued…
Taking the time to identify and publish the many assumptions on which your DR plan is based is not only a key component for plan developers and stakeholders, but is also critical to your ability to survive the post disaster review.
Assumptions are the tactical policies of the DR Plan. These policies can have a critical impact on the plan itself, on the budget for the plan, and even the ability to actually execute the plan. They serve not only to define the scope and constraints of the plan, but more importantly, they set accurate expectations of all the stake holders. Everyone has 20/20 hindsight after a disaster as business units maintain (often accurately) they were unaware of key assumptions that affected their subsequent operational viability.
The assumptions made in planning disaster recovery often include the extent of the disaster, the survival of transportation infrastructure, the survival and subsequent availability of recovery staff as well as many others that will directly impact your plan and your DR architecture. By identifying and publishing these assumptions up front, senior management can review and signoff on the impact on the plan and the budget as a result of the stated assumptions. The assumptions identified and published become the Tactical Policies that provide the foundation philosophy for DR planning.
Transportation infrastructure survival, communications infrastructure survival, survival of the disaster site itself in a wider geographic disaster, are additional assumptions to think about. What about the business volumes after a disaster? .Will business volumes remain static, will they suddenly increase or decrease, can throughput rates change?
We have discussed only a few of many assumptions that are made, sometimes unknowingly, in development of the DR plan. Identifying and including these assumptions as Tactical Policies is key. These policies need to be exposed to the harsh daylight of senior management review before budgeting for or developing your DR plan objectives. It is critical not only to the ability of your organization to survive a disaster, but also to your own ability to survive the post disaster review; assuming you survived the disaster 
-Dick Benton, GlassHouse Principal Consultant
Knowing what is driving your DR plan helps you to be grounded in reality and take a pragmatic approach that fulfills the desires of the sponsoring bodies. Without this grounding you could embark on a quixotic quest to provide far more than the sponsors ever envisaged with consequent shock and awe as you present your first budget request. Alternatively, you might embark on a minimum effort in an earnest but misguided attempt to constrain cost and optimize time to deliver, only to find that the very basic approach you have undertaken will not meet the needs envisaged by the sponsors for a demonstrable recovery to full production.
What should be in a corporate policy on DR? While the actual policy wording will be highly situational and reflective of your own particular culture there are several key components you will want to include. These include the DR plan’s objectives, drivers, assumptions and responsibilities.
Set up preliminary discussions, seek ideas from each executive, offer draft statements. This will demonstrate your ability to seek executive guidance while capturing key drivers, needs, and assumptions for the policy development. Gain additional participation in the drafting of this policy by having discussion sessions one level down with key managers. Offer as a basis for the discussions potential “bullet points” or “draft phrases” for each component of the policy. Typically, at some point in the creative process, the executive team will adopt the policy development, take ownership, refine and publish the policy
Now you have the “legal” foundation for your project and a clear and compelling statement of objectives and the drivers behind the objectives for a significant corporate investment and effort. But the job is not yet done. Now you need to make sure that everyone knows about this. Executives may fail to communicate this policy effectively unless prompted, and you will need to follow up and help the various parties understand the implications of this policy, the drivers behind it, and how this will impact each of them and their role in either infrastructure support or application usage as the DR plan is developed.
- Dick Benton, GlassHouse Principal Consultant
For the dedicated technologist, DR planning can be fun. New technologies, new approaches, vendor presentations, free lunches, new skill sets, and the opportunity to demonstrate once again just how much the business units really need IT (and you). It is all very exciting in an otherwise dull world of operational trench warfare. The temptation to plunge into this pool of excitement is often overwhelming. Resist the temptation. A little creative up front work to properly set the scene will have a big payback down the road.
Step one is to ensure that you, your team, the business units and your CFO all know, understand, agree, and buy in to the disaster scenarios your plan will mitigate and the assumptions on which it is based. This means the development of a corporate policy statement that sets the goals and objectives for disaster recovery at a level that crosses all organizational boundaries. In the absence of such specificity, assumptions can be wildly at variance with the reality of the funded approach. Taking the time now, to document and communicate the disaster recovery goals and objectives as official corporate policy is key. It is key to a corporate understanding of your objectives, it is key to the budget you will require for the project, and it is key for the business units to commit to the constraints such scenarios and assumptions may have on their operational readiness in a post disaster situation.
- Dick Benton, GlassHouse Principal Consultant