Disaster Recovery Site

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Threaded View

I hope that this is an acceptable topic for this ng.  I would
appreciate knowing if there is another one that is more
appropriate; I couldn't find one.

We are a not-for-profit, primarily involved in clinical trials.
We have reasonably good facilities to deal with emergiencies
(diesel generator, backup cooling, independent internet
links, etc), we do not have a disaster recovery site.   The
reality is that we have only one application that is highly
sensitive to downtime, and that one can be performed
manually.  We have the procedures to do switch over to
that at any time.  But if we were to be out for more than
three days, the problems would begin to build up.   And
we do have a large number of applications that are in
many cases unique to individual clients, and many of
these should be made available if we were to face a
long down time of our primary site.

We started work to establish a disaster recovery site
 at a sister office approx. 1,000 miles away.  After considering
all of the costs, we have come around to considering the
possibility of  establishing the site in a building in our office
complex several hundred yards away from ours and to which
we plan to expand in the next year.  There would be a number
of advantages to setting up the disaster site in that building
from our point of view, both in terms of functionality and cost.

We have tried to ennumerate and assign a probability to
all of the possible events we could imagine that would bring
down the primary site  and the secondary site at the same
time.  Our conclusion was that the chances of the latter were
extremely small, with one exception:  an area-wide  communi-
cations failure.  We thought of several possible solutions,
but the most cost-effective would be backup sattelite service.
This of course would be a low-bandwith solution but it appears
to us to be a functionally exceptable one, even if we do not
put users on rationed access times.   If both sites were in
fact destroyed we would have very serious problems until
recovery, but we would have offsite backups that would be
up to date, so ultimately we would be able to recover data-
bases to a point very close  to the failure point.

The questions that we have are 1) is this a practice followed
by any other organizations, and 2), do people believe that it
is defensible on the grounds I layed out above ?   Any input
you would give me will be very much appreciated by me.

Thank you !    

Re: Disaster Recovery Site


Quoted text here. Click to load it

 Sorry, I royally mucked up some sentences, especially the
 above:  "acceptable," not "exceptable."

Re: Disaster Recovery Site

Did you look into services such as this?:


pavlov wrote:

Quoted text here. Click to load it

Re: Disaster Recovery Site

On Sat, 26 Mar 2005 15:52:26 -0600, "Ralph A. Jones"

Quoted text here. Click to load it

 Yes, and that solution is still a possibility: we have pricing
 from several vendors for various configurations and
 capacities.  But we believe that the homegrown solution
 can be more cost-effective for us, especially if we can
 accomplish a few things that we are exploring now.

 Thank you for the input.

Re: Disaster Recovery Site

On Sat, 26 Mar 2005 18:16:18 -0500, pavlov wrote:
Quoted text here. Click to load it

Why not just setup a fiber connection between buildings, add a couple
servers to the domain, put them in the second building, replicate files on
a nightly (or quicker if needed) basis between servers, and be happy.

remove 999 in order to email me

Re: Disaster Recovery Site

Quoted text here. Click to load it

 That is a simplified version of what we have in mind.  My
 concern is whether the lack of any real geographic
 separation will be held against us.

Re: Disaster Recovery Site

On Sat, 26 Mar 2005 21:16:56 -0500, pavlov wrote:
Quoted text here. Click to load it

If the building is in the same earth-quake area, close enough for a fire
to spread around the compound, EMP, Power loss in a grid, Flooding, etc...
If you can't afford to be down for XX hours, then you need to move the
your backup center to another location.

As a side note, we designed a medical center with remote offices, all data
is central to the main office compound. Backups are several layers, but
final backup is to tape - we contract with an off-site storage place that
sends a person to pick up tapes and return the prior ones each day. Some
locations don't do any off-site backup, just have a UPS setup that's
capable of 24 hours service, but their remote offices backup to the main
office nightly.

If your process/data is critical enough that you can't be down for 1 day
in the event of a complete disaster, then you want the remote recovery
location to be in another city where none of the items above can impact
both sites at the same time from the same event.

remove 999 in order to email me

Re: Disaster Recovery Site

pavlov wrote:
Quoted text here. Click to load it

spot on

and you expressed your problem clearly as well, but I snipped it for brevity

I am sure it will raise a lot of debate, so here is my take....and you
get what you pay for :)

Many of my clients are not-for-profits/charities
Most of them didn't even have backups
I come from a background of consultancy and network security before
doing my own thing

so, from there...

You seem to be doing the right things, have a brainstorming session and
get all the risks and consequences down on paper. I'd call this a
FMEACA, failure mode, effects, and consequences analysis. Make sure you
get (very) senior manager buy in at this stage.

Then put a probability on each failure, and a nominal cost. No more than
five categories for each, say; likely, maybe, unlikely; expensive,
moderate, cheap with a final column relating risk & consequence - high,
medium, low (say) or just a nominal cost by multiplying risk and
consequence factors

draw up a table

put in costs to mitigate each failure mode and a VERY brief outline of a
plan (not more than 5 bullet points)

get it agreed by your manaegement team (it's their company)

Then get everyone to have a meeting and thrash out how much cash they
want to spend on the total plan. Then allocate your spending (on a 5
year budget) to go from highest to lowest total risk.

If anything is missed, you have a committee to sack not just you :) You
can't do this on your own, you need your company to back you to the hilt
on it and get their buy-in

Oh, it's hard to get non-profits/charities to actually spend money on
this kind of stuff, even getting backups made is often painful


Re: Disaster Recovery Site

On Sat, 26 Mar 2005 22:46:26 +0000 (UTC), Martin

Quoted text here. Click to load it

 It sounds fine to me.  I don't really have to worry very much about
senior management, but if I'm to look beyond the functional utility of
what you propose to the political utility, it appears that if we
follow through as you suggest, we should be on solid ground if an
auditor of some sort should pop in.  Or if we have a single event that
destroys both sites :-)

 Thanks for the thoughtful response.

Site Timeline