[Open-Bioinformatics-Foundation] Notice: New OBF CVS server and disruptive infrastructure upgrades

[Chris Dagdigian]

##
To avoid active OBF members getting many copies of this message we have 
constructed an announce-only unique email list culled from all the 
discussion lists that we operate. This email is going out to 1982 
unique addresses. Questions about how/why you received this message 
should be addressed to 'chris@bioteam.net'.
##


To all OBF affiliated developers and mailing list subscribers as well 
as related parties like I3C.org, biopathways.org and emboss.org:

Subject:    Significant server downtime beginning on October 2nd

Contents:

(1) Summary for non-developers
(2) Background info & history
(3) Details for developers and powerusers

OBF Developers, I3C.org members & emboss.org CVS users:
There is critical information that you must be aware of in the body of 
this message.


Summary for non-developers
=======================

On Wednesday, October 2, 2002 (all day - EDT timezone) we are going to 
relocate all of our servers including the primary web, email, FTP and 
sourcecode repositories to a new (and faster) network. This move 
requires changing all of our server IP addresses and physically moving 
the machine to a building a short distance away from our current 
colocation facilities located in Cambridge, Massachusetts.  We are also 
going to take advantage of this relocation to move our primary CVS 
repositories on to a dedicated developer-access-only machine (details 
below).

The server maintenance and physical relocation are expected to only 
take a few hours. However it may take up to 48 hours for our new DNS 
information to propagate around the world and replace the old 
information that may be aggressively cached by institutions and ISPs.

For many of you the primary symptom will be a disruption in email 
discussion lists that you may be participating in.

A good sign that the change has propagated to your network will is 
being able to resolve the 'bioperl.org' domain name to something other 
than 199.93.107.70

The reminder of this message is background material and specific 
information for developers and account holders on our system.

We are asking in advance for forgiveness for any inconvenience this 
rushed-to-avoid-disconnection transition will cause. There may be 
issues beyond our control (Wyeth routers/firewall, reverse IP mappings 
with upstream ISPs, domain names we don't control DNS for, etc.) that 
could cause us minor problems over the next few days. Please hang in 
there!

During the downtime/transition process Chris Dagdigian can be reached 
directly at 1-617-877-5498 and 'chris@bioteam.net'. Feel free to 
contact him at any time with questions, problem reports or requests for 
status updates. Once email is flowing again you can also contact the 
OBF sysadmin team at 'root-l@open-bio.org'.



Background & History
=================

One of our longest and most significant supporters has been Genetics 
Institute (now integrated into Wyeth Research) -- for many years now 
they have been allowing us to hang our servers off of their corporate 
internet connection in Cambridge, MA.

Over the years this has been a non-trivial gift as the combined OBF 
websites (particularly bioperl.org and biojava.org) consume a very 
significant amount of internet bandwidth every day. More bandwidth than 
we could probably afford to pay for, especially in the early days of 
our project(s).

The connection we have been using for 4+ years is based on a redundant 
pair of T1 lines connected to the Genuity/BBN backbone.  This was the 
primary corporate internet connection for Wyeth Research in Cambridge 
until several months ago when the company brought in a new OC3 fiber 
line offering 12mbit/second internet connectivity.

After the company moved its internet usage over to the OC3 line someone 
at Wyeth began the process of canceling the T1 pair without the 
knowledge that there were multiple outside-the-firewall servers still 
dependent on that link.

Those of you curious as to why our servers fell off the internet for 
several days a few weeks ago now know the answer as to what happened. 
Our internet circuit was shut off by the carrier on orders from Wyeth 
network admins who thought the lines were unused.

The disconnect caused an immediate panic within our sysadmin core -- we 
knew what happened but were not at all sure if we were still welcome at 
Wyeth Research anymore. We thought that we were going to be forced to 
find a new home for our systems with zero notice.

Fortunately that was not the case.

Once Wyeth Research was alerted to our problem there were many 
employees on both the scientific and IT side who interrupted  their 
busy schedules and went above and beyond their normal duties to (a) 
bring the T1 circuit back temporarily and (b) work with senior IT staff 
on a transition plan to bring our servers into an outside-the-firewall 
managed DMZ zone on the new really fast OC3 internet link.

This is a partial list of people at Wyeth Research who helped us out:

Charles Richard III -- Vice President, Genomics
Steve Howes --- Director of Bioinformatics, Genomics
Steve Tenerelli --- Associate Director, Border Network Services
Rich Hollenbach -- Firewall Administrator, Border Network Services
Jason Blair -- IT Network Engineer
Rich DiNunno -- IT Network Engineer

Without the efforts of these and other folks we'd probably still be 
frantically looking for a new hosting facility at this time.


Specifics for developers and account holders
===================================

(A) CRITICAL INFO FOR DEVELOPERS

We are going to take advantage of this planned downtime to bring a new 
developer server online. You will not be able to CVS commit to the 
current server after 11am EDT on October 2nd 2002.

The new CVS server will be called "dev.open-bio.org" and will be a Sun 
Netra T1 server running Solaris 8 and directly attached to a large 
external RAID storage array. As usual we will create DNS entries for 
"dev.your-project-here.org".

DEV.OPEN-BIO.ORG will be for developers only and will be the new home 
for ALL CVS SOURCECODE REPOSITORIES!

Anonymous CVS for non-developers will still be at 'cvs.open-bio.org' 
and 'cvs.your-project-here.org'

The main reason for this move is to get our developers onto a platform 
with hardware RAID protected storage. The additional benefit is that 
the system will not be encumbered by any of our other web or email 
services.

How the transition will be managed:

(1) The CVS repository on the current main server everyone uses will be 
made read-only at 11am EDT on Wednesday October 2nd 2002.

(2) Shortly after we make the repository unwritable we will disable 
inbound SSH access to the primary server so new connections or commits 
will not be possible.

(3) The locked-down CVS repositories will be copied onto 
dev.open-bio.org along with all user accounts, UIDs, GIDs, passwords 
and home directories.

(4) The new machine will be moved to the new OC3 network and brought 
online as 'dev.open-bio.org'. Additional DNS aliases will follow

(5) At the same time the CVS migration occurs we will be updating all 
of out other servers and physically moving them to the new hosting 
network.


The path of least resistance for developers is to:

(a) get your code changes committed prior to the lockdown deadline

(b) perform a clean checkout of the codebase from dev.open-bio.org when 
it comes up and continue future work on it.

However it is highly likely that developers may have changes that they 
will not be able or willing to commit prior to the lockdown deadline. 
For those users the following example script has been suggested:

% find . -name "Root" ! -type d | xargs perl -p -i .orig -e 
's/ext:(\S+):/ext:dev.open-bio.org:/

That unix shell command (untested) should put the proper 
dev.open-bio.org entries into any existing CVS checkout directories 
that you may have. If you are nervous about performing such an 
operation then please wait until our servers are back online again and 
query the OBF sysadmin mailing list ("rool-l@open-bio.org") to see if 
someone can provide a script that has been used and tested by others.

For more information:

Since our email lists will be down if at any time you need more 
information or updates on our progress please feel free to contact me 
directly at 'chris@bioteam.net' or 1-617-877-5498,



(B) Domains that will be directly affected by our relocation

We control DNS for the following domains and will be in a position to 
make quick changes when the IP transition occurs:

biodas.org
biofetch.org
biojava.org
biomoby.org
bioperl.org
biopython.org
bior.org
biosoap.org
biostandards.org
bioxml.org
open-bio.org

We do not control DNS for the following domains:

i3c.org -- They will need to update DNS information themselves or face 
disruption

biopathways.org -- Not yet hosting on our servers so not a problem

emboss.org -- They only use us for CVS but need to be aware of the new 
hostname for the cvs server