Skip to main content

Unplanned Outage: IT services restored after power failure

Last Updated:
2010-12-03 05:00:00
Event:
2010-12-01 05:00:00
Status:
Closed
Brief Description:
User Impact:
N/A
Workaround:
There is no workaround for this issue
Current Status:
N/A
Services Affected:
Full Description:
At this time, CIT believes that nearly all IT services affected by the Dec. 1-2 overnight power failure have been restored. Please report any ongoing problems to the NOC at 255-9900.
CIT TDX ID:


Timeline of Changes

Description Current Status Date Time
IT services have been restored. Please report any ongoing problems to the NOC at 255-9900.\n IT services have been restored. Please report any ongoing problems to the NOC at 255-9900.\n 2010-12-03 05:00:00
Many of the university's IT services were affected by the campus power outage that occurred at approximately 10 pm on December 1 and lasted for a few hours. CIT has restored many IT services at this point, and is continuing to work on the rest.\n\nThe following services are operational at this time:\n\nAuthusersmtp\nBlackBerry Enterprise Server Express\nBlackboard\nColdFusion 7 (but some databases may not be)\nColdFusion 9\nCommonSpot\nConfluence\nCornell web site (www.cornell.edu)\nCUFS\nCUWebAuth\nE-list service (Lyris) - backlog of messages being sent\nEmail routing\nExchange\nEZ-Backup\nEZ-Print\nGoogle Search Appliance\nJIRA\nkProxy\nKronos\nLAMP\nMainframe\nMODCOMP\nNetwork - campus and WAN\nOrpheus Cluster SERVERS\nPermit Server (Bridge)\nPhone System\nRadius\nRedRover Secure\nRemedy\nSourceForge\nVPN\nWebDAV\n\nThe outage was caused by a power failure that affected much of central campus, followed by a subsequent loss of the backup power supply for the university's main server room. As soon as power was restored to this room at approximately 1:15 am, CIT began working to restore all affected IT services as quickly as possible. Many of the university's IT services were affected by the campus power outage that occurred at approximately 10 pm on December 1 and lasted for a few hours. CIT has restored many IT services at this point, and is continuing to work on the rest.\n\nThe following services are operational at this time:\n\nAuthusersmtp\nBlackBerry Enterprise Server Express\nBlackboard\nColdFusion 7 (but some databases may not be)\nColdFusion 9\nCommonSpot\nConfluence\nCornell web site (www.cornell.edu)\nCUFS\nCUWebAuth\nE-list service (Lyris) - backlog of messages being sent\nEmail routing\nExchange\nEZ-Backup\nEZ-Print\nGoogle Search Appliance\nJIRA\nkProxy\nKronos\nLAMP\nMainframe\nMODCOMP\nNetwork - campus and WAN\nOrpheus Cluster SERVERS\nPermit Server (Bridge)\nPhone System\nRadius\nRedRover Secure\nRemedy\nSourceForge\nVPN\nWebDAV\n\nThe outage was caused by a power failure that affected much of central campus, followed by a subsequent loss of the backup power supply for the university's main server room. As soon as power was restored to this room at approximately 1:15 am, CIT began working to restore all affected IT services as quickly as possible. 2010-12-02 05:00:00
Note: EZ-Backup did not experience an outage. If any restores are needed, please contact ezbackup@cornell.edu\n\nThe following services are operational at this time:\n\nASDT-Time\nAuthusersmtp\nBlackBerry Enterprise Server Express\nBlackboard\nCATS\nColdFusion 7 (but some databases may not be)\nColdFusion 9\nCommonSpot\nConfluence\nCoeus - prod/test\nCornell web site (www.cornell.edu)\nCUFS\nCUWebAuth\nCynergy - prod/stage\nE-list service (Lyris) - backlog of messages being sent\nEmail routing\nExchange\nEZ-Backup\nEZ-Print\nFedora - dev/test/prod\nFormScape\nGoogle Search Appliance\nJIRA\nkProxy\nKronos\nLAMP\nMainframe\nMODCOMP\nNetwork - campus and WAN\nNUBB\nOrpheus Cluster SERVERS\nPermit Server (Bridge)\nPhone System\nRadius\nRedRover Secure\nRemedy\nSourceForge\nStudent Jobs\nuPortal.Cornell\nVPN\nWebDAV\nWebMethods - dev/prod Note: EZ-Backup did not experience an outage. If any restores are needed, please contact ezbackup@cornell.edu\n\nThe following services are operational at this time:\n\nASDT-Time\nAuthusersmtp\nBlackBerry Enterprise Server Express\nBlackboard\nCATS\nColdFusion 7 (but some databases may not be)\nColdFusion 9\nCommonSpot\nConfluence\nCoeus - prod/test\nCornell web site (www.cornell.edu)\nCUFS\nCUWebAuth\nCynergy - prod/stage\nE-list service (Lyris) - backlog of messages being sent\nEmail routing\nExchange\nEZ-Backup\nEZ-Print\nFedora - dev/test/prod\nFormScape\nGoogle Search Appliance\nJIRA\nkProxy\nKronos\nLAMP\nMainframe\nMODCOMP\nNetwork - campus and WAN\nNUBB\nOrpheus Cluster SERVERS\nPermit Server (Bridge)\nPhone System\nRadius\nRedRover Secure\nRemedy\nSourceForge\nStudent Jobs\nuPortal.Cornell\nVPN\nWebDAV\nWebMethods - dev/prod 2010-12-02 05:00:00
Nearly all IT services have been restored. Please report ongoing problems to the NOC at 255-9900.\n\nA statement from Facilities Services regarding the power failure is below:\n\n-------------\nFrom: Special Conditions \nTo: "SpecialConditions-L@cornell.edu" \nDate: Thu, 2 Dec 2010 10:52:53 -0500\nSubject: Repairs to Campus Electric Distribution System Underway\n\nA circuit failed at about 10:30 PM Wednesday evening, December 1, 2010 causing a power outage that affected most of the endowed campus, Weill Hall, West Campus and Wilson Lab. The Cornell University line crews isolated the problem and restored power at about 1:00 AM, Thursday December 2, 2010.\n\nThe transfer switch for the emergency generator at Rhodes Hall failed to operate. This caused the CIT server farm to go down.\n\nRepairs to the failed circuit and transfer switch are occurring today. We expect no interruption to power.\n\nFacility coordinators in areas of campus not affected by the outage may wish to check facilities or equipment.\n\nPlease contact the Facilities Customer Service Center at 255-5322 if you have any questions or concerns. Nearly all IT services have been restored. Please report ongoing problems to the NOC at 255-9900.\n\nA statement from Facilities Services regarding the power failure is below:\n\n-------------\nFrom: Special Conditions \nTo: "SpecialConditions-L@cornell.edu" \nDate: Thu, 2 Dec 2010 10:52:53 -0500\nSubject: Repairs to Campus Electric Distribution System Underway\n\nA circuit failed at about 10:30 PM Wednesday evening, December 1, 2010 causing a power outage that affected most of the endowed campus, Weill Hall, West Campus and Wilson Lab. The Cornell University line crews isolated the problem and restored power at about 1:00 AM, Thursday December 2, 2010.\n\nThe transfer switch for the emergency generator at Rhodes Hall failed to operate. This caused the CIT server farm to go down.\n\nRepairs to the failed circuit and transfer switch are occurring today. We expect no interruption to power.\n\nFacility coordinators in areas of campus not affected by the outage may wish to check facilities or equipment.\n\nPlease contact the Facilities Customer Service Center at 255-5322 if you have any questions or concerns. 2010-12-02 05:00:00