Performance Issue: Rhodes Hall Server Farm
Last Updated:
2019-08-09 14:17:10
Event:
2019-08-08 23:58:00
Status:
Closed
Brief Description:
The power outage that affected parts of the Ithaca Campus resulted in an unplanned outage of the Rhodes Hall server farm, which many Cornell IT services depend on. Most services are available again. Cornell staff are continuing to monitor.
User Impact:
Some Cornell IT services that run on equipment in the Rhodes Hall server farm were unavailable as a result of a power outage.
Workaround:
There is no workaround for this issue
Current Status:
Monitoring and testing is indicating that all services that were affected by the power outage are available and operating normally. Among the restored services are MBAM, which required a manual restart of a component this morning, as well as DirecTV and Cisco video conferencing infrastructure. Cornell staff are continuing to watch the service operations and work on issues that were related to the outage.
Services Affected:
Servers
Subsites Affected:
PrinterLogic
Video on Demand
Full Description:
The power outage that affected parts of the Ithaca Campus resulted in an unplanned outage of the Rhodes Hall server farm, which many Cornell IT services depend on. Most services are available again. Cornell staff are continuing to monitor and resolve remaining issues.
Timeline of Changes
| Description | Current Status | Date | Time |
|---|---|---|---|
| The power outage that affected parts of the Ithaca Campus resulted in an unplanned outage of the Rhodes Hall server farm, which many Cornell IT services depend on. Most services are available again. Cornell staff are continuing to monitor and resolve remaining issues. | Monitoring and testing is indicating that all services that were affected by the power outage are available and operating normally. Among the restored services are MBAM, which required a manual restart of a component this morning, as well as DirecTV and Cisco video conferencing infrastructure. Cornell staff are continuing to watch the service operations and work on issues that were related to the outage. | 2019-08-09 | 14:17:10 |
| The power outage that affected parts of the Ithaca Campus resulted in an unplanned outage of the Rhodes Hall server farm, which many Cornell IT services depend on. Most services are available again. Cornell staff are continuing to monitor and resolve remaining issues. | Most services that were affected by the power outage are available and operating normally. Cornell staff are continuing to monitor systems and are resolving remaining performance issues. | 2019-08-09 | 10:18:14 |
| The power outage affecting parts of the Ithaca Campus resulted in an unplanned outage of the Rhodes Hall server farm, which many Cornell IT services depend on. Cornell staff are working on the issue. | Most services that were affected by the power outage are available and operating normally. Cornell staff are continuing to monitor systems as they stabilize and are resolving remaining performance issues. While staff are working to ensure services stay up, a recurrence of NYSEG issues could again interrupt service availability. | 2019-08-09 | 03:35:17 |
| The power outage affecting parts of the Ithaca Campus resulted in an unplanned outage of the Rhodes Hall server farm, which many Cornell IT services depend on. Cornell staff are working on the issue. | In addition to services previously announced as coming online, Lyris e-list, Jira, NetID password changes, and Perceptive Content are again available. Note: The issues that initially caused the problem are still present and could potentially result in resumed outages. | 2019-08-09 | 03:07:06 |
| The power outage affecting parts of the Ithaca Campus resulted in an unplanned outage of the Rhodes Hall server farm, which many Cornell IT services depend on. Cornell staff are working on the issue. | CIT is continuing the process of bringing Rhodes Hall services back online. Cornell.edu, Academic Web Hosting, and some other Rhodes-hosted websites, Cornell Dropbox, Pinnacle, and Shared File Services (SFS) are again available, and other services are beginning to come back online. Performance may be slow in some cases as multiple services are resuming at the same time. Lyris e-lists are not yet available. | 2019-08-09 | 02:34:46 |
| The power outage currently affecting parts of the Ithaca Campus has resulted in an unplanned outage of the Rhodes Hall server farm, which many Cornell IT services depend on. Currently this outage is making cornell.edu unreachable. More services that are affected will be listed as information becomes available. Cornell staff are working on the issue. For more information and updates about the power outage, see https://emergency.cornell.edu | CIT is continuing the process of bringing Rhodes Hall services back online. The storage farm is being powered up and CIT staff are checking the status, then when this is complete, work will begin on bringing databases online, followed by the servers. CIT is working toward service availability before start of business tomorrow, so long as the process isn't disrupted by outside factors. Note: The issues that initially caused the problem are still present and could potentially result in resumed outages. Shared File Services (SFS) are again available. Affected services include Cornell.edu, Alertus emergency notifications, Lyris e-list, Perceptive Content, Cornell Dropbox, Cisco Video Conferencing, DirecTV, Pinnacle, Academic Web Hosting, Net-Print, ArXiv.org and NetID password changes or activations via netid.cornell.edu. Tonight's data warehouse loads--KDW and PS student and CR--will not occur. | 2019-08-09 | 01:56:58 |
| The power outage currently affecting parts of the Ithaca Campus has resulted in an unplanned outage of the Rhodes Hall server farm, which many Cornell IT services depend on. Currently this outage is making cornell.edu unreachable. More services that are affected will be listed as information becomes available. Cornell staff are working on the issue. For more information and updates about the power outage, see https://emergency.cornell.edu | CIT is beginning the process of bringing Rhodes Hall services back online. The storage farm will be brought back first, then databases, followed by the servers. Afterward the process of bringing servers back up will begin. The issues that initially caused the problem are still present and could result in resumed outages. Affected services include Cornell.edu, Lyris e-list, Perceptive Content, Cornell Dropbox, Cisco Video Conferencing, DirecTV, Pinnacle, Academic Web Hosting, Net-Print, ArXiv.org and Shared File Services. Tonight's data warehouse loads--KDW and PS student and CR--will not occur. | 2019-08-09 | 01:16:17 |
| The power outage currently affecting parts of the Ithaca Campus has resulted in an unplanned outage of the Rhodes Hall server farm, which many Cornell IT services depend on. Currently this outage is making cornell.edu unreachable. More services that are affected will be listed as information becomes available. Cornell staff are working on the issue. For more information and updates about the power outage, see https://emergency.cornell.edu | Numerous services are affected by the outage in Rhodes Hall. In addition to cornell.edu, these include Lyris e-list, Perceptive Content, Cornell Dropbox, and some logins may be slow. If you need to get a phone number and can't through Cornell search, try using a People search at http://outlook.cornell.edu More services are likely affected and will be added as information is available. Cornell staff are continuing to work on restoring services. | 2019-08-09 | 00:43:30 |
| The power outage currently affecting parts of the Ithaca Campus has resulted in an unplanned outage of the Rhodes Hall server farm, which many Cornell IT services depend on. Currently this outage is making cornell.edu unreachable. More services that are affected will be listed as information becomes available. Cornell staff are working on the issue. For more information and updates about the power outage, see https://emergency.cornell.edu | Cornell staff are working on the issue. | 2019-08-09 | 00:06:01 |