Performance Issue: Network Outage Affecting Many Campus IT Services
Last Updated:
2019-11-08 00:38:54
Event:
2019-11-07 15:22:00
Status:
Closed
Brief Description:
An issue in the Rhodes and CCC server farms is affecting many campus IT services, potentially including Office 365 login, Pinnacle, authentication, Tableau, OBIEE, on-campus hosting, Student Essentials, video conferencing,VPN, SFS, Net-Print, and others.
User Impact:
Unavailability or performance issues with many campus IT services.
Workaround:
There is no workaround for this issue
Current Status:
All services appear to be operating normally at this time. Cornell staff are continuing to monitor performance and are investigating the cause of the problem to help reduce the chance of similar issues in the future.
Services Affected:
Network
Full Description:
An issue in the Rhodes Hall and CCC server farms is affecting many campus IT services, potentially including Pinnacle, authentication, Tableau, OBIEE, on-campus hosting, Student Essentials, SIP/H323 video conferencing, VPN, and others. Cornell staff are investigating.
If you are having difficulties getting Office 365 email through one device, it may be due to your credentials expiring on that device. If you have another device or application you use to check your email, it might still be connected.
Following is a list of services that could potentially be affected by this issue. Availability and performance may vary.
Student Essentials
Cornell Directory (LDAP)
CUWebLogin
Office 365 (ADFS) login
AuthSMTP
AppSMTP
Shared File Services
Net-Print
PeopleSoft
Cornell Dining (some systems)
Cornell Housing (some systems)
Facilities (some systems)
CUView signs
SharePoint On Premise
NetID management app
Academic Web Hosting
CUVPN
Tableau
Jira
Confluence
Cisco video conferencing
Shared File Services
Managed Servers
Pinnacle
Perceptive Content (ImageNow)
ArXiv.org
Data warehouses
If you are having difficulties getting Office 365 email through one device, it may be due to your credentials expiring on that device. If you have another device or application you use to check your email, it might still be connected.
Following is a list of services that could potentially be affected by this issue. Availability and performance may vary.
Student Essentials
Cornell Directory (LDAP)
CUWebLogin
Office 365 (ADFS) login
AuthSMTP
AppSMTP
Shared File Services
Net-Print
PeopleSoft
Cornell Dining (some systems)
Cornell Housing (some systems)
Facilities (some systems)
CUView signs
SharePoint On Premise
NetID management app
Academic Web Hosting
CUVPN
Tableau
Jira
Confluence
Cisco video conferencing
Shared File Services
Managed Servers
Pinnacle
Perceptive Content (ImageNow)
ArXiv.org
Data warehouses
Timeline of Changes
| Description | Current Status | Date | Time |
|---|---|---|---|
| An issue in the Rhodes Hall and CCC server farms is affecting many campus IT services, potentially including Pinnacle, authentication, Tableau, OBIEE, on-campus hosting, Student Essentials, SIP/H323 video conferencing, VPN, and others. Cornell staff are investigating. If you are having difficulties getting Office 365 email through one device, it may be due to your credentials expiring on that device. If you have another device or application you use to check your email, it might still be connected. Following is a list of services that could potentially be affected by this issue. Availability and performance may vary. Student Essentials Cornell Directory (LDAP) CUWebLogin Office 365 (ADFS) login AuthSMTP AppSMTP Shared File Services Net-Print PeopleSoft Cornell Dining (some systems) Cornell Housing (some systems) Facilities (some systems) CUView signs SharePoint On Premise NetID management app Academic Web Hosting CUVPN Tableau Jira Confluence Cisco video conferencing Shared File Services Managed Servers Pinnacle Perceptive Content (ImageNow) ArXiv.org Data warehouses | All services appear to be operating normally at this time. Cornell staff are continuing to monitor performance and are investigating the cause of the problem to help reduce the chance of similar issues in the future. | 2019-11-08 | 00:38:54 |
| An issue in the Rhodes Hall and CCC server farms is affecting many campus IT services, potentially including Pinnacle, authentication, Tableau, OBIEE, on-campus hosting, Student Essentials, SIP/H323 video conferencing, VPN, and others. Cornell staff are investigating. If you are having difficulties getting Office 365 email through one device, it may be due to your credentials expiring on that device. If you have another device or application you use to check your email, it might still be connected. Following is a list of services that could potentially be affected by this issue. Availability and performance may vary. Student Essentials Cornell Directory (LDAP) CUWebLogin Office 365 (ADFS) login AuthSMTP AppSMTP Shared File Services Net-Print PeopleSoft Cornell Dining (some systems) Cornell Housing (some systems) Facilities (some systems) CUView signs SharePoint On Premise NetID management app Academic Web Hosting CUVPN Tableau Jira Confluence Cisco video conferencing Shared File Services Managed Servers Pinnacle Perceptive Content (ImageNow) ArXiv.org Data warehouses | Network services have been restored to the server farms. Some IT services may still be experiencing performance issues as Cornell staff continue to resolve secondary effects of the problem. | 2019-11-08 | 00:03:17 |
| An issue in the Rhodes Hall and CCC server farms is affecting many campus IT services, potentially including Pinnacle, authentication, Tableau, OBIEE, on-campus hosting, Student Essentials, SIP/H323 video conferencing, VPN, and others. Cornell staff are investigating. If you are having difficulties getting Office 365 email through one device, it may be due to your credentials expiring on that device. If you have another device or application you use to check your email, it might still be connected. Following is a list of services that could potentially be affected by this issue. Availability and performance may vary. Student Essentials Cornell Directory (LDAP) CUWebLogin Office 365 (ADFS) login AuthSMTP AppSMTP Shared File Services Net-Print PeopleSoft Cornell Dining (some systems) Cornell Housing (some systems) Facilities (some systems) CUView signs SharePoint On Premise NetID management app Academic Web Hosting CUVPN Tableau Jira Confluence Cisco video conferencing Shared File Services Managed Servers Pinnacle Perceptive Content (ImageNow) ArXiv.org Data warehouses | A series of router restarts did not resolve the problem. CIT network engineers are continuing to work into the evening in conjunction with the router vendor to try and identify the cause of the problem and resolve it. | 2019-11-07 | 22:11:05 |
| An issue in the Rhodes Hall and CCC server farms is affecting many campus IT services, potentially including Pinnacle, authentication, Tableau, OBIEE, on-campus hosting, Student Essentials, SIP/H323 video conferencing, VPN, and others. Cornell staff are investigating. If you are having difficulties getting Office 365 email through one device, it may be due to your credentials expiring on that device. If you have another device or application you use to check your email, it might still be connected. Following is a list of services that could potentially be affected by this issue. Availability and performance may vary. Student Essentials Cornell Directory (LDAP) CUWebLogin Office 365 (ADFS) login AuthSMTP AppSMTP Shared File Services Net-Print PeopleSoft Cornell Dining (some systems) Cornell Housing (some systems) Facilities (some systems) CUView signs SharePoint On Premise NetID management app Academic Web Hosting CUVPN Tableau Jira Confluence Cisco video conferencing Shared File Services Managed Servers Pinnacle Perceptive Content (ImageNow) ArXiv.org Data warehouses | The router restart restored some affected services, but some services continue to be unavailable. Cornell network staff are looking into the reasons for the services not coming back online. | 2019-11-07 | 20:54:51 |
| An issue in the Rhodes Hall and CCC server farms is affecting many campus IT services, potentially including Pinnacle, authentication, Tableau, OBIEE, on-campus hosting, Student Essentials, SIP/H323 video conferencing, VPN, and others. Cornell staff are investigating. If you are having difficulties getting Office 365 email through one device, it may be due to your credentials expiring on that device. If you have another device or application you use to check your email, it might still be connected. Following is a list of services that could potentially be affected by this issue. Availability and performance may vary. Student Essentials Cornell Directory (LDAP) CUWebLogin Office 365 (ADFS) login AuthSMTP AppSMTP Shared File Services Net-Print PeopleSoft Cornell Dining (some systems) Cornell Housing (some systems) Facilities (some systems) CUView signs SharePoint On Premise NetID management app Academic Web Hosting CUVPN Tableau Jira Confluence Cisco video conferencing Shared File Services Managed Servers Pinnacle Perceptive Content (ImageNow) ArXiv.org Data warehouses | Server farm routers will shut down completely from 3:15 to 3:30, at which time services should start coming back online. There may be some built up demand on restart, causing performance slowdowns. CIT networking staff will be monitoring. Networking between Cornell and the broader internet should not be affected, just the services that have had performance issues or outages today. | 2019-11-07 | 20:09:31 |
| An issue in the Rhodes Hall and CCC server farms is affecting many campus IT services, potentially including Pinnacle, authentication, Tableau, OBIEE, on-campus hosting, Student Essentials, SIP/H323 video conferencing, VPN, and others. Cornell staff are investigating. If you are having difficulties getting Office 365 email through one device, it may be due to your credentials expiring on that device. If you have another device or application you use to check your email, it might still be connected. Following is a list of services that could potentially be affected by this issue. Availability and performance may vary. Student Essentials Cornell Directory (LDAP) CUWebLogin Office 365 (ADFS) login AuthSMTP AppSMTP Shared File Services Net-Print PeopleSoft Cornell Dining (some systems) Cornell Housing (some systems) Facilities (some systems) CUView signs SharePoint On Premise NetID management app Academic Web Hosting CUVPN Tableau Jira Confluence Cisco video conferencing Shared File Services Managed Servers Pinnacle Perceptive Content (ImageNow) ArXiv.org Data warehouses | Cornell has all available networking staff committed to working on the problem, and the router vendor is handling as a severity 1 issue. They are in constant contact and engaged in troubleshooting the problem. | 2019-11-07 | 19:45:09 |
| An issue in the Rhodes Hall and CCC server farms is affecting many campus IT services, potentially including Pinnacle, authentication, Tableau, OBIEE, on-campus hosting, Student Essentials, SIP/H323 video conferencing, VPN, and others. Cornell staff are investigating. | CIT and the router vendor are on a call and are continuing to examine network traffic in order to identify the root cause of the problem. | 2019-11-07 | 18:38:41 |
| An issue in the Rhodes Hall and CCC server farms is affecting many campus IT services, potentially including Pinnacle, authentication, Tableau, OBIEE, on-campus hosting, Student Essentials, SIP/H323 video conferencing, VPN, and others. Cornell staff are investigating. | CIT has contacted the router vendor. Vendor and CIT staff are continuing to investigate. | 2019-11-07 | 18:04:35 |
| An issue in the Rhodes Hall and CCC server farms is affecting many campus IT services, potentially including Pinnacle, authentication, Tableau, OBIEE, on-campus hosting, Student Essentials, SIP/H323 video conferencing, and others. Cornell staff are investigating. | CIT is continuing to investigate the root cause of the problem. | 2019-11-07 | 17:52:13 |
| An issue in the Rhodes Hall server farm is affecting many campus IT services, potentially including Pinnacle, authentication, Tableau, OBIEE, on-campus hosting, Student Essentials, and others. Cornell staff are investigating. | CIT is currently investigating. | 2019-11-07 | 16:06:34 |
| There appears to be a Network Outage affecting the Rhodes Hall Serverfarm. | CIT is currently investigating. | 2019-11-07 | 15:26:56 |