Unplanned Outage: NYC circuits down between Syracuse and NYC
Last Updated:
2012-05-10 04:00:00
Event:
2012-05-09 04:00:00
Status:
Closed
Brief Description:
User Impact:
N/A
Workaround:
There is no workaround for this issue
Current Status:
N/A
Services Affected:
Full Description:
Both Cornell's 10Gbps Ethernet circuits between Syracuse and New York City. The circuits broke at 07:43 EDT. Our circuit providers confirm the outage, no estimated repair time has been established.
CIT TDX ID:
Timeline of Changes
| Description | Current Status | Date | Time |
|---|---|---|---|
| Level3 continues to make repairs to fiber in Tuxedo, NY that was damaged yesterday by DOT construction activities. One of Cornell's 10G bps Ethernet circuits have been temporarily restored, but the other 10 Gbps circuit is not repaired yet. There may be further outages as repairs continue.\n\nCornell has moved Cogent ISP service to the one operation circuit. This means Cornell has full redundancy and capacity to Internet 1 (commodity Internet). Also, other services in NYC are now reachable (including the NLR .edu backbone), albeit with no confirmed completion of fiber repairs. We continue to monitor the situation.\n\nHere's the update to fiber repairs (06:55 EDT)\nTransport NOC is attempting to escalate this matter with the vendor at this time as the 90 thread fiber has not been spliced yet. Also we are still awaiting the field technicians arrival onsite which is about 1 hour.\n\nPer Level 3: Transport NOC report the fiber splicing has slowed due to the fact the vendor had their splice crews resplice fiber in a previous pit in order to correct a light level issue they have on their network. Transport NOC report that the splicing of the remaining Level 3 fiber has yet to be completed as it is a part of this last 90 strand. Still no ETA\n | Level3 continues to make repairs to fiber in Tuxedo, NY that was damaged yesterday by DOT construction activities. One of Cornell's 10G bps Ethernet circuits have been temporarily restored, but the other 10 Gbps circuit is not repaired yet. There may be further outages as repairs continue.\n\nCornell has moved Cogent ISP service to the one operation circuit. This means Cornell has full redundancy and capacity to Internet 1 (commodity Internet). Also, other services in NYC are now reachable (including the NLR .edu backbone), albeit with no confirmed completion of fiber repairs. We continue to monitor the situation.\n\nHere's the update to fiber repairs (06:55 EDT)\nTransport NOC is attempting to escalate this matter with the vendor at this time as the 90 thread fiber has not been spliced yet. Also we are still awaiting the field technicians arrival onsite which is about 1 hour.\n\nPer Level 3: Transport NOC report the fiber splicing has slowed due to the fact the vendor had their splice crews resplice fiber in a previous pit in order to correct a light level issue they have on their network. Transport NOC report that the splicing of the remaining Level 3 fiber has yet to be completed as it is a part of this last 90 strand. Still no ETA\n | 2012-05-10 | 04:00:00 |
| The ORIE office at 55 Broad Street in NYC was moved to the one operational circuit. Full connectivity is now restored to this site. Once repairs to the other circuit are complete (no ETA) we will move this connection back during a maintenance window. | The ORIE office at 55 Broad Street in NYC was moved to the one operational circuit. Full connectivity is now restored to this site. Once repairs to the other circuit are complete (no ETA) we will move this connection back during a maintenance window. | 2012-05-10 | 04:00:00 |
| Fiber circuits between Ithaca and New York City (NYC) have been repaired. All Cornell circuits and services are operational.\n\nThough all service are operating completely, some temporary configurations to patch around the outage will be restored tomorrow morning during maintenance windows.\n\n\nUpdate from NYSERNet:\n\nLevel3 has reported that all of the fiber splicing is now complete, although there's still work being done to clean up the site. Our network is fully restored, with all optical connections back online and traffic for the R&E IP network once again flowing along the normal paths.\n\nUpdate from NLR:\n\nDESCRIPTION: All alarms have been cleared since 16:46 PDT, TUE 05/09. \nLevel 3 just confirmed that all fiber\nsplicing has been completed. No further outages are expected at this time.\n | Fiber circuits between Ithaca and New York City (NYC) have been repaired. All Cornell circuits and services are operational.\n\nThough all service are operating completely, some temporary configurations to patch around the outage will be restored tomorrow morning during maintenance windows.\n\n\nUpdate from NYSERNet:\n\nLevel3 has reported that all of the fiber splicing is now complete, although there's still work being done to clean up the site. Our network is fully restored, with all optical connections back online and traffic for the R&E IP network once again flowing along the normal paths.\n\nUpdate from NLR:\n\nDESCRIPTION: All alarms have been cleared since 16:46 PDT, TUE 05/09. \nLevel 3 just confirmed that all fiber\nsplicing has been completed. No further outages are expected at this time.\n | 2012-05-10 | 04:00:00 |
| The following services are connected via 32 AoA and nyc1-6500:\n\n Cogent ISP : Cogent ISP (Internet peering at 10GE)\n NLR Peering : NLR PacketNet and NLR FrameNet.\n Weill Medical School private peering : Weill Medical College private Internet peering (New York and Qatar).\n CIT off-site tape backup : Remote Tape collection at Weill Medical,\n575 Lexington Avenue,NY, NY\n AzzurroHD video feed : Video feeds from Cornell to production studios via AzzurroHD.\n ORIE remote office, NY, NY : ORIE remote site at 55 Broad Street, NY, NY\n\nNYSERNet sees an apparent fiber break between New Paltz and Suffern NY. \nNLR confirms a Layer-2 problem as well.\n | The following services are connected via 32 AoA and nyc1-6500:\n\n Cogent ISP : Cogent ISP (Internet peering at 10GE)\n NLR Peering : NLR PacketNet and NLR FrameNet.\n Weill Medical School private peering : Weill Medical College private Internet peering (New York and Qatar).\n CIT off-site tape backup : Remote Tape collection at Weill Medical,\n575 Lexington Avenue,NY, NY\n AzzurroHD video feed : Video feeds from Cornell to production studios via AzzurroHD.\n ORIE remote office, NY, NY : ORIE remote site at 55 Broad Street, NY, NY\n\nNYSERNet sees an apparent fiber break between New Paltz and Suffern NY. \nNLR confirms a Layer-2 problem as well.\n | 2012-05-09 | 04:00:00 |
| Level 3 Field Services has confirmed that fiber damage resulted from third party construction being performed to install a new overpass bridge. A steel pole was being set, which damaged the vendors conduit. The duct is 18 feet underground, which will require excavation to uncover for repair. Escalation to the second level with the conduit vendor for repair; however, there is no estimated time of arrival for fiber restoration crews. More information will be given as it becomes available. | Level 3 Field Services has confirmed that fiber damage resulted from third party construction being performed to install a new overpass bridge. A steel pole was being set, which damaged the vendors conduit. The duct is 18 feet underground, which will require excavation to uncover for repair. Escalation to the second level with the conduit vendor for repair; however, there is no estimated time of arrival for fiber restoration crews. More information will be given as it becomes available. | 2012-05-09 | 04:00:00 |
| Level 3 Field Services has confirmed that excavation is completed in the south pit. Splicing crews are working to prepare fiber in the south pit for splicing. Excavation continues to expose cable in the north pit. There is no estimated time available for excavation to be completed and no estimated time for splicing to begin. It has been confirmed that spare cable has arrived onsite.\n***********************************************************\n17:59 GMT – Excavation has begun. There is no estimated time for completion of excavation. Spare cable is en route to the site, with an estimated time of arrival of 20 minutes. Splicing crews remain on standby for excavation to complete. Once complete splice crews will begin cable repair.\n | Level 3 Field Services has confirmed that excavation is completed in the south pit. Splicing crews are working to prepare fiber in the south pit for splicing. Excavation continues to expose cable in the north pit. There is no estimated time available for excavation to be completed and no estimated time for splicing to begin. It has been confirmed that spare cable has arrived onsite.\n***********************************************************\n17:59 GMT – Excavation has begun. There is no estimated time for completion of excavation. Spare cable is en route to the site, with an estimated time of arrival of 20 minutes. Splicing crews remain on standby for excavation to complete. Once complete splice crews will begin cable repair.\n | 2012-05-09 | 04:00:00 |
| The Transport NOC received a progress report from the conduit vendor who advised that splicing is ongoing at the south pit, and has just begun at the north pit. The 120-count fiber will be repaired first and then the splicing crew will proceed to the 96-count fiber. The Transport NOC requested an ETTR, but the conduit vendor was unable to provide one at this time. The Transport NOC will continue to monitor alarms on the Level 3 network to watch for restoration, and will provide information as it becomes available from the vendor.\n | The Transport NOC received a progress report from the conduit vendor who advised that splicing is ongoing at the south pit, and has just begun at the north pit. The 120-count fiber will be repaired first and then the splicing crew will proceed to the 96-count fiber. The Transport NOC requested an ETTR, but the conduit vendor was unable to provide one at this time. The Transport NOC will continue to monitor alarms on the Level 3 network to watch for restoration, and will provide information as it becomes available from the vendor.\n | 2012-05-09 | 04:00:00 |
| There have been some continued difficulties getting to some of the affected fibers, but this is the latest update available:\n\nTransport NOC report the Level3 fiber pit has been completed and they have pulled the fiber into it and they have pulled the fiber into the first pit for prepping and splicing. The field technician report an ETTR of 05:00 GMT.\n | There have been some continued difficulties getting to some of the affected fibers, but this is the latest update available:\n\nTransport NOC report the Level3 fiber pit has been completed and they have pulled the fiber into it and they have pulled the fiber into the first pit for prepping and splicing. The field technician report an ETTR of 05:00 GMT.\n | 2012-05-09 | 04:00:00 |