Skip to main content

Unplanned Outage: Connectivity from Ithaca campus to NYC connected sites is down - Fiber

Last Updated:
2014-04-01 00:00:00
Event:
2014-03-31 00:00:00
Status:
Closed
Brief Description:
User Impact:
N/A
Workaround:
There is no workaround for this issue
Current Status:
N/A
Services Affected:
Full Description:
Connectivity from Ithaca campus to NYC connected sites is down due a fiber cut in the Suffern area. These services are connected via 32 AoA and nyc1-6500:\n • Weill Medical College private Internet peering (New York and Qatar)\n • CIT off-site tape backup: Remote tape collection at Weill Medical\n • Video feeds from Cornell to production studios via AzzurroHD\n • ORIE remote site at 55 Broad Street, NYC\n • Cornell Tech (NYC)\n • NYSERNet/Internet2 backup 10GE peering



Timeline of Changes

Description Current Status Date Time
The Transport NOC advised that the excavation has completed and the damaged fibers have been exposed. It has also been advised that the fiber provider's splice crew is currently splicing an additional damaged cable that does not carry Level 3 traffic. Upon completion, the splice crew will begin repairs of the 120-count fiber. An estimated time of restoral is not available at this time. The Transport NOC advised that the excavation has completed and the damaged fibers have been exposed. It has also been advised that the fiber provider's splice crew is currently splicing an additional damaged cable that does not carry Level 3 traffic. Upon completion, the splice crew will begin repairs of the 120-count fiber. An estimated time of restoral is not available at this time. 2014-04-01 00:00:00
The Transport NOC advised that the splicing of the additional damaged cable continues. In attempt to complete repairs quickly, four splice crews are at the site working on the damaged fibers. An estimated time of restoral is not available at this time. The Transport NOC advised that the splicing of the additional damaged cable continues. In attempt to complete repairs quickly, four splice crews are at the site working on the damaged fibers. An estimated time of restoral is not available at this time. 2014-04-01 00:00:00
Level3 is reporting that the crews are prepping the 120 count cable containing their fiber now and are expecting to start the splicing at 08:00. Level3 is reporting that the crews are prepping the 120 count cable containing their fiber now and are expecting to start the splicing at 08:00. 2014-04-01 00:00:00
The Transport NOC has advised that the additional damaged cable has been fully repaired and the restoration cable for the 120-count fiber has been set in the conduit. The Transport NOC is currently attempting to obtain an estimated time of restoral. The fiber was cut somewhere along the Hudson down to NYC. The Thruway Authority was repairing a guardrail and accidentally cut the fiber. CIT has most of the phones back up at this time. The Transport NOC has advised that the additional damaged cable has been fully repaired and the restoration cable for the 120-count fiber has been set in the conduit. The Transport NOC is currently attempting to obtain an estimated time of restoral. The fiber was cut somewhere along the Hudson down to NYC. The Thruway Authority was repairing a guardrail and accidentally cut the fiber. CIT has most of the phones back up at this time. 2014-04-01 00:00:00
The fiber provider has advised that an entire section of the 120 count cable has to be replaced. Splicing is ongoing on one end of the cable and the other end of the cable is currently being prepped for splicing. No estimated time of restoral has been provided from the provider as of yet, but once available, it will be communicated. The fiber provider has advised that an entire section of the 120 count cable has to be replaced. Splicing is ongoing on one end of the cable and the other end of the cable is currently being prepped for splicing. No estimated time of restoral has been provided from the provider as of yet, but once available, it will be communicated. 2014-04-01 00:00:00
The Transport NOC reports that splicing is now underway on both ends of the cable. No estimated time of restoral has been provided as of yet. The Level 3 field technician at the site is working to obtain an estimated restoral time. The Transport NOC reports that splicing is now underway on both ends of the cable. No estimated time of restoral has been provided as of yet. The Level 3 field technician at the site is working to obtain an estimated restoral time. 2014-04-01 00:00:00
The fiber is partially repaired. Testing is being done at this time. Further updates will be provided when information becomes available. The fiber is partially repaired. Testing is being done at this time. Further updates will be provided when information becomes available. 2014-04-01 00:00:00
The fiber cable repair is complete but cleanup and testing continues. The final OK has not yet been received. The fiber cable repair is complete but cleanup and testing continues. The final OK has not yet been received. 2014-04-01 00:00:00
Our monitoring is clean and traffic is flowing. \n Our monitoring is clean and traffic is flowing. \n 2014-04-01 00:00:00
There is no ETR at this time. Updates will be posted when available. There is no ETR at this time. Updates will be posted when available. 2014-03-31 00:00:00
Level 3 has been engaged and technicians are in route to fix the fiber cut but still no ETR at this time. Level 3 has been engaged and technicians are in route to fix the fiber cut but still no ETR at this time. 2014-03-31 00:00:00
Level 3 has isolated the fiber cut and has reached out to the underlying vendor. We still do not have an ETR at this time. Level 3 has isolated the fiber cut and has reached out to the underlying vendor. We still do not have an ETR at this time. 2014-03-31 00:00:00
Technicians are working on the fiber cut but there is no ETR yet. Technicians are working on the fiber cut but there is no ETR yet. 2014-03-31 00:00:00
The fiber provider has reported that their field technicians are currently on site at two adjacent locations working to isolate the exact break point, but until the fault is completely isolated there is no ETR. The fiber provider has reported that their field technicians are currently on site at two adjacent locations working to isolate the exact break point, but until the fault is completely isolated there is no ETR. 2014-03-31 00:00:00
The fiber provider has advised that the fault has been isolated to a fiber section along a bridge. Splice crews have been engaged to assist with locating the physical damage, and are currently en route to the area. An estimated arrival time is not yet available. The next update is planned for 6AM. The fiber provider has advised that the fault has been isolated to a fiber section along a bridge. Splice crews have been engaged to assist with locating the physical damage, and are currently en route to the area. An estimated arrival time is not yet available. The next update is planned for 6AM. 2014-03-31 00:00:00