Skip to main content

Unplanned Outage: e-list service

Last Updated:
2010-11-30 05:00:00
Event:
2010-11-29 05:00:00
Status:
Closed
Brief Description:
User Impact:
N/A
Workaround:
There is no workaround for this issue
Current Status:
N/A
Services Affected:
Full Description:
CIT has received reports that messages being sent to some e-lists are not being received. It appears that there is an issue with one of our e-lists servers at this time.
CIT TDX ID:



Timeline of Changes

Description Current Status Date Time
At the current time all elists are processing email at normal speeds ( < 10 min).\nThe logs reveal that the slowdown problem affected only @cornell.edu email addresses, starting around 2010 Nov 29 07:00, ending around 19:00.\n\nThe problem was triggered by an upgrade to sendmail on the 'elist "@cornell.edu" outgoing gateway' (tulip).\nInadvertently and incorrectly the upgrade included a 4 sec startup delay for each SMTP connection, normally used to manage mail from botnets. Unfortunately this triggered a bug in the elist server which effectively limited delivery to @cornell.edu addresses to 9000 recipients per hour.\n\nAlthough the upgrade was performed at 2010 Nov 29 06:30, problems did not really set in until 07:30 because before that the volume of elist traffic was low. The sendmail problem was identified and fixed around 13:50. Most of the delayed email was processed by 16:30. All of it by 19:00.\nAbout 50% of the @cornell.edu email experienced delays of more than 10 min. About half of that experienced more that 1.5 hr delay and some messages experienced more than 3 hr delay. \n\n At the current time all elists are processing email at normal speeds ( < 10 min).\nThe logs reveal that the slowdown problem affected only @cornell.edu email addresses, starting around 2010 Nov 29 07:00, ending around 19:00.\n\nThe problem was triggered by an upgrade to sendmail on the 'elist "@cornell.edu" outgoing gateway' (tulip).\nInadvertently and incorrectly the upgrade included a 4 sec startup delay for each SMTP connection, normally used to manage mail from botnets. Unfortunately this triggered a bug in the elist server which effectively limited delivery to @cornell.edu addresses to 9000 recipients per hour.\n\nAlthough the upgrade was performed at 2010 Nov 29 06:30, problems did not really set in until 07:30 because before that the volume of elist traffic was low. The sendmail problem was identified and fixed around 13:50. Most of the delayed email was processed by 16:30. All of it by 19:00.\nAbout 50% of the @cornell.edu email experienced delays of more than 10 min. About half of that experienced more that 1.5 hr delay and some messages experienced more than 3 hr delay. \n\n 2010-11-30 05:00:00
We are currently investigating this problem and will notify you with updates on this situation. We are currently investigating this problem and will notify you with updates on this situation. 2010-11-29 05:00:00
The main e-list server is up and running but users are experiencing mail delays for mail going through e-lists. We are still investigating this problem and will notify you with further updates. The main e-list server is up and running but users are experiencing mail delays for mail going through e-lists. We are still investigating this problem and will notify you with further updates. 2010-11-29 05:00:00
CIT staff are still investigating the e-list mail delays. CIT staff are still investigating the e-list mail delays. 2010-11-29 05:00:00
CIT staff report the backlog of messages is clearing,\nthey are actively monitoring the problem at this time.\nThey are still investigating to discover the cause.\nThank you for your patience. CIT staff report the backlog of messages is clearing,\nthey are actively monitoring the problem at this time.\nThey are still investigating to discover the cause.\nThank you for your patience. 2010-11-29 05:00:00