Unplanned Outage: e-list service
Last Updated:
2010-11-30 05:00:00
Event:
2010-11-29 05:00:00
Status:
Closed
Brief Description:
User Impact:
N/A
Workaround:
There is no workaround for this issue
Current Status:
N/A
Services Affected:
Full Description:
CIT has received reports that messages being sent to some e-lists are not being received. It appears that there is an issue with one of our e-lists servers at this time.
CIT TDX ID:
Timeline of Changes
| Description | Current Status | Date | Time |
|---|---|---|---|
| At the current time all elists are processing email at normal speeds ( < 10 min).\nThe logs reveal that the slowdown problem affected only @cornell.edu email addresses, starting around 2010 Nov 29 07:00, ending around 19:00.\n\nThe problem was triggered by an upgrade to sendmail on the 'elist "@cornell.edu" outgoing gateway' (tulip).\nInadvertently and incorrectly the upgrade included a 4 sec startup delay for each SMTP connection, normally used to manage mail from botnets. Unfortunately this triggered a bug in the elist server which effectively limited delivery to @cornell.edu addresses to 9000 recipients per hour.\n\nAlthough the upgrade was performed at 2010 Nov 29 06:30, problems did not really set in until 07:30 because before that the volume of elist traffic was low. The sendmail problem was identified and fixed around 13:50. Most of the delayed email was processed by 16:30. All of it by 19:00.\nAbout 50% of the @cornell.edu email experienced delays of more than 10 min. About half of that experienced more that 1.5 hr delay and some messages experienced more than 3 hr delay. \n\n | At the current time all elists are processing email at normal speeds ( < 10 min).\nThe logs reveal that the slowdown problem affected only @cornell.edu email addresses, starting around 2010 Nov 29 07:00, ending around 19:00.\n\nThe problem was triggered by an upgrade to sendmail on the 'elist "@cornell.edu" outgoing gateway' (tulip).\nInadvertently and incorrectly the upgrade included a 4 sec startup delay for each SMTP connection, normally used to manage mail from botnets. Unfortunately this triggered a bug in the elist server which effectively limited delivery to @cornell.edu addresses to 9000 recipients per hour.\n\nAlthough the upgrade was performed at 2010 Nov 29 06:30, problems did not really set in until 07:30 because before that the volume of elist traffic was low. The sendmail problem was identified and fixed around 13:50. Most of the delayed email was processed by 16:30. All of it by 19:00.\nAbout 50% of the @cornell.edu email experienced delays of more than 10 min. About half of that experienced more that 1.5 hr delay and some messages experienced more than 3 hr delay. \n\n | 2010-11-30 | 05:00:00 |
| We are currently investigating this problem and will notify you with updates on this situation. | We are currently investigating this problem and will notify you with updates on this situation. | 2010-11-29 | 05:00:00 |
| The main e-list server is up and running but users are experiencing mail delays for mail going through e-lists. We are still investigating this problem and will notify you with further updates. | The main e-list server is up and running but users are experiencing mail delays for mail going through e-lists. We are still investigating this problem and will notify you with further updates. | 2010-11-29 | 05:00:00 |
| CIT staff are still investigating the e-list mail delays. | CIT staff are still investigating the e-list mail delays. | 2010-11-29 | 05:00:00 |
| CIT staff report the backlog of messages is clearing,\nthey are actively monitoring the problem at this time.\nThey are still investigating to discover the cause.\nThank you for your patience. | CIT staff report the backlog of messages is clearing,\nthey are actively monitoring the problem at this time.\nThey are still investigating to discover the cause.\nThank you for your patience. | 2010-11-29 | 05:00:00 |