Service Notice: Mail delay (mostly fixed)

[forhire]

At about 6:00 PM on Tuesday the 20th we began to notice an abnormal level of mail being spooled on the antivirus server. By the morning of the 21st we had over a million messages pending delivery. Throughout the day we worked to purge the spam and bogus bounce notices that were backing up everyone's e-mail. At about midnight things started looking pretty good. At present most of yesterday's mail has been delivered. I expect things will be back to normal by morning.

[forhire]

All the mail pending in the incoming queue has been passed to the active queue where the virus scanner is working on the last 10,000 messages or so. I've received most of my mail from yesterday. The backup should be eliminated during the next hour or so as all the pending mail is finally delivered.

I am looking into an upgrade for this server. The bottleneck appears to be the antivirus daemon itself (which is fairly slow). Plans are in the works to eliminate this issue.

[forhire]

This issue has been resolved. Please see below for details.

Late Tuesday evening, 12/20/2005, we began to notice delays in mail delivery due to an abnormally high volume of mail. We expected the issue to sort itself out but by Wednesday morning it was obvious the antivirus server wasn't keeping up with the volume.

After investigating it was discovered a spam bot was causing millions of bounced e-mail to be delivered to one of our domains. Yesterday afternoon we started purging the offending bounce notices. The purge process took over 12 hours to complete in several stages. This morning we manually completed the final cleanup. At present nearly all the spooled e-mail has been delivered.

Due to how e-mail works some messages may continue to trickle in from other servers around the net. Some messages are still slightly delayed. We expect everything to be back to normal over the next few hours.