Glitch Slows Faculty, Staff E-Mails

A problem on the University Information Services (UIS) e-mail system caused 2150 e-mail addressed to @harvard.edu to be delayed up to four weeks until they were delivered Sunday afternoon.

The delay caused some consternation among the users of the @harvard.edu alias, according to Director of FAS Computer Services Franklin M. Steen.

"Someone called to complain [and] was worried that mail was being lost," Steen said.

UIS Network Operation Manager Jay Tumas, however, said the problem causing the delay has been fixed, and that UIS apologizes for the confusion.

According to Tumas, UIS first discovered the undelivered e-mails in the secondary mailque of their mailhub during regular system maintenance last Friday morning.

The problem was caused by blockage in the mailqueue of the secondary machine.

"Four weeks ago, the secondary machine received a message with a control character in the header, a `carrot'. The effect [was that] sendmail--the UNIX program that delivers mail--couldn't operate on the message," Tumas explained.

UNIX has a legal and illegal system that it runs on. The carrot is an illegal sign, meaning that a computer would look at it and be unable to understand it.

"[To fix it] we went and we got a copy of the core dump--a dump of information that UNIX will give you that allows you to troubleshoot the problem," Tumas said.

The UIS team spent four and a half hours determining the problem, and 10 hours actually fixing it.

Tumas said he did not know what caused the particular carrot.

"It could be one of a million reasons the computers put carrots," Tumas said. "[But] it's not very common."

Mailhub services are comprised of two machines. Anything addressed to @harvard.edu goes through the primary machine.

If the primary machine cannot deliver the message, it puts it on hold in its mailqueue to try again every 15 minutes.

Sometimes, however, when the primary machineitself is unable to receive a message, the backupsecondary machine receives the message in itsstead. When the secondary machine cannot deliverthe message either, it puts the message in its ownmailqueue.

Tumas emphasized that the messages that weredelayed formed only a tiny fraction of the dailyload the mailhub processed.

According to him, the secondary machine is usedinfrequently, accounting for only 2 to 4 percentof the total messages. Those messages relegated tothe secondary machine's mailqueue are an evensmaller number.

Tumas did acknowledge, however, that illegalcharacters have blocked the mailqueues before. Thelast time was a year ago.

Tumas said such a problem would have much lesspossibility of reoccurring in the future.

To protect it from happening again, UIS applieda patch to the operating system of the mailhub, sothat in the event the control character comes upagain, the computer will know not to let it sit.

Currently, there are 23,000 individual aliaseson the @harvard.edu mailhub, although Tumasestimates there are only about 9,000 to 11,000actual users

Sometimes, however, when the primary machineitself is unable to receive a message, the backupsecondary machine receives the message in itsstead. When the secondary machine cannot deliverthe message either, it puts the message in its ownmailqueue.

Tumas emphasized that the messages that weredelayed formed only a tiny fraction of the dailyload the mailhub processed.

According to him, the secondary machine is usedinfrequently, accounting for only 2 to 4 percentof the total messages. Those messages relegated tothe secondary machine's mailqueue are an evensmaller number.

Tumas did acknowledge, however, that illegalcharacters have blocked the mailqueues before. Thelast time was a year ago.

Tumas said such a problem would have much lesspossibility of reoccurring in the future.

To protect it from happening again, UIS applieda patch to the operating system of the mailhub, sothat in the event the control character comes upagain, the computer will know not to let it sit.

Currently, there are 23,000 individual aliaseson the @harvard.edu mailhub, although Tumasestimates there are only about 9,000 to 11,000actual users