Lamont Weekly Report – October 24, 2003

OFFICE OF THE DIRECTOR
P.O. Box 1000, 61 Route 9W Palisades, New York 10964

<> REPORT SUMMARY <>

– Spam up-date –
– Letter from the Director –
____________________________________

<> SPAM UP-DATE

On Thursday morning (Oct 23rd) in addition to the security patch applied to the LDEO email system we installed additional SPAM filtering software. The additional software includes Bayesian filters, external checks against the Razor database, external checks against the Distributed Server Boycott List and additional Rule-Based Filtering.
We are seeing an increase from 11-14% of email classified as SPAM to over 25%. The emails identified as SPAM are flagged in the subject line with **SPAM** and continue to be delivered. The default filtering is automatic and there is nothing you have to do.

The SPAM not SPAM decision is made by applying the above tests and checks and calculating a weight for each. When the total weight exceeds our current conservative default (now set at 9.0) the email is declared SPAM. In the near future we intend to stop delivering SPAM with a score over our default setting. It is extremely unlikely that good email will be identified as SPAM at the default level, but anyone wishing to 'opt out' of this filtering can send email to request. We will announce this change prior to implementation. For those that override the default with a lower number (required_hits line in your .spamassassin/user_prefs file) we propose to deliver the mail with weights lower than the system wide default and continue to flag them, since the probability of a false positive is raised with a lower required_hits value.

NOTE: These improvements have not yet been applied to the LDEO Administrative mail server. We intend to upgrade the Admin server in the very near future.

Background information on email filtering
Bayesian filters are the latest in spam filtering technology. They recognize spam by looking at the words (or "tokens") they contain.

Bayesian filters are particularly good at avoiding "false positives"-- legitimate email misclassified as spam. This is because they consider evidence of innocence as well as evidence of guilt. A Bayesian filter is unlikely to reject an otherwise innocent email that happens to contain the word "sex", as a rule-based filter might.

The disadvantage of Bayesian filters is that they need to be trained.
Of course, after the filter has seen a couple hundred examples, it rarely guesses wrong, so in the long term there is little extra work involved.

Razor is a distributed, collaborative, spam detection and filtering network. Through user contribution, Razor establishes a distributed and constantly updating catalogue of spam in propagation that is consulted by email clients to filter out known spam. Detection is done with statistical and randomized signatures that efficiently spot mutating spam content. User input is validated through reputation assignments based on consensus on report and revoke assertions which in turn is used for computing confidence values associated with individual signatures.

The Distributed Server Boycott List (DSBL) contains the IP addresses of servers which have relayed special test messages to listme@listme.dsbl.org; this can happen if the server is an open relay, an open proxy or has another vulnerability that allows anybody to deliver email to anywhere, through that server. Note that DSBL itself doesn't do any tests; it simply listens for incoming test messages and lists the server that delivers the message to DSBL's mail server.

Rule-Based (aka Heuristic) Filtering means detection of specific character strings within the header, subject or body of the message.
Words like "Viagra" or phrases like "Italian-crafted Rolex" are included in the weighting or force the email to be declared as SPAM with no further checks.

<> LETTER FROM THE DIRECTOR

I am pleased to announce that Robin Robertson has joined the LDEO Executive Committee as the representative of the Junior staff, replacing Suzanne Carbotte, who was just promoted to Doherty Research Scientist. Thank you to Robin for taking on this important task.

Stan Jacobs and Steve Goldstein have taken on the leadership role of planning the NSF OPP visit to Lamont on November 13th. We will insure that some part of the day's agenda will provide an opportunity for open interaction with Tom Pyle and Scott Borg and their accompanying program managers. More details and a draft agenda will be forthcoming.

I had lunch with George Rowe, who leads the Vetlesen Foundation, and Ambrose Monell, who leads the Monell Foundation, on Tuesday. Besides giving them an update on recent events at the Observatory I thanked them for their continuing support of our efforts. They remain very good and important friends.

I greatly enjoyed my participation in the Borehole group's retreat at the Arden Homestead on Wednesday - planning for leadership in the new ocean drilling program is well underway.

Remember the Chilli cookoff next week,

Have a great weekend,

– Mike
____________________________________

You can subscribe / unsubscribe to the Lamont Weekly Report mailing list.

– To subscribe - send your request to:
majordomo@ldeo.columbia.edu
Subject: Request
The following command must be in the BODY of the message not in the subject:

subscribe lamont-l <email address>

– To unsubscribe - send your request to:
majordomo@ldeo.columbia.edu
Subject: Request
The following command must be in the BODY of the message not in the subject:

unsubscribe lamont-l <email address>

If you have questions or problems send e-mail to the list owner:
owner-lamont-l@ldeo.columbia.edu