The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
Creating an Open Source Email Filtering Gateway - a How-To by Larkin Cunningham
1. COVER STORY
Amavisd-new
Filtering spam and viruses at the mail server with Amavisd-new
SCAN MANAGER
Deutsche Post WorldNet
Sometimes the best time to stop bad mail is before it arrives. Amavisd-
new is a Open Source interface for integrating spam and virus filtering
a daemonized set of master and child
with your mail server. BY LARKIN CUNNINGHAM processes. Amavisd-new acts as an SMTP
server, receiving email messages from
your SMTP server (e.g., Postfix, exim, or
qmail), processing the messages, and
T
he majority of viruses are propa- duces the risk of spam and viruses sending them or reinjecting them back
gated via email. While it is still reaching your desktop. An additional ad- into your SMTP server.
possible to receive viruses from vantage of providing this extra layer of Amavisd-new supports anti-spam
floppy disks and CD-ROMs, or from In- protection is that, by using a virus tools such as SpamAssasin [2], as well
ternet worms and root kits that directly checker on your desktop that is different as a wide range of commercial and open
attack vulnerabilities in your operating from the content checker on your SMTP source virus checkers. The popular Clam
system, you are much more likely to re- server, you reduce the risk of a virus Antivirus [3] is supported in three ways;
ceive viruses via email – often by inad- passing undetected. the clamd daemon (the best perfor-
vertently opening seemingly innocent at- Amavisd-new [1] is an Open Source mance), the Mail::ClamAV Perl package
tachments. tool that serves as an interface from a (not so good performance) and the clam-
Of course it is important to have an mail server to virus scanners and other scan command line option (as a backup
up-to-date virus checker on your desk- forms of content checkers. Although when clamd is unavailable, for exam-
top, but defense in depth is the best ap- some virus tools provide their own ple). Many other popular virus checkers
proach, and this is why you should be mechanisms for filtering mail at the are also supported, including F-Prot,
protected from spam and viruses at your server, Amavisd-new offers performance Sophos, Grisoft’s AVG, KasperskyLab
receiving SMTP server. Virus protection advantages in some environments, and AVP, AntiVir, F-Secure, McAfee and
at the mail server prevents undesirable it also provides a single, vendor-neutral Panda. But be aware of the licensing for
email from reaching your desktop, and it configuration point for managing both the product you choose. Obviously
also keeps viruses out of your POP3 or virus and spam filtering. ClamAV will not incur any license fee,
IMAP account. Filtering from the mail but some of the other options may have
High Performance
server reduces the load on your desktop a license fee for an SMTP server that is
PC or laptop, reduces the amount of Amavisd-new is a flexible, high perfor- significantly greater than the desktop li-
bandwidth you will use, and greatly re- mance Perl-based application that runs as cense fee.
35
ISSUE 62 JANUARY 2006
W W W. L I N U X - M A G A Z I N E . C O M
2. COVER STORY Amavisd-new
You can configure Amavisd-new to bases are also sup-
block attachments with commonly dan- ported, including
gerous extensions, such as .exe, .bat, Oracle and DB2.
and .vbs (particularly dangerous to Win- Be aware that
dows clients who may be using the mail many of the tables
server). You can also specify a wide have serial / se-
range of decoders/decompressors to ex- quence primary
amine archives such as .cpio, .rpm, .deb, keys, so for Ora-
. zoo, .tar, .gz, and .bz2. cle, you’ll need to
Amavisd-new can theoretically sup- add several se-
port any SMTP server, but it works bet- quences and trig-
ter with the usual suspects, including gers in addition to
Sendmail, exim, and qmail. Amavisd- the tables.
new works best with Postfix [4], which You must spec-
can reinjectï¿œmail back into itself after ify two DSNs
content filtering. (Data Source
Figure 1: Amavisd-new is just one big Perl script.
Names). One DSM
Installation Requirements you have set up the background applica- is for lookups and the other for storage
To install Amavisd-new, you need a tions, the installation is relatively straight- (logging). You can improve performance
working Perl installation. Perl 5.8.2 or forward. Amavisd-new does not require by using a database that is very fast for
greater is recommended. Though earlier the compilation of any code, since it uses read-only access to do lookups and a da-
versions of Perl will work, later versions the Perl interpretor. tabase that is fast for writes for storage.
work better. You need to install a num- Amavisd-new also lets you specify multi-
Configuring Amavisd-new
ber of Perl packages (Table 1). Be sure to ple DSNs to allow for failover.
with SQL
update any packages you already have to
Policies
the latest versions. There are several ways to configure
Obviously, for effective spam filtering, Amavisd-new. You can define your Ama- If you are familiar with firewall configura-
the latest Mail::SpamAssassin package visd-new configuration using the ama- tion, you will be familiar with the term
should be installed. Spammers attempt visd.conf file, lists files, hash lookup policy. A firewall policy is a set of instruc-
to stay one step ahead of SpamAssassin, files, regular expressions, and LDAP or tions to a firewall specifying what to do
so you should always update the SQL lookups. with certain types of network traffic. For
SpamAssassin package as soon as a new The SQL solution gives the greatest example, a firewall policy might specify
version becomes available. opportunity to build a front-end configu- whether to block traffic to a certain net-
You should also install a number of ad- ration tool using a scripting language work port or whether to redirect traffic
ditional programs to allow for the widest such as PHP. You can use any database from one port to a different port. An
range of decoding and decompressing of that supports standard SQL and is sup- Amavisd-new policy is similar. With each
email attachments. These additional pro- ported by Perl DBD/DBI libraries. SQL policy, you can direct Amavisd-new to
grams include gzip, bzip2, arc, lha, rar, DDL (Data Definition Language) is pro- bypass spam checking or virus filtering,
zoo, pax, cpio, freeze, ripole, cabextract, vided in the README.sql file to create be tougher or more lenient with spam,
and many others. See the Amavisd-new tables, indexes, and some sample data. tag the spam email’s subject line with a
website [1] for more on installation. After Notes are included for MySQL, Post- marker (like {Spam?}), and forward all
greSQL, and SQLite. Many other data- spam to a designated spam account.
Table 1: Required Perl
Table 2: Some of the options in the policy table
Packages
Archive::Tar virus_lover Do not reject the email even if virus infected
Archive::Zip spam_lover Do not reject the email even if identified as spam
Compress::Zlib Note: The _lover options do not tell Amavisd-new not to scan, just to ignore the results of the scan.
Convert::TNEF bypass_virus_checks Do not scan the email for viruses
Convert::UUlib bypass_spam_checks Do not scan the email for spam and do not add the X-Spam headers
MIME::Base64 spam_modifies_subj Add some text (a tag) to the beginning of the subject if it is spam
MIME::Parser virus_quarantine_to An email account to send all virus infected emails to
Mail::Internet spam_quarantine_to An email account to send all spam to
Net::Server spam_tag_level The SpamAssassin score above which spam emails get X-Spam headers
Net::SMTP spam_tag2_level The SpamAssassin score above which spam gets the subject tagged
Digest::MD5 spam_kill_level The SpamAssassin score above which ‘evasive’ action is taken, determined
by $final_spam_destiny in the configuration file (default is
IO::Stringy
to discard the email, i.e. Send it to a blackhole)
Time::HiRes
spam_subject_tag The text to use for the spam tag (see spam_modifies_subj)
Unix::Syslog
spam_subject_tag2 The text to use for the high scoring spam tag
BerkeleyDB
36 ISSUE 62 JANUARY 2006 W W W. L I N U X - M A G A Z I N E . C O M
3. COVER STORY
Amavisd-new
Black or white listed emails are added to
Logging and the Law the mailaddr table with a unique identi-
A hot topic for ISPs at the moment is a fier. The rows in the wblist table specify
potential requirement to store email the email address and target user to
data for up to three years. This is in re-
which the rule applies.
sponse to recent terrorist atrocities and
is designed to give law enforcement
Logging and Archiving
agencies information to track the activi-
One of the newer features of Amavisd-
ties of terrorists, in addition to telephone
new is the ability to log all email activity.
and mobile phone call and SMS logs.
The logging information is comprehen-
While storing email content would inevi-
sive and includes information such as
tably lead to a massive storage burden
for ISPs, the archiving of the logging in- senders, recipients, times, dates, sub-
formation generated by Amavisd-new jects, and spam or virus scores. Loging Figure 2: Amavisd-new provides detailed
could address many of the requirements lets you keep accurate statistics on the logging information that allows for informa-
of any future European or local govern- level of clean versus spam versus virus tive charts.
ment legislation.
emails received. The fact that you can
record these statistics at the user level taking a look at the Postfix and Amavisd-
Each email domain or user can have gives the system administrator an oppor- new combination. Particularly for ISPs,
its own policy. You can specify a policy tunity to see where content checking re- there is an imperative to providing cus-
to cover all email accounts within a do- sources are most needed. For an ISP (In- tomers with spam and virus filtering,
main and also decide to have additional ternet Service Provider), this gives the whether it is free or as an add-on to their
policies for individual email accounts opportunity to bill for bandwidth, pro- service.
within the same domain. For example, cessing time, number of messages pro- Growing levels of spam and viruses
you might have a default policy to cover cessed, or the number of spam or virus and greater numbers of customers are
all users in a domain (like a catch-all emails processed. putting a strain on the infrastructure of
policy), but perhaps Tom’s email ac- many ISPs. An open source solution
Conclusion
count is being flooded with spam. An in- using Postfix and Amavisd-new could be
dividual policy for Tom’s account could For those who are considering building a the answer. Both Postfix and Amavisd-
have a lower tolerance. Table 2 details robust and scalable architecture for bat- new can be configured to use MySQL,
just some of the policy options. tling spam and viruses, I recommend PostgreSQL, or Oracle for retrieving data.
A user table allows you to assign poli- It is possible to have a central database
Administration and
cies to specific domains and individual accessed by multiple Postfix and Ama-
Statistics Made Easy
email accounts. The user table’s email visd-new servers. And it is possible to
field determines whether multiple email employ multiple databases, allowing for
Using a database instead of configura-
accounts or just a single email account greater fault tolerance. By using a num-
tion files allows for easy administration.
will be affected by the policy. For exam- ber of failover techniques, including
The database option also allows for dy-
ple, the email @domain.com refers to all configuration of DNS records, it is possi-
namic updating of configuration data
email accounts in the domain.com do- ble to build a scalable, fully fault toler-
without having to reload the Amavisd-
main. But, tom@domain.com refers to a ant, and load balanced infrastructure. ■
new daemons. The same is true if you
opt to configure your Postfix users and
single email address. If both @domain.
INFO
domains in a SQL database. By query-
com and tom@domain.com exist in the
ing the storage database using SQL, it is
users table, precedence is given to the [1] Amavisd-new:
easy to garner useful information on
specific email address first. http://www.ijs.si/software/amavisd/
spam and virus statistics. You could
[2] SpamAssassin:
achieve this using phpMyAdmin, for ex-
Black and White Listing http://spamassassin.apache.org
ample, if MySQL is the chosen database.
When important emails you want to re- [3] Clam Antivirus:
An ISP with a large number of custom- http://www.clamav.net
ceive continually get blocked because
ers could develop an interface that
[4] Postfix SMTP Server:
they are identified as spam, you can take
would allow customers to update their http://www.postfix.org
action to ensure that the sender of the own policies to be more or less aggres-
emails is exempt from spam checking by sive against spam or viruses. It would
adding them to the white list. Con- Larkin Cunningham is an IT Consul-
also allow users to view statistics on the
versely, when you continually receive
THE AUTHOR
tant specializing in the areas of
levels of clean, spam, and virus infected
Linux Administration, Java, Applica-
spam from a source you know is a spam- emails. These types of statistics can only
tion Frameworks, Oracle and just
help in the education of email users. Fig-
mer, you can add the email address to
about anything in the open source
ure 2 shows an example of the type of
the black list.
community that makes the lives of
statistics you can easily present to cus-
Amavisd-new’s black and white lists
IT Managers easier and Finance Di-
tomers. The example was developed
work by cross referencing the mailaddr
rectors happier. He can be contacted
using ColdFusion MX 7, but you could
(email address) table with the user table
at larkin.cunningham@gmail.com.
also use PHP with jpGraph.
and the wblist (white / black list) table.
37
ISSUE 62 JANUARY 2006
W W W. L I N U X - M A G A Z I N E . C O M