New Weapons In The Fight Against Spam
At a Glance:
- A layered anti-spam solution
- Customizable filtering and response features
- Configuring and using Sender ID
- SmartScreen machine learning technology
Early on, there was a friendly Internet spirit of trust and openness. And users maintained a notion of being generous when receiving e-mail and restrictive when sending.
This open environment is long past—users are now more defensive, as they deal with malicious hackers and Inboxes that are overwhelmed by unsolicited bulk e-mail, or spam. This problem poses more than just frustration. It takes time and eats up system resources. According to some industry surveys, the price tag of spam in terms of lost productivity in 2004 exceeded $21 billion.
Exchange Server has long been a part of the technology aspect of a complete spam solution. Many of you probably remember the early days of e-mail when many products (including Exchange Server) had an open relay enabled by default. Today, as new threats abound, Exchange Server 2003 Service Pack 2 (SP2) moves one step closer to providing an end-to-end, layered anti-spam solution.
In this article I’ll take a closer look at new features in Exchange Server 2003 SP2 that can help keep spam under control in your organization. Along the way I’ll show you how SP2 is already reducing spam problems within the Microsoft corporate e-mail networks.
A Layered Anti-Spam Solution
With the release of Exchange Server 2003, Microsoft established a strong anti-spam foundation that includes a number of features intended to reduce the volume of spam. The foundation is based on the message and spam taxonomies shown in Figure 1. These taxonomies call for a multilayered approach based on the idea that most spam should be stopped at the gateway, before reaching the recipient’s mailbox. This concept resulted in the Exchange Server spam-filtering model shown in Figure 2. This filtering model is an integral part of the Exchange Server 2003 SP2 release.
Figure 1 Message and Spam Taxonomies
The first step in such a system is identity verification, which is the process of making sure the mail is coming from its stated source. The first line of defense is the connection filtering layer. Highly effective, this connection filtering layer accounts for blocking approximately 25 percent of all e-mail coming to Microsoft.
Figure 2 Exchange Layered Spam-Filtering Model
Exchange Server 2003 SP2 provides the following features to achieve this connection filtering functionality:
- Global allow/deny list of individual IP addresses or ranges by subnet mask
- Global allow/deny list of Internet domain names
- Support for multiple Realtime Block List (RBL) and DNS Block List (DNSBL) providers, including paid subs
- Custom rule-based block list service configuration
- Customized server responses based on provider and connection initiation source (a known open relay)
- Configurable exception list that overrides the block lists
- DNS lookup to resolve fully qualified domain names to IP addresses
The global allow/deny list will immediately protect the system by specifying an administrator-defined action. The Realtime Block List acts next. The connection filter will send a DNS query to verify whether the connecting identity has been listed as an open relay or is a known source of spam. If the DNS query returns a connecting IP address that is on the RBL or DNSBL, the connection filtering layer will refuse the mail transaction and trigger a server response to the sender. This entire process will happen after the RCPT TO: command in the SMTP conversation.
If the sender is legitimate and was mistakenly placed on the block list, he can correct the problem based on the custom non-delivery report (NDR) he receives. In addition, the blocked mailer is allowed to submit mail to the postmaster or administrator of the Exchange system to initiate corrective actions.
The majority of Exchange 2003 servers have been deployed behind a perimeter and do not face the Internet directly. This is a good architecture, but can limit the capabilities of connection filtering, as the feature relies on getting the original sender’s IP address to run the DNS query. With the release of SP2, this issue has been addressed by introducing a new header-parsing algorithm that is used to retrieve the originating IP address. Figure 3 outlines the path of an incoming message for servers on or behind the perimeter.
Figure 3 Message Filtering Paths
SMTP Filtering Layer
If an incoming connection successfully passes through the connection filtering layer, the next line of defense is SMTP filtering. Exchange Server makes extensive use of SMTP protocol filtering, which has been implemented to monitor live SMTP sessions and functions as a real-time filter. At Microsoft, this feature accounts for 35 percent of all blocked messages.
SMTP filtering has been complemented by comprehensive SMTP session RFC compliance enforcement, which starts with the first RFC2821 HELO/EHLO command. The SMTP implementation in Exchange Server 2003 SP2 monitors the session for potential violations and refuses to accept mail if the sending identity does not conform to SMTP governing RFC standards.
No mail transaction will be allowed if the remote server does not begin the SMTP session with an appropriately structured greeting or if the greeting contains a domain part that is not RFC2821 compliant. RFC compliance is enforced on MAIL FROM: and RCPT TO: commands as well, to ensure that malicious users cannot get around SMTP filtering by providing, for example, 8-bit characters in the RFC2821 stream.
Another common tactic employed by spammers is to confuse anti-spam solutions by changing the order of commands in an SMTP session. Exchange Server 2003 SP2 enforces RFC compliance in this area as well, preventing spam senders from executing an attack in this way. To prevent dictionary attacks and e-mail address harvesting, Exchange 2003 Server SP2 responds to the VRFY command, but does not release the directory information to the remote party. And by default, Exchange Server 2003 SP2 does not support the EXPN command.
SMTP filtering is based on rules configured by the administrator and consists of sender filtering and recipient filtering. Sender filtering starts with the first RFC2821 MAIL FROM: command and filters messages originating from certain domains or e-mail addresses. A blank sender, for example, is one of the spam attack vectors.
Sender filtering provides a mechanism for spoof detection by persisting the submission method. Based on the administrator-configurable actions, the filter can drop the incoming connection if the sender’s address matches the filter. To minimize the amount of information being disclosed to malicious users, sender filtering can silently accept a message and delete it without notifying the sender. The filtering mechanism also provides an option to archive filtered messages so they can be analyzed at a later time.
Once an item passes through sender filtering, it faces recipient filtering. This starts with the first RFC2821 RCPT TO: command and enables inbound e-mail filtering for a particular recipient in the Exchange environment. Recipient filtering can block messages based on wildcards, allowing administrators to use patterns to block entire ranges of recipients. Messages sent to non-existent recipients are rejected at the protocol level. By rejecting non-existent recipients on the RFC2821 RCPT TO: command, the Exchange server avoids doing unnecessary work generating NDR messages.
Enabling filtering of the recipients who are not in the directory potentially allows spam senders to discover internal directory information by enumerating valid e-mail addresses in the Exchange organization. A malicious user can execute address book mining by parsing the server responses to RCPT TO: commands. To mitigate this threat, Exchange Server 2003 SP2 supports SMTP command tarpitting. An administrator can configure Exchange to delay the server response to the RCPT TO: command (or, in the case of SMTP RFCs, a command compliance violation). When a malicious user tries to harvest responses, the Exchange server significantly slows down its responses and the attack becomes infeasible. To implement this feature, simply create a new TarpitTime entry under the HKLM\System\CurrentControlSet\Services\SmtpSvc\Parameters registry key. The value of the TarpitTime entry is equal to the number of seconds to delay the server response.
Also contributing to the recipient filtering framework, you can restrict distribution list e-mail submissions to authenticated users. Note that recipient filtering applies only to anonymous connections, so all authenticated identities bypass the recipient filtering rules. After passing through the recipient filtering layer, inbound e-mail traffic is subject to inspection by the Sender ID filter.
Sender ID Filter
Sender ID is an industry initiative created to counter e-mail domain spoofing. Sender ID aims to remove the ambiguity associated with the sender identity by verifying that each e-mail message originates from the Internet domain from which it claims to come. This process is based on the sending server’s IP address. Eliminating domain spoofing can help legitimate senders protect their domain names and reputations, and it can help recipients identify and filter junk e-mail and phishing scams.
The Sender ID process is illustrated in Figure 4. Basically, to verify a legitimate sender, your Exchange server checks for the sending domain’s SPF (Sender Policy Framework) record, which is published in the DNS record. The mail server then determines whether the sending e-mail server’s IP address matches the IP address that is published in the DNS record.
Figure 4 Sender ID Authentication Process
Sender ID defines an algorithm that detects the e-mail address of the entity that is most recently responsible for injecting a message into the e-mail system. This is done by extracting the Purported Responsible Address (PRA). The extraction of the PRA ensures that Sender ID verifies the appropriate sender against the correct IP addresses as e-mail systems can legitimately forward mail on behalf of other mail servers.
The Sender ID feature has three modes: delete, reject, and accept. In delete mode the message is accepted and then deleted, with no NDR returned to the sender. In reject mode the message is not accepted and the sending server is responsible for NDR generation. In accept mode (which is the default mode), the Sender ID status will be attached to the message for further anti-spam processing.
The delete and reject modes either delete or reject mail that fails the Sender ID verification. Any other mail items are stamped with the Sender ID status and passed along to the Intelligent Message Filter (IMF). The accept mode just stamps the Sender ID status onto the mail item (even in the case of spoofing). This status is passed to the IMF, and will trigger appropriate Spam Confidence Level (SCL) score modification.
Internal IP Addresses
When setting up Sender ID, you configure an internal IP range to be excluded from the verification process. This is to ensure you extract the correct IP address even if a message takes a few additional hops within your internal network. (This configuration is also used for connection filtering when it is deployed behind the perimeter.)
The logic is this: you enter the static IP addresses of the e-mail servers that face the Internet and your internal subnet addresses. An inbound message is routed through the Internet-facing gateway, an internal SMTP server, and your Exchange server with its Sender ID filter. When the message hits the Exchange server, Sender ID parses the P2 headers backwards and sees that the message arrived most recently from an IP address that belongs to the internal subnet. As such, that address is excluded from Sender ID verification. The next IP address belongs to your Internet gateway. It has also been entered into the configuration table and is therefore excluded from Sender ID verification. The next IP has no match in the configuration table—this is the IP on which you want to run Sender ID verification.
If you do not enter the required configuration data, the Sender ID filter will not work even if you enable it on the corresponding SMTP Virtual Server Instance (VSI) level. If you enable the Sender ID on the VSI without completing the IP address filter configuration, you’ll get an EventID:7518 error. See the sidebar entitled "How to Enable Sender ID Filtering
" for more information on configuring Sender ID.
Content Filtering Layer
Content filtering in Exchange Server 2003 SP2 relies on Microsoft Research SmartScreen machine learning technology, which is incorporated into the IMF. Messages from the Internet arrive at the Exchange SMTP gateway and enter the Exchange server anti-spam framework. Previous layers of the Exchange anti-spam solution (connection, sender, and recipient filtering) block message submissions before actual message data is received. If a message successfully passes all of these previous filters, then the message body is received.
A custom event sink (msgfilter.dll) is invoked when the SMTP End of Data event occurs. The sink passes the message to the IMF SmartScreen DLL (see Figure 5). The SmartScreen technology determines the SCL rating of the message, which is returned to the sink for comparison against the gateway SCL threshold. The administrator-defined action is applied if the message SCL is greater than the gateway threshold. Otherwise, the SCL rating is added to messages for transmission to the recipient's Inbox.
Figure 5 Smart Screen Content Filtering
Custom Message Weighting
The custom message weight feature allows administrators to tailor the IMF filter to account for specific words or phrases, and to adjust filtering on the fly to act on particular message content as needed. Custom message weighting (also known as the bad words list) is configured through an XML file-based implementation that has no corresponding UI. Within the custom weighting XML file, specific words or phrases can be added along with their relative text part location (Subject or Body) and their associated modifier value. The supported modifier values can include positive and negative increments, as well as MAX and MIN values.
During anti-spam processing, the IMF looks into files and searches inside mail items for a string, word, or phrase match. If a match is found, the weight of the SCL is adjusted according to the modifier value. If MAX or MIN values are specified for particular words or phrases, they move the SCL towards MAX or MIN.
To mitigate false positives, a problem common to all types of automated anti-spam technologies, IMF can be configured to work in reject mode. This allows for investigating false positives if a message is submitted by a legitimate sender and rejected by the filter.
Exchange Server 2003 SP2 allows customization of the server response string that is generated and appended to the NDR that is returned to the sender. The default response, "550 5.7.1 Requested action not taken: message refused," can be altered to provide blocked legitimate users with a meaningful explanation of why their message was blocked, along with suggested steps for mitigation. The CustomRejectResponse entry under the HKLM\Software\Microsoft\Exchange\ContentFilter registry key can be configured to enable this functionality. If a value is present, that string will be used in 550 5.7.1 responses instead of the default server response.
The layered filtering approach in SP2 gives you defense-in-depth, helping to keep your Exchange deployment free of spam and phishing messages. It also gives you fine-grained control over how malicious e-mail is identified and handled by your system.
Alexander Nikolayev is the Program Manager at Microsoft in charge of SMTP/NNTP protocols, Transport Core, and Anti Spam components for Exchange Server and Windows Server. He holds an MBA degree from the University of Mary. Read Alex’s posts on the Exchange team blog at
blogs.technet.com/exchange.© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited
For more details on the new anti-spam framework, as well as guidance for downloading and installing SP2 in your Exchange Server 2003 environment, see the
Exchange Server Web site