Exchange Edge Transport Servers at Microsoft: Part 2
At a Glance:
- Using Edge Transport logs and traces
- Anti-spam configuration with scripts
- Testing typical spam scenarios
Download the code for this article: Edge2007_11.exe (161KB)
Faced with a flood of spam and other malicious content flowing across the Internet, how do you ensure reliable and safe messaging to users within your organization? One way, which I started to describe last month in the first part of this series, is to deploy
Exchange Server 2007 Edge Transport servers and Forefront Security for Exchange Server in a perimeter network between the Internet and your production environment.
In this final installment of my two-part series, I will explain how to analyze Edge Transport agents in action by using standard agent logging and tracing features of Exchange Server 2007. I will also use a custom method to dump all event objects from the transport pipeline into XML-based files. I will then illustrate how to apply anti-spam settings to an Edge Transport test lab according to the configuration that Microsoft uses in its own corporate production environment. Finally, I'll conclude with a few realistic and interesting test scenarios to see how Edge Transport agents respond to various spam situations.
For more information on the installation of the Edge Transport test lab and the general Edge Transport architecture, see the first article of this series in the October 2007 issue of TechNet Magazine, available at technetmagazine.com/issues/2007/10/Edge.
I make heavy use of scripts and batch files to automate the most important configuration tasks, and you can find these scripts in the code download that accompanies this article (go to technetmagazine.com/code07.aspx). Furthermore, for more detailed background information regarding the Microsoft IT configuration of Edge Transport agents, I recommend the technical white paper "Microsoft® Exchange Server 2007 Edge Transport and Messaging Protection," produced for Microsoft IT Showcase. You can download this IT Showcase white paper from microsoft.com/technet/itshowcase/content/edgetransport.mspx.
Logging and Tracing
Exchange Server 2007 includes a number of logging and tracing features, including protocol logging, agent logging, and pipeline tracing, that facilitate diagnosing and troubleshooting transport agent problems. Although I am not exactly troubleshooting transport agents in this article, the logging and tracing information is useful for examining how Edge Transport agents work.
If you want to track information about SMTP send and receive activities, you should enable protocol logging for send and receive connectors. Using the test environment I set up in the first part of this series (see Figure 1), I enabled protocol logging by means of the following setting on the New-SendConnector and New-ReceiveConnector cmdlets:
Figure 1 The test environment
You can find the resulting SMTP log files by default in subfolders under %ProgramFiles%\Microsoft\Exchange Server\TransportRoles\Logs\ProtocolLog.
Protocol logging is useful if you want to analyze SMTP conversations, yet anti-spam agents do not always interfere with the SMTP session. For example, the Content Filter agent might only stamp inbound messages with a Spam Confidence Level (SCL) value. As long as the SCL rating is below a specified threshold (the default is 7), the Content Filter agent does not cause the Edge Transport process to reject the message. Therefore the SMTP conversation progresses unaffected.
To analyze the actions that anti-spam agents (Connection Filter, Content Filter, Edge Rules, Recipient Filter, Sender Filter, and Sender ID) perform on messages that pass through the transport pipeline, you need to check the agent logs instead of the protocol logs. Agent logging is typically enabled according to the AgentLogEnabled parameter in the EdgeTransport.exe.config file. By default, agent logs are located in the %ProgramFiles%\Microsoft\Exchange Server\TransportRoles\Logs\AgentLog folder. You can open the logs in Notepad, though the Get-AgentLog cmdlet displays the agent log information more intuitively in the Exchange Management Shell.
Another useful feature is pipeline tracing, which enables you to capture snapshots of messages from a particular sender as the messages pass through the transport pipeline. The idea is relatively straightforward; the Edge Transport process creates a snapshot of each message before and after invoking SMTP receive and routing agents registered for the OnEndOfHeaders, OnEndofData, OnSubmittedMessage, and OnRoutedMessage events. By comparing the message snapshots, you can see how individual agents modify each message. Of course, it is imperative that you enable this valuable feature in the test environment.
The following usage of the Set-TransportServer cmdlet activates pipeline tracing for a test user. The PipelineTracingEnable parameter set to $true enables pipeline tracing, as you would expect. The test user's e-mail address is set in the PipelineTracingSenderAddress parameter:
Set-TransportServer –Identity EDGE01 –PipelineTracingEnabled $true -PipelineTracingSenderAddress Contoso.User@contoso.com
With pipeline tracing enabled, you can find message snapshot files for each message from the specified sender (in this case, it is Contoso.User@contoso.com) in the %ProgramFiles%\Microsoft\Exchange Server\TransportRoles\Logs\PipelineTracing\MessageSnapshots folder. This is the default folder. For each message, the snapshot files are located in a subdirectory named according to a GUID assigned to the message. Note that you can find the original message in a file called Original.eml and copies of the message in .eml files that start with SmtpReceive or Routing according to the encountered events and installed agents. To analyze the processing of anti-spam agents registered for the SMTP receive events OnEndofData and OnEndOfHeaders, check out the SmtpReceive*.eml files. To analyze the processing of antivirus and other agents registered for the routing events OnSubmittedMessage and OnRoutedMessage, analyze the Routing*.eml files instead.
Custom Pipeline Dumping
The standard agent logging and tracing features of Exchange Server 2007 address all troubleshooting needs despite the fact that the standard features do not cover all transport events. For example, pipeline tracing cannot capture information about OnConnectEvent, OnEhloCommand, OnMailCommand, OnRcptCommand, OnDataCommand, and similar transport events because the sending host has not yet transferred any messages that could be written to the %ProgramFiles%\Microsoft\Exchange Server\TransportRoles\Logs\PipelineTracing folder. If you want to capture detailed information about these events, you need to implement a custom agent that dumps the event data into files on the file system.
As discussed in my previous article, it is not complicated to create custom agents, and so I could not resist building an SMTP receive agent and a routing agent to dump out all event objects from the transport pipeline into XML files. You can find a corresponding Microsoft Visual Studio® 2005 project, called PipelineXMLDump, in the download that is available for this article. The two important files are DumpAgents.cs, which implements the agent factories and agents, and XMLDump.cs, which implements a helper class to write the fields of event source and event argument objects by means of System.Re- flection methods into XML files. If you compile the source code and copy the resulting PipelineXMLDump.dll file into a folder called C:\PipelineXMLDump on the Edge Transport test server, you can register and enable the custom agents by using the scripts RegisterPipelineXMLDump.ps1 and EnableDumpAgents.ps1. The scripts are included in the PipelineXMLDump project folder in the download. Of course, my custom agents are for illustrative purposes only, have not been tested sufficiently, and should never be installed on a production server. Use these agents at your own risk.
DumpAgents.cs shows you how to implement all SMTP receive and routing event handlers that the transport pipeline supports. This can be a good starting point if you want to implement your own custom agents. My dump agents write XML files in subfolders under %ProgramFiles%\Microsoft\Exchange Server\TransportRoles\Logs\PipelineXMLDump that correspond to the identifier of the SMTP session (SMTP receive agent) or the message identifier (routing agent). Figure 2 shows an example of event argument information passed to the SMTP receive agent in the OnConnectEvent handler. The example reveals the local IP endpoint (IP address and port) that accepted an incoming connection request.
Figure 2 Argument information dumped during the OnConnect event (Click the image for a larger view)
Preparing the Test Runs
Enough theory about logging, tracing, and dumping. Let's get ready for a few test runs that show anti-spam agents in action. The following is a brief rundown that mirrors the Microsoft IT configuration as closely as is possible in your test environment. As mentioned earlier, background information is available in the technical white paper "Microsoft Exchange Server 2007 Edge Transport and Messaging Protection."
Start by deactivating the Address Rewriting Inbound, Attachment Filter, and Address Rewriting Outbound agents by running the EDGE01_disable_agents.ps1 script from the accompanying download. Address rewriting is not necessary because recipients use the same e-mail addresses internally and externally. I do not use the Attachment Filter agent because of the fact that superseding attachment filtering capabilities are available in Forefront Security.
The test environment is not connected to the Internet. This makes it difficult to test the Connection Filter agent. To remedy the issue, you can create an IP block list on the Internet host by running the INTERNET01_create_IP_block_list.bat file. Make sure you have Windows® Support Tools installed, as these include the DNS command-line tool (dnscmd.exe). Otherwise, you have to spend time creating the DNS records manually using the DNS console.
Now you can configure the Connection Filter agent by running the EDGE01_add_block_list_provider.ps1 script. This script adds an IP Block List provider for ip-bl.consolidatedmessenger.com. This is the IP Block list of Consolidated Messenger, already created in the test environment by running the INTERNET01_create_IP_block_list.bat file on the Internet host, as mentioned previously. Microsoft IT also uses the local IP Block list and IP Allow list, but these features do not require manual configuration. It's important to note that Microsoft IT does not use IP Allow List providers.
Next, configure the Recipient Filter agent by running the EDGE01_recipient_filter.ps1 script to block messages to non-existent recipients as well as messages to mailboxes and global distribution lists that are for internal or outbound use only.
The Sender Filter agent also needs a configuration update. Microsoft IT uses sender filtering as a countermeasure against mail flooding attacks from specific sender addresses and to block certain list servers and similar sources that are deemed not relevant for business. Microsoft IT furthermore uses sender filtering to block messages with invalid or missing sender information. Let's apply this configuration to the test environment by running the EDGE01_sender_filter.ps1 script on the Edge Transport server.
You also need to configure Sender Policy Framework (SPF) records in DNS if you want to mirror the Microsoft IT configuration. To generate an SPF record that attests the authority of your Edge Transport server for messages from your domain, adventure-works.com, you can use the Microsoft Sender ID Framework SPF Record Wizard available at microsoft.com/mscorp/safety/content/technologies/senderid/wizard. Give it a try and specify an existing Internet domain name. In the test environment, you can run the INTERNET01_create_SPF_record.bat file on the Internet host.
That's it! No further parameters need to be configured because most of the agents come with reasonable default settings. To perform a quick check of the most important default settings for the Edge Rules, Sender ID, Protocol Analysis, and Content Filter agents, run the EDGE01_check_defaults.ps1 script on the Edge Transport server.
Edge Transport Agents in Action
Now, let's put the Edge Transport agents to work in a few realistic messaging scenarios. To send test messages to the Edge Transport server, I use the SMTP and POP3 service included in Windows Server 2003 and Outlook® Express as the messaging client, using the default settings most of the time, but making sure logging is enabled.
But first, a warning: do not follow any of the steps or run the scripts described in this section if your test environment has any connections to a production system or the Internet. You are testing at your own risk.
The first agent I am going to test is the Connection Filter agent. It deserves to be tested first because it is the hardest working agent on an Edge Transport server. At Microsoft, the Connection Filter agent blocks about 88 percent of all Internet message submissions. Out of 13 million submission attempts on a regular business day, only about 1.5 million are from legitimate senders.
So let's send an e-mail as Fabrikam.User@Fabrikam.com to Administrator@adventure-works.com. If things work properly, Administrator should not receive the message. Instead, Fabrikam.User should receive a delivery status notification (DSN) stating that delivery to the recipient has failed. Fabri- kam.User verifies that the e-mail address is correct, sends the same message again, and again Fabrikam.User receives a DSN. Then Fabrikam.User calls the help desk and reports the issue. She is not the first. In fact, nobody at Fabrikam can communicate with Adventure Works anymore. Something has happened. The issue is urgent. Trying to resolve it quickly, you check the SMTP logs in the %systemroot%\system32\Logfiles\SMTPSVC1 folder on INTERNET01, and as shown by the red text in Figure 3, there it is!
... 220 mail.adventure-works.com Microsoft ESMTP MAIL Service ready at Thu, 31 May 2007 12:21:33 -0700, ... EHLO, -, INTERNET01, ... 250-mail.adventure-works.com Hello [192.168.1.100], ... MAIL, -, FROM:<Fabrikam.User@fabrikam.com> SIZE=840, ... 250 2.1.0 Sender OK, ... RCPT, -, TO:<Administrator@adventure-works.com>, ... 550 5.7.1 E-mail rejected because your IP address is listed by ip-bl.consolidatedmessenger.com. Please visit http://www.consolidatedmessenger.com/ip-bl for more information. If you still need assistance, contact email@example.com., ... RSET, -, -, ... 250 2.0.0 Resetting, ... QUIT, -, -, ... 221 2.0.0 Service closing transmission channel,
The 550 error response tells you that Consolidated Messenger put your Internet host's IP address on an IP Block list. Fair enough. You contact Consolidated Messenger to get removed from that list. As it turns out, it is easier to get on the Consolidated Messenger block list than to be removed from it. It will take days if not weeks to clean this up.
You need help more quickly, and so you contact Adventure Works through the e-mail address firstname.lastname@example.org. You send an e-mail as Fabrikam.Admin@Fabrikam.com, describing the situation and asking to unblock your host's Internet IP address (in this case 192.168.1.100). After verifying your identity, Adventure Works unblocks the address by using the following command to grant a temporary exception that automatically expires after 30 days:
Thirty days should give you enough time to sort things out with Consolidated Messenger, and Adventure Works does not need to spend time maintaining the exception manually. By granting self-expiring exceptions, Adventure Works keeps the administrative overhead associated with maintaining IP Allow lists at a minimum.
Next, let's test the safe sender awareness of content filtering, which is a new and interesting feature that increases the precision of spam filtering. EdgeSync propagates safe sender information from the internal environment in hashed and encrypted form to Edge Transport servers. The Content Filter agent uses this information during the filtering process to make informed decisions. You only need to get the safe sender information (that is, Safe Senders list, Safe Recipients list, Blocked Senders list, and external contacts) into Active Directory® by using the Update-SafeList cmdlet and then replicating the information to the Edge Transport server by using the Start-EdgeSynchronization cmdlet. Microsoft IT runs the Update-SafeList cmdlet on a scheduled basis during non-peak hours. In our test environment it is sufficient to run a corresponding script (HUB-MBX-01_update_safelist.ps1) manually.
First, clean up the test environment by running the EDGE01_clean_up.ps1 script on the Edge Transport server to remove the previously created IP Allow list entry. This is necessary because the Content Filter agent does not act on messages from sources that are on this list. Next, remove the IP Block list entry for 220.127.116.11.ip-bl.consolidatedmessenger.com from the Consolidated Messenger block list by running the INTERNET01_clean_up.bat file on the Internet host. Otherwise, the simulated Internet host will not be able to send messages, as demonstrated earlier.
Finally, send a message that the Content Filter agent almost certainly considers to be spam. This is tricky because the Content Filter agent uses the default SCL reject threshold to deliver most messages to the users to minimize false positives. Yet after some testing, I found one message version that brings the SCL rating up to the reject level. To send the message as Contoso.User@Contoso.com, run the INTERNET01_contoso_msg.vbs script on the Internet host.
My script uses an anonymous SMTP connection to submit the message to the SMTP service on INTERNET01. For it to work, you need to configure the SMTP service to relay messages for anonymous users. In IIS Manager, open the properties for the Default SMTP Virtual Server. On the Access tab, click on the Relay button, and then select All except the list below. I will cover the implications of this configuration in a moment.
When you run the script to send this message, Contoso.User should receive a DSN stating that delivery failed with: Diagnostic-Code: smtp;550 5.7.1 Message rejected due to content restrictions. You can also see this response in the SMTP log on the Internet host, and if you run the command Get-AgentLog | Select-Object -Last 1 on the Edge Transport server to display the last entry from the agent log, you should be able to see that the message was rejected because its SCL rating was 7, as shown in Figure 4.
Figure 4 Message blocked due to content restrictions (Click the image for a larger view)
Now, Contoso.User is not a spammer. It is unfortunate that her message did not go through in this case, but Contoso.User can usually communicate with recipients at Adventure Works. So Contoso.User informs Administrator in another e-mail that her original message was mistakenly blocked. Administrator knows Contoso.User and, to ensure that spam filters do not block her messages in the future anymore, Administrator adds Contoso.User to the Safe Senders List. In order to do this in Office Outlook 2007, right-click on a received message from Contoso.User, point to Junk E-mail, and click on Add Sender to Safe Senders List.
At the usual interval, the Update-SafeList cmdlet runs to refresh the safelist information in Active Directory and after the next Edge Synchronization, Contoso.User will be able to communicate with Administrator without false positives. To simulate this procedure, run the HUB-MBX-01_update_safelist.ps1 script.
Now run the INTERNET01_contoso_msg.vbs script to send the Contoso.User message again. Check the agent log on the Edge Transport server afterward by using the command Get-AgentLog | Select-Object -Last 1, and you can see under ReasonData that content filtering was now bypassed. No more false positives for Contoso.User!
To conclude the test runs, let's go overboard. As you no doubt have noticed, I configured the Internet host during the previous test as an open relay, which opens the door to spammers. So now let's see what happens if Spam.Sender@cohovineyards.com notices this weak configuration and abuses the host to send thousands of spam messages to Adventure Works. The INTERNET01_spam_msg.vbs script only submits one spam message, but you can edit the script and change the max_loop value to generate more.
To simulate a spam attack, I submitted 1,000 messages to my compromised host for relay to the Edge Transport server. The SMTP service limits the number of messages per outbound connection to 20, so it took some time to relay a sufficient number of spam messages before the Protocol Analysis agent on the Edge Transport server had enough and kicked in.
The Protocol Analysis agent assists the Connection Filter agent by automatically maintaining IP Block list entries based on SMTP protocol analysis and IP reputation updates from Microsoft Update. In this case, the SMTP protocol analysis caught the spam e-mail messages. The Protocol Analysis agent keeps track of each message's SCL rating to calculate an average SCL value that it uses to determine a sender reputation level (SRL). As more and more illegitimate messages arrived during my simulated spam campaign, the Internet host's SRL increased until it exceeded the SRL block threshold. As a result, the host landed for 24 hours on the IP Block list. No more spam whatsoever from this host! It happened automatically. No administrator intervention was necessary.
Figure 5 shows the resulting IP Block list entry and corresponding changes in message handling. Prior to blocking the host, the Edge Transport server rejected each spam message during the OnEndOfData event because it was the Content Filter agent that acted upon each message based on the SCL threshold (for reference, see Figure 4). The message was submitted, although unsuccessfully. After blocking my host, the connection was rejected during the OnMailCommand event by the Connection Filter agent. Messages are no longer submitted. It saves network bandwidth and processing resources to disconnect spammers before message submission.
Figure 5 Stopping an offending Internet host (Click the image for a larger view)
Of course, there are many more agent features and scenarios that you can test. For example, you can test sender filtering by sending messages as Blocked.User@Contoso.com or Blocked.User@Fabrikam.com. You can test recipient filtering by sending messages to Blocked.User@adventure-works.com. You can also use Telnet to simulate directory harvesting attacks with varying tarpitting intervals. Microsoft IT uses the default interval of five seconds for SMTP 5yz responses on the Internet-facing receive connector, which is a sufficient amount of time for Microsoft IT to render directory-harvesting attacks impractical for spammers. But you can apply different values using the Set-ReceiveConnector cmdlet, as demonstrated in the EDGE01_tarpitting.ps1 script.
You can also go deeper and check out the snapshot files that pipeline tracing creates for the messages from Contoso.User@Contoso.com. Examine the antivirus stamps, X-MS-Exchange-Organization-AVStamp-Mailbox: MSFTFF;1;0;0 0 0, which Forefront Security uses to mark messages as scanned. When Forefront Security on the Hub Transport server finds this stamp with recent version information, it does not have to scan the message a second time. If you try to fool Forefront Security by inserting fake antivirus stamps into your test messages, you can see the header firewall in action, which removes these stamps right after the OnDataCommand event, long before the OnSubmittedMessage event is triggered that invokes the FSE Routing Agent.
While you are looking at the message headers in the snapshot files, perhaps you might want to create an SPF test record for Contoso.com that identifies a wrong IP address as the legitimate sender. One way to accomplish this is by running the INTERNET01_wrong_SPF_record.bat file on the Internet host. Then you should send a test message as Contoso.User@Contoso.com and look at the X-MS-Exchange-Organization-SenderIdResult and Received-SPF header:
X-MS-Exchange-Organization-SenderIdResult: SoftFail Received-SPF: SoftFail (EDGE01.extranet.adventure-works.com: domain of transitioning Contoso.User@Contoso.com discourages use of 192.168.1.100 as permitted sender)
As you've seen in this and my previous article, Exchange Server 2007 Edge Transport servers come with a comprehensive set of transport agents to implement reliable, high-precision anti-spam capabilities, including connection filtering, protocol analysis, and content filtering. The transport architecture is extensible and supports the implementation of further agents for additional functionality. The FSE Routing agent is a good example of an additional agent that integrates Forefront Security for Exchange Server into the transport architecture.
Combined with additional features, such as connection tarpitting, header firewall, TLS, and authentication, Edge Transport servers provide the necessary means to achieve a very high level of messaging protection at multiple levels and without single points of failure. By deploying Edge Transport servers and Forefront Security in a perimeter network, strictly separated from your internal Exchange Server organization, you can keep the flood of illegitimate and malicious content away from your messaging environment while delivering legitimate messages with very low false positive rates to your users.
Kay Unkroth is an entrepreneur who has worked as a support engineer, system developer, consultant, trainer, and author focused on Microsoft server technologies for more than 15 years. Kay is also co-founder and chairman of Biblioso Corporation, a company that specializes in managed documentation and localization services.
© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.