Microsoft ISA Server 2000 - Configuring and Securing Microsoft Exchange 2000 Server and Clients

Article
02/20/2014

Archived content. No warranty is made as to technical accuracy. Content may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.

Updated : October 3, 2001

Introduction

Microsoft Internet Security and Acceleration (ISA) Server enables you to publish internal servers to the Internet without compromising the security of your internal network. You can configure Web publishing and server publishing rules that determine which requests should be sent to a server on your local network, providing an increased layer of security for your internal servers.

As business-to-business e-commerce becomes more prevalent, more organizations realize the need to protect internal servers, while at the same time making them accessible to specific external users. The publishing feature in ISA Server secures internal server access by external clients.

For example, you can place your Microsoft Exchange 2000 server on the Intranet, protected by the ISA Server computer, and create server publishing rules that allow the e-mail server to be published to the Internet. Incoming e-mail to Exchange is intercepted by the ISA Server computer, which gives the appearance of an e-mail server to clients. ISA Server can filter the traffic and forward it on to the Exchange Server. Your Exchange Server is never exposed directly to external users and sits in its secure environment, maintaining access to other internal network services.

A common ISA Server scenario involves securing the Simple Mail Transfer Protocol (SMTP) communication of mail servers. For example, ISA Server can protect an Exchange Server. The Mail Server Security Wizard configures the policy needed to allow communication between an Exchange Server computer and the Internet. The wizard adds a set of server publishing rules, which redirect communication from Internet users at a particular port to a specified internal Internet Protocol (IP) address. The wizard also creates protocol rules that dynamically open ports for outgoing communication.

ISA Server also includes a Remote Procedure Call (RPC) application filter, which uniquely provides an extra layer of security to the Exchange Server publishing model. This RPC application filter enables secure communication between Microsoft Outlook clients and an Exchange Server, over the Internet.

This document describes some common Exchange Server publishing scenarios, how to configure the Mail Server Security Wizard included with ISA Server, and how to configure Outlook clients running on the Intranet, protected by an ISA Server.

RPC Application Filter

Outlook clients connect to Exchange Server through RPC. ISA Server's RPC application filter protects the RPC communication, as described in this section. In this way, ISA Server protects not only POP3 and SMTP communication, but also uniquely secures RPC communication.

ISA Server's RPC application filter enables secure communication between Outlook clients and an Exchange Server, over the Internet. The RPC application filter protects RPC communication over the Internet, by identifying which specific RPC interface is requested, and allowing only calls to those interfaces. Furthermore, the RPC application filter opens ports dynamically, meaning that the communication is allowed only when it is specifically requested.

In addition, Exchange Server communicates with Outlook clients using a lightweight UDP-based protocol. The RPC application filter also processes new mail notification, as follows: when an Outlook client logs on to an Exchange Server, it registers to receive new mail notifications, by passingthrough RPCa port number on which it will listen. When new mail arrives, the Exchange Server sends a single UDP packet to the port. To allow this type of notification, standard firewalls must typically open a wide range of ports. With the RPC application filter enabled, ISA Server intercepts registration for new mail, and dynamically opens only the necessary ports.

Thus, Exchange Server publishing is more secure with the ISA Server firewall.

See the "New Mail Notification" section for more information on notification and network address translation (NAT).

How the RPC Application Filter Works

In an Exchange Server/Outlook client scenario, the RPC application filter works as follows:

The Outlook client issues request over Port 135 (TCP) through ISA Server to the Exchange Server, to find the service port number associated with the Exchange RPC UUID.
The Exchange Server sends a response back, through the ISA Server, to the Outlook client, with a port number on which the client can communicate. The connection to Port 135/tcp is then closed.
ISA Server uses the RPC application filter to capture this information, and maintains it in a table.
ISA Server allocates a new port on the ISA Server itself, and changes the response that it sends to the Outlook client, to reflect this change. This information is also maintained in the table.
The Outlook client issues a requestseemingly to the Exchange Server, but actually to the new port on the ISA Server. The ISA Server then sends the packet to the Exchange Server. Only communication over this port is allowed.

Changing the Authentication Method

When the Outlook client connects to an Exchange Server, the Exchange Server instructs the Outlook client to communicate directly with an Active Directory domain controller for authentication. In the publishing scenarios described in this paper, this direct communication will not function properly if the Outlook client is on the Internet, while the domain controller is on the corporate Intranet. Because ISA Server does not publish the server running Microsoft Active Directory directory service, the Outlook client cannot contact the domain controller for authentication.

To allow this type of communication between the Internet-based Outlook client and the Intranet-based Exchange Server, set the value of this registry key on the Exchange Server:

HKLM\System\CurrentControlSet\Services\MSExchangeSA\Parameters

to:

Value name: No RFR Service 
Value type: DWORD 
Value data: 0x1

By making the registry change below, the Exchange Server will "proxy" authentication requests to the domain controller (Active Directory server), instead of referring Outlook clients to communicate with the directory service directly. In this way, the Outlook client can authenticate itself to the domain controller, via the Exchange Server. The authentication traffic is carried out using the Exchange RPC protocol, so no additional publishing rules are required.

New Mail Notification

To enable new mail notification, Exchange Server relies on routable addressing between the Outlook client and the Exchange Server. In the presence of a network address translation (NAT) device (such as ISA Server) between the client and the Exchange Server, new mail notification will not function properly.

New mail notifications sent to the private address are dropped, because these notifications are unsolicited inbound UDP packets not associated with some previous outbound flow. The ISA Server Exchange RPC filter does not know how to access the registration payload to change the client-registered address. Currently no NAT editor is available that knows how to access and change client-registered addresses.

The Outlook client also receives mail notifications during other communications with the Exchange Server. However, if an error in any of the RPC packets occurs (this can happen when RPC is carried over the Internet), the Outlook client does not receive the new mail flag at the end of the packet. To work around this limitation, do the following:

For Outlook 2000 periodically press F9 to check for new mail
For Outlook 2002, configure a polling interval.

Publishing Scenarios

The Exchange Server that you are publishing can be installed on the ISA Server computer or on the local network. The following sections describe some Exchange Server publishing scenarios:

Exchange Server on local network (recommended)
Exchange Server on the ISA Server computer (not recommended)

Exchange Server on Local Network

In this scenario, the server running Exchange is on the local network, protected by the ISA Server computer, as illustrated in the figure.

You can use the ISA Server Mail Server Security Wizard to configure the Exchange Server so that it is available to external clients by using one or more of the following protocols:

Messaging Application Programming Interface (MAPI)
Post Office Protocol 3 (POP3)
Internet Messaging Access Protocol 4 (IMAP4)
Network News Transfer Protocol (NNTP)
RPC for direct access by Outlook/Exchange clients (as described in the RPC Application Filter section)

The wizard creates one or more server publishing rules corresponding to each mail service that ISA Server protects. The wizard also creates a protocol rule and a client address set. The server publishing rules created by the wizard have the following parameters:

The mail server's internal IP address
The external address exposed by the ISA Server
The protocol for the selected mail service

The new rules created by the wizard are all named with the prefix Mail wizard rule.

The Mail Server Security Wizard also creates protocol rules, to allow outgoing mail traffic. The protocol rules have the following parameters:

The protocol is SMTP (client).
The client set includes the internal IP address of the Exchange Server computer.
Name resolution for clients

Since POP3, IMAP4, SMTP, NNTP, and Hypertext Transfer Protocol (HTTP) clients can access the computer that is running Exchange Server either by DNS name or IP address, map the DNS name used by mail clients to the ISA Server computer external IP addresses.

For MAPI clients, a DNS server on the Internet must resolve the name of the computer running Exchange Server and match it to an IP address on the ISA Server computer's external network interface card. The DNS should resolve the published name to the external IP address on the ISA Server.

When communicating with the Exchange Server, the Outlook client specifies the fully-qualified domain name (FQDN) of the Exchange Server. The Exchange Server responds to the client, specifying its internal namewhich typically differs from the FQDN. When the client next communicates, it attempts to use the internal namewhich is not recognized over the Internet. To work around this issue, you should set the Exchange Server computer's internal name to the same name as its FQDN.

Exchange Server on the ISA Server computer

In this scenario, ISA Server and Exchange Server are on the same computer, as illustrated below.

isaexc02

You can use the Mail Server Security Wizard to publish the Exchange Server located on the ISA Server computer. In this scenario, the Mail Server Security Wizard creates IP packet filters. IP packet filters are created for each mail service that you select. For example, imagine that you run the Mail Server Security Wizard and specify Outgoing SMTP mail and POP3 client requests.

Outlook/Exchange clients cannot access the Exchange Server from outside the local network using RPC connections. Only POP3 and IMAP4, also supported by Outlook, can be used.

The following IP packet filters are created:

An IP packet filter allowing Inbound Transmission Control Protocol (TCP) connections on local Port 25 from any remote port (to allow incoming SMTP packets)
An IP packet filter allowing Outbound TCP connections from all local ports to remote Port 25 (to allow outgoing SMTP packets)
An IP packet filter allowing Inbound TCP connections on local Port 110 from any remote port (to allow incoming POP3 packets)

Message Filtering

ISA Server includes message-filtering functionality, which can be used to control the incoming SMTP traffic communicated through the ISA Server. In this architecture, ISA Server sends messages to the SMTP Service, which then sends it to the Message Screener, which sends the messages back to the SMTP service, which may then relay it. The message filtering functionality is implemented in two components:

The SMTP filter is installed by default, but not enabled. It is used to specify which SMTP commands are allowed in the incoming SMTP traffic. The SMTP filter is always installed, regardless of whether you install the message screener. However, if the message screener is not installed, then the filter can screen only SMTP commands and not message content.
The message screener is an extension to the Exchange 2000 SMTP server that can be optionally installed. During ISA Server setup, you can select to install the message screener only if an SMTP server is installed on the computer. The message screener is used to filter messages, specifying which keywords and attachments are permitted and which users and domains are denied. You configure the message screener using the SMTP filter's property sheets.

ISA Server and Exchange 2000 can be deployed on the same (local) computer or on different computers. Depending on your network configuration, you will need to set up and configure SMTP Server, ISA Server, the message screener, and the SMTP filter differently. This document describes the different deployment scenarios, and how you should configure ISA Server and Exchange Server for the different scenarios.

The Mail Server Security Wizard in ISA Server does not configure the SMTP filter when SMTP Server and ISA Server are located on the same computer. The ISA Server and the SMTP Server must be specially configured if you want the SMTP Filter functionality.

Configuring Co-located ISA Server and Exchange Server

This section describes how to configure Exchange 2000 and ISA Server when they are located on the same computer. The figure below illustrates the scenario.

isaexc03

In this scenario, you should install both the message screener and the SMTP filter on the computer. When you install ISA Server and select the Full installation option, the message screener is installed with the Full install option during ISA Server setup. The SMTP Filter is always installed.

Note: The message screener can only be installed if an Exchange 2000 or IIS5 SMTP service is installed on the ISA Server computer.

To verify that the message screener is installed, check to see if the following Registry Key exists on the computer:

HKEY_CLASSES_ROOT \CLSID{4F2AC0A5-300F-4DE9-821F-4D5706DC5B32}

If this Registry Key does not exist, then the message screener was not installed.

To install the message screener

To run the ISA Server setup in maintenance mode, in Control Panel, double-click Add/Remove Programs, then click Microsoft Internet Security and Acceleration Server, and then click Change.
In ISA Server setup, click Continue, type the CD key, select the appropriate installation folder, and then select Custom Installation.
In the Options box, verify that the ISA Services and Administration tools options are not selected.
Highlight the Add-in services option and then click Change Option.
Select only the Message Screener option and then click OK. Then, finish the setup process, selecting the default options.

Note: These options are also available when you install ISA Server and specify Custom installation.

The Mail Server Security Wizard does not configure the SMTP filter when Exchange Server and ISA Server are located on the same computer. The ISA Server and the Exchange Server must be specially configured. The following sections describe how to configure the SMTP Server.

Configure the SMTP Server

In order to be fully secured by the ISA Server, Exchange 2000 must be specially configured to listen only on the internal interface. Perform the following steps:

To configure Exchange 2000 to listen for SMTP traffic on internal interface:

Open the Exchange System Manager. Click Start, click Programs, click Microsoft Exchange, and then click System Manager.
In the console tree of System Manager, click Servers, click the applicable server, click Protocols, click SMTP, right-click Default SMTP Virtual Server, and then click Properties.
On the General tab, click Advanced.
Verify/ensure that only internal IP addresses are listed in the Address box. Remove any other addresses by selecting them and clicking the Remove button.
To add the internal IP address, click Add. Then, select the internal IP address from the list. In TCP port, type:

25

By default, Socket Pooling is enabled. That is, even if you configure Exchange Server's SMTP service to listen on Port 25 for just one interface, it will still listen on all interfaces.

To ensure that the Exchange Server listens on the specified interface: Use MDUTIL.exe or ADSI to set the Metadata raw property ID numbered 1029 (DisableSocketPooling).

Example:
mdutil set -path smtpsvc/1 -value 1 -dtype 1 -prop 1029 -attrib 1

Configure the ISA Server

In order to fully secure the co-located Exchange Server, ISA Server must be specially configured by performing the following tasks:

Enable the SMTP Filter. For instructions, see the "To enable the SMTP filter" section below.
Configure a server publishing rule to make the Exchange Server accessible. For instructions, see the "To create a server-publishing rule to publish the local Exchange Server" section below.

This section describes how to perform these tasks.

To enable the SMTP filter

In the console tree of ISA Server, click Internet Security and Acceleration Server, click Servers and Arrays, click the applicable array, click Extensions, and then click Application Filters.
In the details pane, right-click SMTP Filter, and then click Properties.
On the General tab, verify that Enable this filter is selected.

To create a server-publishing rule to publish the local Exchange Server

Note: Do not use the Mail Server Security Wizard.

In the console tree of ISA Server, click Internet Security and Acceleration Server, click Servers and Arrays, click the applicable array, click Publishing, click Server Publishing Rules, click New, and then click Rule.
Type a name for the rule and then click Next..
On Address Mapping, in IP address of internal server, type the IP address on which the Exchange Server is configured to listen. In this case, this should be one of the ISA Server computer's internal IP addresses.
In External IP address on ISA Server, type the ISA Server's external IP address. Then, click Next.
On the Protocol Settings page, select SMTP Server. Then, click Next.
On the Client Type page, select the clients that can access the SMTP Server. Then, click Next, and then click Finish to exit the wizard.

Configuring a Separate Exchange Server

This section describes how to configure message screening when the Exchange Server and ISA Server are located on different computers. In this case, two configurations are possible, as illustrated in the figures below.

In the configuration illustrated below (Scenario 1), the message screener is installed on the server running Internet Information Services (IIS). The advantage of this configuration is that the Exchange Server is further isolated from the network edge. In this scenario, the message screener screens only incoming messages.

In the configuration illustrated below (Scenario 2), the message screener screens all messages, both incoming and outgoing. It is possible to configure the Exchange Server computer so that only incoming messages are screened.

The ISA Server and the Exchange Server are configured similarly for both these scenarios. The following sections describe how to configure the computers.

Configure the Exchange Server

In scenario 2 (ISA ServerExchange Server) you must create a virtual SMTP Server on the Exchange Server computer. The process is described below.

To install the message screener on a virtual SMTP Server (Scenario 2 only)

Install the message screener on the Exchange Server computer. See the To install the message screener section for instructions.

Note: Install the message screener only after you set up the virtual SMTP server.

The message screener is installed only on the first virtual server (in the list), as listed in the Exchange System Manager.

Configure the Virtual SMTP Server

In this step, you configure the virtual SMTP Server. In Scenario 1, the virtual SMTP Server is located on the IIS computer. In Scenario 2, the virtual SMTP Server is located on the Exchange Server computer.

To configure the virtual SMTP Server

Install the ISA Server Message screener as described in the steps below.

Note: If you have configured your network as described in the To install the message screener on a virtual SMTP Server (Scenario 2 only) section, then you can skip this step.
On the computer running Exchange Server, run the ISA Server setup.
In ISA Server setup, click Continue, type the CD key, select the appropriate installation folder, and then select Custom Installation.
In the Options box, verify that the ISA Services and Administration tools options are not selected.
Highlight the Add-in services option and then click Change Option.
Select only the Message Screener option and then click OK. Then, finish the setup process, selecting the default options.
If you are running ISA Server Standard Edition with configuration Scenario 2 (ISA Server-Exchange Server), or if you are running ISA Server Enterprise Edition as a standalone server, then run the SMTPCred.exe tool. When you run the tool, enter appropriate credentials and the name of the ISA Server computer. The user account required for SMTPCred.exe does not need any special rights, but it does need to exist in the domain. SMTPCred.exe is available on the ISA Server CD in the .\isa\i386 folder.

Configure the ISA Server

On the ISA Server computer, you must allow SMTP Server publishing, by using the ISA Server Mail Server Security Wizard to publish the virtual SMTP Server. In Scenario 1, you will publish the virtual SMTP Server located on the IIS computer. In Scenario 2, you will publish the virtual SMTP Server located on the Exchange Server computer.

To publish the virtual SMTP Server

In the console tree of ISA Server, click Internet Security and Acceleration Server, click Servers and Arrays, click the applicable array, click Publishing, right-click Server Publishing Rules, and then click Secure Mail Server.
Click Next.
On Mail Services Selection, select the protocols to publish and then click Next.
On ISA Server's External IP Address, type an external IP address on the ISA Server computer and then click Next.
On Internal Mail Server Address, select At this IP address, and then type the IP address as appropriate (depending on the configuration scenario). Click Next and then click Finish.

Distributed COM and ISA Server

The ISA Server SMTP filter transmits data over Distributed COM (DCOM). Make sure that DCOM is working properly between ISA Server and the server where the SMTP message screener is installed. Also, you need to consider carefully the security implications of using DCOM when configuring it.

To configure DCOM on ISA Server

Open the Distributed COM Configuration utility by typing dcomcnfg.exe at a command prompt.
On the Applications tab, select VendorData class and then click Properties.
On the Security tab, select Use custom access permissions, Use custom launch permissions, and Use custom configuration permissions.

For each of these permission settings:
1. Click Edit.
2. In Registry Value Permissions, click Add, select Everyone, and then click OK.
3. On Registry Value Permissions, in Type of Access, select
  
  Allow Access when setting access permissions,
  Allow Launch when setting launch permissions,
  Full Control when setting configuration permissions.

This grants access permission only to internal users, who are specifically familiar with the Vendor Class's program ID, to add COM objects to the ISA Server computer.

The dialog box below illustrates the registry value settings after you configure permissions for Everyone.

Relay Considerations

The Exchange Server can be used as a relay for inbound and outbound SMTP traffic. For maximum security, you should use Exchange 2000 as the endpoint server in your organization with the Message Screener installed on this computer. In this case, configure the relay option to allow only computers and domains in your organization.

To configure the relay option for Exchange Server

To open the Internet Services Manager, click Start, click Programs, click Microsoft Exchange, and then click System Manager.
In the console tree of System Manager, click System Manager, click Servers, click the applicable server, and then right-click Default SMTP Virtual Server.
Click Properties.
On the Access tab, click Relay.
In Relay Restrictions, select Only the list below. Add the computers and domains in your organization.

If you are using other mail servers for client mail boxes, the server with the message screener (virtual SMTP server) should be configured to route the SMTP traffic to those servers. For more information on routing configuration, refer to Microsoft SMTP Service in Microsoft Windows Help.

Administering the SMTP Filter

You can use ISA Management to configure the SMTP filter and the message screener.

To configure the SMTP filter and the message screener

In ISA Management, click Internet Security and Acceleration Server, click Servers and Arrays, click the applicable array, click Extensions, and then click Application Filters.
In the details pane, right-click SMTP Filter, and then click Properties.
To configure the message screener, click the Attachments tab or the Keywords tab and set the fields appropriately. For more information, see the ISA Server Help.
To configure the SMTP Filter, click the Users/Domains tab or the SMTP Commands tab and set the fields appropriately. For more information, see the ISA Server Help.

Note: For ISA Server Enterprise Edition, when ISA Server is installed as an array member, you must have permissions to modify the enterprise configuration in order to modify and configure the SMTP filter. This is because the SMTP filter applies to all the arrays in the enterprise.

Outlook Web Access

You can also publish a Web server to allow Outlook clients Web access. In this case, the Outlook Web server is published as a Web server and not as a mail server. This section describes how to publish a Web server to allow Outlook clients access to their e-mail.

To set up Microsoft Outlook Web Access (OWA), first use ISA Server to create destination sets which should include the paths that are being published on the Web server hosting OWA. Perform the following steps:

In ISA Management, click Internet Security and Acceleration Server, click Servers and Arrays, click the applicable array, click Policy Elements, right-click Destination Sets, point to New, and then click Set.
In Name, type the following as the name of the destination set:

OWA
Click Add, and then click IP address. In From and To, type the IP address of the Web server hosting OWA. In Path, type the following, and then click OK:

/exchange/*
Click Add, and then click IP address. In From and To, type the IP address of the Web server hosting OWA. In Path, type the following, and then click OK:

/exchweb/*
Click Add, and then click IP address. In From and To, type the IP address of the Web server hosting OWA. In Path, type the following, and then click OK:

/public/*

Next, use ISA Management to create a Web publishing rule that publishes the destination set. Perform the following steps:

In ISA Management, click Internet Security and Acceleration Server, click Servers and Arrays, click the applicable array, click Publishing, right-click Web Publishing Rules, point to New, and then click Rule.
In Name, type OWA Access Rule as the name of the rule, and then click Next.
In This rule applies to, select Specified Destination Set, and select the destination set named OWA. Click Next.
Apply the rule to Any Request, and click Next.
Select Redirect the request to this internal Web server. In Destination Site, type the name or the IP address of the Web server hosting OWA.
Select the Send the original header to publishing server instead of the actual one, and then click Next. Be sure that the virtual Web site has the same name that users from the internet connect to.
Click Finish.

For More Information

Microsoft will continue to share information with customers in hopes that it will help you deploy and manage ISA Server more successfully.

For the latest information on ISA Server, go to https://www.microsoft.com/isaserver.

For the latest information on other .NET Enterprise servers, go to https://www.microsoft.com/net/.

For the latest information on Windows 2000 Professional and Windows 2000 Advanced Server, go to https://www.microsoft.com/windows.

For support information and self-help tools for Microsoft products on the Microsoft Knowledge Base, go to https://search.support.microsoft.com/search/?adv=1.

For more information on Exchange Server, go to https://www.microsoft.com/exchange.

For more information on SMTP and message routing, go to https://www.microsoft.com/Exchange/en/55/help/default.asp?url=/Exchange/en/55/help/documents/server/XCP07008.HTM.

For more information on Exchange Server's feature front-end/back-end topology feature, which allows Exchange Server to proxy requests to back-end servers, see the white paper located on https://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=AFAD8426-572E-40F8-99DA-EB7198F374C4.