Configuring Firewalls

  • Article

Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2

The most secure firewall configuration is one in which the firewall permits only IKE and IPSec traffic to flow between the specific IP addresses of the peers. However, if these addresses are not static, or if there are many addresses, a less secure configuration might be required to permit IPSec and IKE traffic to flow between subnets.

When a firewall or filtering router exists between IPSec peers, it must be configured to forward IPSec traffic on UDP source and destination port 500, IP protocol 50 (ESP), or IP protocol 51 (AH). If you are using IPSec NAT-T, the firewall or filtering router must also be configured to forward IPSec traffic on UDP source and destination port 4500.

First, to permit IPSec traffic on UDP source and destination port 500, use the following settings to create a firewall filter called Permit ISAKMP traffic on UDP port 500:

  • Source address = Specific_IP_address

  • Destination address = Specific_IP_address

  • Protocol = UDP

  • Source port = 500

  • Destination port = 500

If you are using IPSec NAT-T, to permit traffic on UDP source and destination port 4500, use the following settings to create a firewall filter called Permit ISAKMP traffic on UDP port 4500:

  • Source address = Specific_IP_address

  • Destination address = Specific_IP_address

  • Protocol = UDP

  • Source port = 4500

  • Destination port = 4500

The firewall filter must also permit or track fragments for ISAKMP. IKE with certificate or Kerberos authentication requires ISAKMP packets to be fragmented because the IKE protocol uses UDP. ISAKMP messages that are larger than the local interface MTU are automatically fragmented by IP. If only certificate authentication is used, Windows Server 2003 implements a method to avoid IKE message fragmentation. When certificate authentication is used for communication between computers running Windows Server 2003 IPSec and Windows XP IPSec or Windows 2000 IPSec, fragmentation is required.

You must also allow IKE to be initiated from either a source or destination IP address. RFC 2408, Internet Security Association and Key Management Protocol (ISAKMP), specifies that the IKE protocol must be able to negotiate security in either direction. Stateful filtering that allows only one computer to initiate IKE to a responder typically times out and deletes the stateful inbound filter in the firewall. As a result, IKE cannot rekey IPSec security associations, and IPSec connectivity is lost.

During the security negotiation, IKE detects whether IPSec NAT-T is required, and, if so, IKE automatically uses UDP port 4500 as the source and destination port. However, the exact firewall configuration that is required to permit this traffic depends on the location of the network address translator relative to the firewall. Typically, the network address translator modifies the source address and the source port of the IKE UDP header.

To permit IPSec traffic on IP protocol 50 (ESP) or IP protocol 51 (AH), use the following setting to create a firewall filter called Permit IPSec traffic on ESP or AH protocol (50 or 51):

  • Source address = Specific_IP_address

  • Destination address = Specific_IP_address

  • Protocol = 50 or 51

When TCP traffic is protected by IPSec transport mode or tunnel mode, it should not require fragmentation. But UDP traffic that is protected by IPSec might require fragmentation. Therefore, you might need to enable fragment tracking for protocol 50 (ESP) or protocol 51 (AH).

  • ESP and AH traffic should be allowed to flow in either direction. The upper layer protocols such as TCP will determine when IPSec packets are generated. However, ESP fully encapsulates the TCP header, so stateful filters might not be able to inspect the TCP header to track the TCP connection state and determine when to close the hole. IPSec SA state is controlled by IKE in the encrypted quick mode negotiation. Accordingly, IPSec SA creation and deletion can occur in either direction and cannot be detected by inspection of the IKE negotiation packets. Stateful filtering on the IPSec protocols might impact connectivity. To assess the possible impact, ensure that you test how effectively IPSec protects your application traffic with the appropriate firewall or filtering router.

After the firewall is opened to allow the IKE and IPSec protocols, the firewall might not be able to inspect packets to control which traffic is secured by IPSec. IPSec policy filters determine which traffic IPSec can secure, so if you want only a specific protocol to flow between two peers, you must create IPSec filters that enforce this behavior. Port-specific filters can control the direction in which connections are made. Therefore, if a computer is in a more trusted network (inside the firewall) and you want IPSec to secure traffic only over certain protocols and ports on that computer, do not enable the default response rule in the IPSec policy for that computer. If the default response rule is enabled and an attacker compromises the remote computer, the attacker might be able to modify the IPSec policy to negotiate security for all traffic through the firewall.