Troubleshooting Windows Firewall with Advanced Security and Office Communications Server

Installing Office Communications Server Defines Firewall Rules for You

The Advanced Firewall enables you to apply specific settings. There are three profiles that you can choose from: public, private, and domain. Typically, these profiles are automatically detected and set as active. If the computer is a domain member, the domain profile is set to active. In Windows Server 2008 there can be only one active profile at a time.

When creating a rule, either by using the Firewall Rule Wizard or using the command-line tool, NETSH, you will configure the following settings. (I have listed the most commonly used settings; there are others used for more advanced scenarios):

• Rule Name. Unique, and aids in identifying what this rule is and what it is for.

• Group. If present, is defined to associate a rule with a larger group of rules, such as "core networking" or "distributed transacting coordinator".

• Profile. Defines what profile this rule is active with and applies the rule to the Public, Private, or Domain profile.

• Enabled. Is either true or false, and indicates if the rule is currently able to be active.

• Action. Indicates if the rule will either allow or deny the rule behavior.

• Override. Indicates if a particular rule is overriding the default behavior, for example all server message block (SMB) traffic is denied, unless the traffic is pre-authenticated. This overrides the first rule that denies all SMB traffic.

• Program. Fully qualified path on the local computer of the program to which this rule is allowing traffic.

• Local Address. The IP address of the local computer to which traffic is being allowed. Useful when you have multiple IP addresses configured on one local computer.

• Remote Address. Indicates what address traffic will be accepted from by using IP address networking.

• Protocol. Can be any of the well-known TCP/IP protocols.

• Local Port. Specifies what port on the local computer will be allowed by this rule.

• Remote Port. Specifies what port will be allowed from a sender.

During the installation of Office Communications Server 2007 R2, definitions are created for the firewall to ensure that the proper communication is open. For example, the Office Communications Server 2007 R2 Installation Wizard creates a rule on the Standard Edition Server or Enterprise Edition front end server to allow traffic from outside the Windows Server 2008 system to the RTCSrv.exe program. This rule is constructed programmatically during installation and is defined as the fully qualified path to RTCSrv.exe. The rule also allows any profile, any TCP protocol and port, and any remote or local IP address to connect to RTCSrv.exe.

It might be of interest to know that logo requirements for Windows Server 2008 and Windows Server 2008 R2 require that applications (in a more generic sense) perform one of the following:

• Open required ports and protocols by creating rules in the firewall.

• Provide guidance on how to create rules needed by the application and notify the administrator during the installation process of where product documentation can be located.

Office Communications Server 2007 R2 automatically defines firewall rules for the most common scenarios to ensure servers can communicate with each other. SQL Server 2008, on the other hand, refers the administrator to documentation during installation for how to create the necessary firewall rules.

However, there are a few things that weren’t thought of, or are just not part of what would be considered the responsibility of the Installation Wizard to create rules for. Often times, the decision is best left up to the administrator. And, here is where things can get a little tricky.

Figuring Out Why Something Just Doesn’t Work and Creating Custom Rules

Information Technology has become complex enough that most people are specialized in their roles. What this means is that the typical server or Office Communications Server administrator is not going to have all the depth skills necessary for firewall management. Learning the basics of how the firewall operates in Windows Server 2008 and 2008 R2, or at least how it affects the operation of your role applications, is now an important skill to attain.

Regardless of your specialty, here you are with a shiny new server running Windows Server 2008 with Office Communications Server 2007 R2, and you need to make firewall rule changes. For example, adding the first Office Communications Server in a new pool. You want to log on to the new server by using the administration tool from you Windows Server 2003 x64-based pool.

As luck would have it, the administration tool says that it can’t connect to your new server. Huh? You’re looking right at it, and it is working fine. You bring up the administration tool on the local computer, and it works just fine. So, you try to connect to your existing pool from the new server. You’re able to connect to the existing pool without a problem.

What is going on here? You check networking and discover that you can do many other things that you would expect to and your test clients are able to connect to the new pool. It is networked properly. Why can you connect to the Windows Server 2003 pool, but not to the Server 2008-based machine? Server 2003 has a different, and much more lenient firewall - which allows the connection from the Server 2008 machine. So, what do you do? Shut off the firewall on the Server 2008 machine? Of course not! You figure out why the connection is not allowed and design a rule to allow the connection!

You might remember that there are some logging capabilities in the Advanced Firewall. This is a good time to turn on logging on the remote pool and find out why you are unable to connect from the existing pool. To do this, perform the following procedure.

To turn on logging

1. Click Start, then Administrative Tools, then Windows Firewall with Advanced Security.

2. From within the Windows Firewall with Advanced Security MMC, right-click Windows Firewall with Advanced Security.

3. Click Properties.

4. Select Customize for one or more profiles (listed on tabs as Domain Profile, Private Profile, Public Profile) (Figure 1).

   Figure 1. Turning on logging for the firewall profiles

   

5. Enable logging on selected profiles by clicking the Log dropped packets drop-down menu, and then clicking Yes (Figure 2).

   Settings to enable the logging of dropped packets

   

6. Click OK.

7. Select other profiles for which you want enable logging and repeat the previous steps until done.

Notice that in Figure 2 there are a number of options that are available. Name defines the log file that the firewall will log information to. By default, the log file is located in %systemdrive%\system32\LogFiles\Firewall and is named pfirewall.log. Each profile that you enable for logging will write to this file. Log dropped packets is a Yes or No option. By default, it is set to No. If you want to log dropped packets for this profile, you must set it to Yes. Log successful connections is set to Yes if you want any successful connection to be logged to the firewall log. By default, this is set to No, no connections will be logged.

After you enable logging ,you can retry connecting the Administration console to the remote pool. The dropped packets are going to be logged in the firewall log. This is the kind of thing that you’re looking for! The dropped packets are what are preventing your administration console from connecting to the remote pool. If you take the information (such as the source IP address, the destination IP address, the protocol, and the port) that you are seeing in the dropped packets and convert it to a new Allow rule, the administration console should be able to connect.

There are some caveats. If you are connecting to a program that uses a dynamic port (in other words, the port is allocated at the time of connection), you have two options. You can open all ports from 1025 to 65536. Doing this basically opens the entire possible range for dynamic ports and is often used in Distributed COM (component object model) programs. Or, you can use the Programs option to select the program and allow all ports. If you want to allow only connections from a computer or computers, subnet scope, address range, or users - these can be defined as well. So when would you use a computer, a user, a subnet scope, or address range? Some examples might help.

• Computers. The firewall rule configuration enables you to define source or destination computers by name. On the Computers tab in the properties of a rule, and when you define a rule, you have the option to select computers you will allow by selecting Authorized Computers or Only allow connections from these computers or computers to deny by selecting Exceptions or Skip this rule for connections from these computers. You can use computer groups that you have previously created in the selection. You must have configured the rule for authentication using NTLM, Kerberos, or certificate-based authentication to specify computers.

• Users. Much like the computers, you can configure users who have authorized access, or you can define users to deny. Like computers, you can select user groups, and you must have already enabled the rule for authentication.

• Scope. Scope enables you to define from which subnets, hosts, or IP address ranges traffic will be accepted. The subnet entry is configured on the Scope tab and is in the form of <network portion of IP address>/<bits in subnet mask>. This is also known as classless inter-domain routing or CIDR. For example, if the network is 192.168.10.0 and the subnet mask is 255.255.248.0, this would yield a CIDR notation of 192.168.10.0/21, meaning that there are 21 bits for the network portion of the IP address and 11 bits for host addresses. Instead of a subnet entry, you can also input a single host address by specifying the entire IP address of the host computer.

• IP Address Range. This is also on the Scope tab. If you select This IP address range, you can define a range of IP addresses, rather than a subnet or a single host. For example, you want all servers to be accepted by the rule, but no workstations. Your servers are using 192.168.10.1 to 192.168.10.20. If you input these IP address in the From and To entry fields, only these computers will match the rule and be allowed. The workstations will be denied.

The recommended approach is going to change, based on what the circumstances are. Generally, it’s a good idea to use subnets when on your internal network, and a defined IP address when the computer is in a perimeter network. If you want only specific computers to connect and use dynamic host configuration protocol, this might be a good use of computer or computer group authentication for defining what can connect using a given rule.

Example Scenario to Illustrate the Basic Process of Troubleshooting a Connection Problem

Here’s an actual case where the administration console can’t connect to another server in the enterprise. An administrator at Contoso, Inc. has a pool and a Director. When the administrator logs on to the pool and attempts to connect to the Director, the administrator sees the following as shown in Figure 3.

Figure 3. Administration console cannot connect to Director

Obviously, the console is having a problem connecting to the director (Figure 3). If the Advanced Firewall is active, pinging the Director won’t help, because the firewall will drop the ping (or drop the ICMP message) by default. This isn’t the problem because you first need to determine if the firewall is configured correctly. You can determine this by running a validation from the pool, and the pool validation will show that the director is listening and responding properly.

This is a good time to activate the logging of dropped packets. If you open the Advanced Firewall on the director, turn on logging, and try to connect again, you will see hundreds of lines being written to the log file as shown in Figure 4. You will have to note the time, run your attempt to connect again, and parse through the log again to figure out what’s going on.

Figure 4. Firewall Log file

There is a lot of information in a busy network. Most network cards get a lot of traffic from many sources, especially broadcast and multicast traffic. The only traffic that really matters to you right now is the traffic between the pool server and the director.

Fortunately, there is a nice and free tool (however, if you are going to use it for commercial purposes, you should look into registering it) that is a graphical tail tool for Windows, and it works great with Windows Server 2008. For the uninitiated, tail is a very popular command in Unix. The purpose of tail is to display the end of a file, rather than the beginning, which is typical of tools like notepad or the type command.

The name of the Windows tool is mTail (available at https://ophilipp.free.fr/op_tail.htm), and it has some interesting features for just this kind of use. When you start mTail, you can open a file (in our case, %systemdrive%\system32\LogFiles\Firewall\pfirewall.log) and it will display the end of the file. Then as new entries are added to the file, they are displayed in real time. Additionally, there is a filter capability to display only matching elements, color matching lines, an alert capability, and a DNS lookup option to replace those IP addresses with host names or fully qualified domain names.

In our example, you would start mTail on the director and then filter the results based on the address of your pool (the source of the connection attempts). Now you get just the conversation between the pool and the Director. It seems clear that our problem is the dropped TCP packets from the pool to the Director.

As a refresher, the firewall log file has the following format: date time action protocol src-ip dst-ip src-port dst-port size

With the following additional fields at the end of each line: tcpflags tcpsyn tcpack tcpwin icmptype icmpcode info path

Your real interest is with the action, protocol, src-ip, dest-ip, src-port, and dst-port fields. Figure 5 shows the line in mTail that should catch your eye as the probable root cause.

Figure 5. mTail results

The packet that was dropped is of type TCP, and it came from contoso-fe.contoso.com (this is the front end server in the pool). This packet was destined for ContosoDirector.contoso.com. The source port is 56626 and the destination port is 49154.

The next task is to take this information and convert it into a rule on the Director that will allow the traffic that is being dropped. To do this, perform the following procedure.

To create a rule on the Director

1. Click Start, click Administrative Tools, and then click Windows Firewall with Advanced Security.

2. Right-click Inbound Rules, and then select New Rule as shown in Figure 6.

   Figure 6. Open Windows Firewall with Advanced Security

   

3. Under Rule Type, select Port, and then click Next.

4. On the Protocols and Ports page, select TCP and enter 49154 in the Specific local ports field. Click Next. (Note that your port may be different.)

5. Under Action, select Allow the connection, and then click Next.

6. Under Profile, select the profiles to which you want this rule to apply. In this case, Domain is the only required profile.

7. Under Name, type the name and a description of the rule. For this example, the rule is named "Allow Office Communications Server 2007 R2 console from pool". In this example there is no description, but it’s a good idea to describe what this rule is solving and what ports and protocols it affects.

8. Click Finish.

After you configure the rule, apply it and try connecting. Voila! This time the console connects and you can control the Director from the main console on the pool server as shown in Figure 7. Repeat this process for each pool server if you don’t have a specific server from which you manage the Director.

Figure 7. Successful connection

You are also likely to have the same issue if you use the console from your desktop computer. To resolve the problem for a larger scope, you can configure the Advanced Firewall to allow an IP subnet rather than just a specific computer. Select the rule and then right-click. Click the Scope tab. Select These IP addresses, and then click Add. The next window enables you to define an IP address or subnet, with the subnet allowing for classless inter-domain routing (CIDR) notation. For the Contoso network, we want to allow any internal network computer to connect, so in the This IP address or subnet field, type 192.168.10.0/24. This means that the network portion of the IP address is 192.168.10, with a 24-bit (or a 255.255.255.0) subnet.

Summary

In this article, I’ve identified some tools and techniques to use with Windows Server 2008 and the Windows Firewall with Advanced Security to resolve problems where the Admini Tools console cannot connect to the Director or other Office Communications Servers. By turning on firewall logging, and using mTail to review the firewall log, you can figure out why the Admin Tools cannot connect to your Office Communications Servers. Once the root cause is discovered, it’s a simple matter of creating a firewall rule on each Windows Server 2008 running Office Communications Server to resolve the issue. The rest is up to you! Read up on the firewall and what you should do to resolve other problems as they arise. Remember, disabling the firewall is going to reduce your Defense in Depth strategy, and I would strongly recommend against shutting off that firewall!

Additional Resources

• Windows Firewall with Advanced Security and IPsec at https://go.microsoft.com/fwlink/?linkid=161149

• Tools and Procedures Used to Troubleshoot Windows Firewall at https://go.microsoft.com/fwlink/?LinkId=166902

• mTail-Windows Tail Program at https://ophilipp.free.fr/op_tail.htm

Office Communications Server Resources

• Visit the Office Communications Server main page at https://go.microsoft.com/fwlink/?LinkId=132607.

• View the complete Office Communications Server documentation library at https://go.microsoft.com/fwlink/?LinkId=132106.

• Follow tweets from the Office Communications Server team at https://go.microsoft.com/fwlink/?LinkId=167909.

• Download all the Office Communications Server content as a Word document at https://go.microsoft.com/fwlink/?LinkId=133609.

• Download all the Office Communications Server content as a compiled help file at https://go.microsoft.com/fwlink/?LinkId=160355. (Make sure you scroll down to the Additional Information section to download OCSDocumentation.chm.)