Troubleshooting Configuration Storage Servers in ISA Server 2004 Enterprise Edition

  • Article

This document provides general guidelines and recommendations for troubleshooting issues encountered during the installation, replication, and administration of the Configuration Storage server component of Microsoft Internet Security and Acceleration (ISA) Server 2004 and ISA Server 2006 Enterprise Edition. This document is organized as follows:

  • Introduction to the Configuration Storage server concept.
  • Troubleshooting connectivity issues. Many problems encountered are due to connectivity issues with the Configuration Storage server. This section of the document provides troubleshooting steps for you to follow.
  • Specific issues. Troubleshooting other common issues encountered when you install, replicate or manage a Configuration Storage server.
  • An appendix with information about useful tools and scripts.

Troubleshooting

Troubleshooting Steps for Connectivity Issues

Appendix

ChangeStorageServer.exe

Additional Resources

The Configuration Storage server component of ISA Server 2004 and ISA Server 2006 Enterprise Edition is the repository of the enterprise layout and the configuration for each server in the enterprise. This repository is an instance of Active Directory Application Mode (ADAM). When you install the Configuration Storage server component during ISA Server Setup, you also install ADAM on the designated Configuration Storage server computer. You do not need to administer ADAM directly. All management of the Configuration Storage server is done using the ISA Server Management console.

Multiple Configuration Storage servers can be deployed in the enterprise. Each ISA Server array member points to a specific Configuration Storage server, from which it receives updated configuration settings. This configuration occurs at array level, so you cannot configure two different Configuration Storage servers for two members of the same array. However, you can specify an alternate Configuration Storage server that an array member should use in case the first Configuration Storage server fails. A situation can arise in which one array member is using the primary Configuration Storage server, while another member of the same array is using the alternate Configuration Storage server. Such a situation is usually a temporary issue with the primary server, and eventually all array members will switch to using it.

Each ISA Server has a local copy of its configuration settings stored in ADAM. This information includes relevant portions of the enterprise configuration, array configuration information, and server-specific information. If a Configuration Storage server fails, ISA Server Management console will not provide access to any server functionality because it requires a connection to a working Configuration Storage server. At the same time, the ISA Server computer will continue to provide firewall, VPN, and proxy services based on the last known configuration it received from the Configuration Storage server. However, you will not be able to monitor or change the ISA Server configuration until the Configuration Storage server is restored, or until you connect to a different Configuration Storage server in the enterprise. When the Configuration Storage server is back online, array members connect and synchronize automatically.

A single Configuration Storage server can store firewall policy for multiple ISA Server arrays in a number of different configuration scenarios, as follows:

  • The Configuration Storage server and ISA Server are installed on the same computer, which is configured in workgroup mode or as a domain member.
  • The Configuration Storage server and ISA Server array members are installed on different computers, with any of the following configurations:
    • Both the Configuration Storage server and ISA Server array members belong to the same domain or to trusted domains.
    • The Configuration Storage server and ISA Server array members are members of different domains, with no trust between them.
    • Both the Configuration Storage server and ISA Server array members are installed in workgroup mode.
    • The Configuration Storage server is installed in workgroup mode, and ISA Server array members belong to a domain.
    • The Configuration Storage server is a domain member, and ISA Server array members are installed in workgroup mode.

In a workgroup scenario, server certificates are used for authentication between ISA Server array members and the Configuration Storage server. If you are using certificates in a workgroup scenario, and then move to a domain configuration, you can continue to use certificate authentication. Moving to Windows authentication (Kerberos) in such a scenario is not supported.

For more information on deploying Configuration Storage server, see the following resources:

The information in these documents also applies to ISA Server 2006 Enterprise Edition.

During normal ISA Server operations, ADAM does not require direct administration. If you do need to troubleshoot ADAM issues, see the following resources:

Troubleshooting

Most issues with Configuration Storage server result in connectivity problems, with their source in one of the following areas:

  • Physical connectivity
  • Name resolution
  • Flawed credentials

This section provides the following troubleshooting information:

  • General troubleshooting steps to follow for all connectivity issues.
  • Troubleshooting tips and hints for other common issues.

Troubleshooting Steps for Connectivity Issues

Connectivity issues with the Configuration Storage server may occur during Setup and uninstall, or when managing and configuring ISA Server. Connectivity issues can be related to physical network problems, failed name resolution, or service availability. Losing connectivity with the Configuration Storage server is not fatal for the operation of ISA Server services, which continue using the local copy of the configuration settings. However, connectivity to the Configuration Storage server is required to keep the local copy of the configuration up to date, which may be crucial for correct operations and protection of the organization’s network resources.

Connectivity issues may manifest themselves with a number of different errors and events. The most common types of errors include the following:

  • Errors indicating that configuration changes cannot be saved or loaded
  • Error messages specifying that array members cannot connect to a specified Configuration Storage server
  • Name resolution errors
  • Errors with the ISASTGCTRL service of Configuration Storage server
  • Authentication issues that arise when ISA Server array members cannot authenticate with the Configuration Storage server. In workgroup scenarios, or across untrusted domains, issues might be related to incorrect certificate configuration.

The following troubleshooting steps can be used whenever there is a problem with connection to the Configuration Storage server. Perform each troubleshooting step in order. If a problem is diagnosed, fix appropriately, and then re-check connectivity. If the problem persists, continue on to the next step.

Step 1: Verify Configuration Storage Server Name Settings

To verify that the Configuration Storage server name is specified correctly

Verify that the Configuration Storage server name specified in ISA Server Management console is correct. By default the ISA Server Management console uses the credentials of the logged on user and the Configuration Storage server name that is specified for the array members. To access a different Configuration Storage server, you may need to provide the name of the server and credentials. Alternatively, run the ISA Server Management console on the Configuration Storage server. Verify the Configuration Storage server name as follows:

  • In the console tree of ISA Server Management, click Microsoft Internet Security and Acceleration Server 2004 or Microsoft Internet Security and Acceleration Server 2006, click Arrays, right-click the specific array, and then click Properties.
  • On the Configuration Storage tab, in Configuration Storage server, verify that the fully qualified domain name (FQDN) is correctly specified.

Step 2: Verify Name Resolution Settings

To verify that the forward name lookup is properly configured on the computer running ISA Server services

At the command prompt, type:

ping name

(where name is the name of the Configuration Storage server).

Note that when the ISA Server computer is installed in workgroup mode, ISA Server may not be able to resolve the name of the Configuration Storage server with a DNS query (DnsQuery_W), even though ping is successful. In this case, events 21257 and 21271 may be logged in the Application log of the Windows Event Viewer.

To resolve any name resolution issues, ensure that the DNS server used by ISA Server has an entry to resolve the name of the Configuration Storage server.

Step 3: Check Service Availability

To check that the ISASTGCTRL service of Configuration Storage server is available

  • In the console tree of ISA Server Management, click Microsoft Internet Security and Acceleration Server 2004 or Microsoft Internet Security and Acceleration Server 2006, click Arrays, right-click the specific array, and then click Properties. On the Configuration Storage tab, in Configuration Storage server, note the name of the server.
  • Verify that the Configuration Storage server computer that you noted is available, and has the ISASTGCTRL service for Configuration Storage server running.
  • If you cannot connect to the Configuration Storage server with the ISA Server Management console, check that the Configuration Storage server service is available. Open the ISA Server Management console locally on the Configuration Storage server computer. If the local ISA Server Management console cannot connect to the local Configuration Storage server, verify that the ISASTGCTRL service of Configuration Storage server is running. To do this, type: net start ISASTGCTRL at the command line.

Step 4: Check Firewall Policy Rules

To check the firewall log and verify that firewall policy rules are not blocking access to the Configuration Storage server

Perform the following steps if the log indicates that access is blocked.

  1. Verify that the system policy rule Allow remote access to Configuration Storage servers is enabled on the array member, and that the destination domain name set (specified in the To tab) includes the name of the required Configuration Storage server.

    Note

    Even when this system policy rule is correctly configured, access to the Configuration Storage server can still be blocked. This can be due to incorrectly configured networks in which legitimate packets are being blocked as spoofed, or some required protocol that is not specified in the system policy rule (for example, DNS required for name resolution) is being blocked. For more information on configuring networks, see Best Practices for Configuring Networks in ISA Server 2004.

    After creating the rule, the Configuration Agent cannot propagate the new configuration settings because access to the Configuration Storage server is blocked.

To propagate new configuration settings by stopping and starting the firewall

  1. In the console tree of ISA Server Management, click Microsoft Internet Security and Acceleration Server 2004 or Microsoft Internet Security and Acceleration Server 2006, click Arrays, and then click Monitoring. On the Services tab, right-click Microsoft Firewall, and then click Stop.

  2. Verify that the configuration was updated for the array. In the console tree of ISA Server Management, click Microsoft Internet Security and Acceleration Server 2004, click Arrays, and then click Monitoring. On the Configuration tab, verify that the Status column indicates Synced.

  3. Start the Microsoft Firewall service. In the console tree of ISA Server Management, click Microsoft Internet Security and Acceleration Server 2004, click Arrays, and then click Monitoring. On the Services tab, right-click Microsoft Firewall, and then click Start.

    Note

    As an alternative to stopping the Firewall service you can use the Fwengmon.exe tool. Using this tool, you can use the /Allow command to open the firewall for free access to an from a specified IP address. You must then cancel the free access using /NoAllow. Do this as follows:
    Download Fwengmon.exe from the .
    At the command prompt, type: fwengmon.exe /a IP, where IP is the IP address of the Configuration Storage server.
    After checking that the configuration was updated for the array, and that the status indicated Synched, at the command prompt, type: fwengmon.exe /noallow.

Step 5: Verify that Local System Account on Array Member Can Authenticate with ADAM

The LDP tool is used for general administration of an LDAP directory service such as ADAM. It is located in the %windir%\ADAM folder on the Configuration Storage server. Copy it to ISA Server array member computers for troubleshooting purposes. Check that you can connect and bind to ADAM using LDP.exe tool, and that the Local System account can authenticate. To do this, use the at command to run an instance of LDP.exe running in the Local System context.

To verify that Local System Account on array member can authenticate with ADAM

  1. Click Start, point to All Programs, point to ADAM, click ADAM Tools Command Prompt, and then type the following at the command line:at time /interactive ldp.exe (where time is the current time plus 1 minute)
  2. An LDP window running as the LocalSystem account will appear within 1 minute.
  3. On the Connection menu, click Connect.
  4. In Server, type the fully qualified domain name (FQDN) of the Configuration Storage server.
  5. In Port, type the following:
    • For Windows authentication, type 2171
    • For authentication over an SSL connection, type 2172, and then select SSL.
  6. On the Connection menu, click Bind. Do one of the following:
    • If you are using Windows authentication, verify that User, Password, and Domain are all empty, and then click OK.
    • If you are authenticating over an SSL connection (LDAPS), specify the credentials of an ISA Server Administrator account (either array administrator or enterprise administrator), and then click OK.
      If you have connected and authenticated successfully, you will be able to browse the ADAM directory with the same permissions as the Local System account.
      If you cannot connect or bind, this indicates a permissions issue. Check the following:
    • Verify that time on the computer running ISA Server services is the same as on the domain controller.
    • Verify that required Service Principle Names (SPNs) are properly registered. SPNs get created when ADAM service starts, and are created as an attribute on the User account running the ADAM service. For instructions see Administering ADAM service principal names topic in ADAM.chm help file located in %windir%\help folder on the Configuration Storage server computer.
      If you cannot connect over SSL, check the following:
    • Verify that a valid server certificate with the exact name as specified in the Array Properties page is installed on the Configuration Storage server computer. If a certificate is not installed or is invalid, then install a new certificate. To do this, either run Setup in Repair mode, or use the ISACertTool. For more information, see ISACertTool for ISA Server 2004 Enterprise Edition.
    • Verify that a valid root certificate of the certificate authority is installed on ISA Server array members running ISA Server services. If such a certificate is not installed or is invalid, then install a root certificate.
    • Verify that you have installed the update described in Knowledge Base article 894609: An update is available to prevent Configuration Storage server account settings from expiring when you use certificate authentication in ISA Server 2004 Enterprise Edition. This update addresses an issue caused by expiration of ADAM account settings. If this problem occurs, Event ID 21238: ISA Server cannot connect to the Configuration Storage server ConfigurationStorageServer_Name, may be issued.
    • If you manually installed a server certificate and did not use ISA Server Setup or ISACertTool.exe, the keyset file does not have read permissions for the ISASTGCTRL service account. If this is the issue, try uninstalling and then reinstalling the server certificate by running ISA Server Setup in Repair mode, or by using ISACertTool.exe.

Troubleshooting Specific Issues

Most issues that occur with the Configuration Storage server are the result of connectivity problems. To troubleshoot, refer to Troubleshooting Steps for Connectivity Issues in this document. You may also encounter issues when installing, uninstalling, replicating, or managing Configuration Storage servers. A description of common problems, along with the underlying issue and the solution, are presented in this section to help you troubleshoot and resolve these issues.

Installation and Replication Issues

The following information will assist you when installing, uninstalling, and replicating Configuration Storage server:

  • When Configuration Storage server is installed in workgroup mode, replication is not supported.
  • When installing a replicate Configuration Storage server, ensure the following:
    • The computer on which you are installing the replica should be a member of the same domain as the original Configuration Storage server, or there must be a trust relationship between the domains.
    • The replicate Configuration Storage server must belong to the enterprise-level Replicate Configuration Storage servers computer set. Otherwise, the system policy rules that, by default, allow access between the local and the replicate Configuration Storage servers will not apply to the new Configuration Storage server.
  • If you receive a message indicating that an object already exists in ADAM during replication, uninstall the ISA Server instance of ADAM from the Control Panel using Add/Remove Programs, and then run Setup again to install the replica.
  • During uninstall, you may receive a message that Configuration Storage server objects cannot be deleted when there is no connection to the Configuration Storage server. Because objects cannot be deleted from ADAM, the computer will still retain read permissions for ISA Server objects in ADAM at the end of uninstall. You can manually remove the server node using the ISA Server Management console (or COM objects) on one of the other array members, or on the Configuration Storage server computer.
  • A useful practice when encountering issues during installation is to check events in the Event Viewer, or use the ISA Server Setup logs to troubleshoot installation issues. For more information on ISA Server Setup log, see Knowledge Base article 837347: ISA Server Setup log files.

Configuration Storage Servers are Not Accessible.

Problem: Neither primary nor secondary Configuration Storage servers are accessible. You want to specify an alternative Configuration Storage server.

Cause: The Configuration Agent can switch to another Configuration Storage server only by reading a configuration change from the currently configured Configuration Storage server, which is not available.

Solution: You can specify an alternate Configuration Storage server by using the ChangeStorageServer.vbs script, available in the FPC\Program Files\Microsoft ISA Server folder on the ISA Server CD. For script usage instructions, run:

cscript ChangeStorageServer.vbs ?

For more information, see Appendix A: Useful Tools in this document.

Users Who Do Not Have Permissions Can Create an ISA Server Object

Problem: Users who have been removed from ISA Server Array Administrators group can still manipulate ISA Server rules and rule elements that they created.

Cause: Users who create objects in ISA Server are owners of those objects, and can grant themselves permissions on those objects.

Solution: When you revoke permissions for an ISA Server array administrator, ensure that you do the following:

  • On the ISA Server computer, delete the user account.
  • On the Configuration Storage server, review the ADAM objects created by the users. Modify ownership of objects that belong to the revoked accounts.

Error in Creating ADAM SCP Object When Configuration Storage Server is Installed on Domain Controller (Alias)

Problem: When running the Configuration Storage server on a domain controller, Event 2537 periodically appears in the Event Viewer:

\ The directory server has failed to create the ADAM serviceConnectionPoint

object in the Active Directory. This operation will be retried.

Cause: In an Active Directory environment, services can publish information about their existence using serviceConnectionPoint (SCP) objects. When an ADAM instance runs in such an environment, it makes a best effort attempt to publish updated information about itself in Active Directory using SCP. When the Configuration Storage server is installed on a domain controller, the ISASTGCTRL service runs as a domain account that does not have write access to the required location in Active Directory and the attempt to register an SCP fails. This does not prevent ADAM from running as required or accepting client connections. ISA Server does not use SCP information.

Solution: Although this issue does not interfere with ISA Server operations, it introduces a lot of noise in the Event Viewer because registration is attempted every hour. You can disable the SCP registration by adding the Distinguished Name (DN) of the [NTDS Settings] object of the instance to the msDS-DisableForInstances attribute on the SCP publication configuration object.

Synched Status is Not Indicated in the Configuration Status Page for Long Periods of Time

Problem: Why does it take so long for Configuration Status to show the status as Synched for synchronization between an array member and the Configuration Storage server?

Cause: Updating the configuration from the Configuration Storage server is handled by the Configuration Agent component running separately on each array member as part of ISACTRL service. The update process includes:

  • Copying the changes from the Configuration Storage server to the local registry-based cache.
  • Preparing a new copy of the effective configuration (a mix of enterprise and array configuration settings).
  • Uploading the new configuration to array members.

The status only shows as Synced after all these steps are completed.

Solution: In the Configuration Storage tab of the array properties, you can set when the Configuration Storage server checks for updates. Reducing this value will make the process start earlier, but all three phases of the update process still must be performed. The amount of time this takes depends on the size of the configuration changes.

Improving Replication Synchronization Times Between Configuration Storage Servers

Problem: Synchronization between replicate Configuration Storage servers is slow.

Cause: By default, intrasite replication takes place once every hour. This time interval can be customized in ADAM.

Solution: To configure replication frequency within a single ADAM site, use the AdamSites.exe tool. For more information, see ADAMSites Tool for ISA Server 2004 Enterprise Edition.

Appendix A: Useful Tools

ChangeStorageServer.vbs

ChangeStorageServer.vbs allows you to specify an alternative Configuration Storage server if the primary and alternate servers are not available. A situation can occur in which the primary Configuration Storage server is not available (for example, because of an unrecoverable hardware failure), and the alternate Configuration Storage server is also unavailable (or not configured). It is difficult to fix this issue because the only way that the Configuration Agent running on ISA Server array members can switch to an alternate Configuration Storage server is by reading a change from the current Configuration Storage server.

You can address this issue by running the ChangeStorageServer.vbs script, which is located in the FPC\Program Files\Microsoft ISA Server folder on the ISA Server CD. Run the script on all array members. For more information, see “To specify a Configuration Storage server for this array” in the ISA Server online Help.

Fwengmon.exe

The FWEngMon.exe tool allows you to analyze and troubleshoot firewall connectivity issues by monitoring the ISA Server kernel mode driver (fweng.sys). A set of command-line options provide a way of looking at the state of the ISA Server firewall engine at a specific point in time. You can open and close firewall access for a specified IP address range to unconditionally allow traffic to and from addresses in the range, and then cancel unconditional traffic when troubleshooting is complete. For more information on this tool, see Firewall Kernel Mode Tool for ISA Server 2004.

ISACertTool.exe

ISACertTool.exe allows you to change settings for Configuration Storage server authentication after installation. Use the tool to perform the following tasks:

  • Install a server certificate on the Configuration Storage server.
  • Install a root certificate on each array member to indicate that it trusts the Certification Authority that issued the server certificate.

For more information, see ISACertTool for ISA Server 2004 Enterprise Edition (https://go.microsoft.com/fwlink/?LinkId=82083) or ISACertTool for ISA Server 2006 Enterprise Edition (https://go.microsoft.com/fwlink/?LinkId=82084).

Additional Resources

For more information on ADAM, see Active Directory Application Mode (ADAM). Note that this link provides information on Windows Server 2003 R2 in Beta, and some procedures may vary from those described in this document.

For more information, see the ISA Server 2006 TechCenter (https://go.microsoft.com/fwlink/?LinkId=82085) and the ISA Server 2004 TechCenter (https://go.microsoft.com/fwlink/?LinkId=82086).