Usage of Repadmin When Troubleshooting Event ID 1311

Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2, Windows Server 2008, Windows Server 2008 R2

By all means, this topic is not about how to troubleshoot events that have Event ID 1311. In this topic, we are attempting to expose the various usage of repadmin while troubleshooting 1311 in Windows 2000 domains based on Microsoft Knowledge Base (KB) article 307593, How to Troubleshoot Event ID 1311 Messages on a Windows 2000 Professional Domain (https://go.microsoft.com/fwlink/?LinkId=121799). Some or all of the repadmin subcommands used here may be used in Windows Server 2003 environments as well.

The KB article RESOLUTION section has the following action plan. This topic examines how to apply the various repadmin subcommands against each action plan. All of the repadmin subcommands listed in this topic have associated examples either in this section or elsewhere in this document.

Resolution steps from the KB article

Action plan by using repadmin

Determine if the event ID 1311 messages are site-specific or forest-wide.

To determine the scope of event ID 1311 messages:

  1. First, find all the Inter Site Topology Generators (ISTG) in the forest.

  2. Then, examine the Directory Service logs of all the ISTG domain controllers in the forest.

To determine the ISTG’s, use Repadmin /ISTG.

Determine if site link bridging is turned on and if the network is fully routed.

To determine this, use repadmin /showattr (Determine if site link bridging is turned on).

Verify that all of the sites are defined in site links.

Every site defined in Active Directory must be hosted or reside in a site link.

The repadmin /showism command (Verify inter-site cost matrix and orphaned sites) is useful for locating improperly configured sites.

Detect and remove preferred bridgeheads.

To search for preferred bridgehead servers use repadmin /showattr (Determine if site link bridging is turned on).

Resolve Active Directory replication failures in the forest

When you want to discover and troubleshoot replication failures, the following repadmin subcommands can be useful:

Determine if source servers are overloaded.

A domain controller that is overloaded with a large number of direct replication partners or a replication schedule that is overly aggressive can create a backlog in which some partners never receive changes from a hub domain controller. The following subcommands can be useful in this situation:

Determine if site links are disjointed.

"Disjoint site links" is an Active Directory configuration in which the topology is broken into two or more parts in which some sites do not replicate because site definitions and site link definitions are incorrect. Disjoint site links are the most difficult improper configuration to troubleshoot. The following subcommands may be useful in this situation:

Delete connections if the KCC is in "Keep Connection Mode."

If the Knowledge Consistency Checker (KCC) builds a different path around a site-to-site connection failure, but it retries the failing connection every 15 minutes because it is in "connection keeping mode," delete all broken connections and let the KCC rebuild them. Wait two times the longest replication schedule in the forest.

Site link bridging is enabled in Active Directory if the following conditions are true:

  • The Bridge all site links check box is selected for the IP protocol and the SMTP protocol in the Active Directory Sites and Services snap-in.

  • The Options attribute for the IP protocol and the SMTP protocol is NULL or set to zero (0) for the following distinguished name (DN) paths:

    CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=root domain of forest

    CN=SMTP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=root domain of forest

Repadmin /showattr

There are two values that we could set from the graphical user interface (GUI): Ignore Schedules and Bridge all site links. In our example (figure 3.11.1), the IP transport has Bridge all site links enabled and SMTP transport has both values selected.

The following table lists the various values that the options attribute take.

Option value

Description

0X0

Only Bridge all site links is selected from the above

0X1

Both the values are selected

0X2

None selected

0X3

Only Ignore schedules is selected

Detect preferred bridgeheads

Preferred bridgeheads are selected when the following condition is true:

bridgeheadTransportList attribute is set to either one of the following values or both values:

  • CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=<root domain of forest>

  • CN=SMTP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=<root domain of forest>

By using repadmin /showattr, we are setting the base at configuration partition and applying a filter for server ObjectClass and looking for all of the domain controllers that have this value set to use either IP or SMTP transports.

If the search returns any results, note the name of server in the distinguished name path in which the bridgeheadTransportList attribute is populated.

Repadmin /showattr

In the example in figure 3.11.2, ROOTDC01 is selected as a preferred Bridgehead for IP transport in site HUB.

Verify inter-site cost matrix and orphaned sites

Repadmin /showism displays intersite messaging routes calculated by the Intersite Messaging Service and is very useful for locating improperly configured sites. This operation cannot be executed remotely.

As the KCC runs through the progressions of analyzing intersite site links and connections, it must query the Intersite Messaging Service (ISM) service to retrieve data about the network configuration to make intelligent decisions about routing changes.

To display cost and frequency configurations of replication between sites, use the following command:

Syntax

repadmin /showism [<TransportDN>] [/verbose]

Parameter

Description

<TransportDN>

Specifies whether the mail server is using SMTP or remote procedure call (RPC) to send messages.

/verbose

Lists detailed information.

Note

The repadmin.exe /showism cannot be executed against a remote domain controller.

Example: Display inter-site cost matrix

Repadmin /showism

How to interpret the data

  • Showism was used against the IP transport and hence the output is specific to IP.

  • If a specific transport is not specified, the output will contain both IP and SMTP details.

  • The numbers in an entry appear in the following order:

    • Cost: Replication interval: Options
  • There are four key pieces of information:

    • Text regarding the status of bridgehead servers.

    • Total cost between two sites. The cost value indicates the preference for a network link for replicating directory information between sites.

    • Frequency of replication in minutes between the two sites.

    • Options for each replication link.

  • In the example in figure 3.11.3, we have five sites and Bridge all site links is enabled, which means that site link transitivity is enabled. Therefore, if we see any "-1:0:0" entries for one or more covered Active Directory sites, we must ensure that the affected sites are listed in a site link. In this example, site Branch4 is not included in any site links and therefore disconnected from rest of the sites. Event 1311 will certainly occur here due to this configuration problem.

Fields of interest

Definition

"0:0:0"

Each site matrix contains one "0:0:0" entry that refers to itself.

"200:30:1"

An entry that contains positive numbers for the cost value and replication interval value (for example, "200:30:1" or "100:15:1") indicates that the site connection is good. Specifically in our example for Site BRANCH1

Site(0) CN=BRANCH1,CN=Sites,CN=Configuration,DC=contoso,DC=com

0:0:0, 200:30:1, 200:30:1, -1:0:0, 100:15:1

  • 200 stands for the cost to replicate from site(1) which is BRANCH2 that is an aggregate cost between two hops (100 + 100) because a direct replication link between the two sites does not exist.

  • 30 is the replication interval that is common between the two branches

  • 1 is the option on the site link which denotes “Change Notifications are enabled across the site link”

And so on the rest of the sites…

"-1:0:0"

A "-1:0:0" entry indicates that the site connection is not working. This occurs if one or more of the following conditions are true:

  • Site is not included in a site link.

  • Site does not host any domain controllers (this is known as an "uncovered" site).

  • Replication protocol is not used. For example, if SMTP replication is not configured, the entries in the SMTP portion of the /SHOWISM matrix all appear as "-1:0:0".

Note

  • If site link bridging is enabled and the repadmin /showism command returns a site with a full complement of "-1:0:0" entries and one "0:0:0" entry is orphaned unless the site is uncovered (no domain controllers reside in that site).

  • If site link bridging is disabled, "-1:0:0" entries are less meaningful. If this is the case, you must manually determine if each site is included in a site link. To do so, write down the list of sites and site links, and manually map each site to a site link.

  • Repadmin /failcache

    Repadmin /failcache displays a list of replication failures that KCC is aware of. Run this command from the console of each ISTG domain controller in the forest to discover replication failures for bridgeheads in the site for that ISTG.

    Syntax

    repadmin /failcache <DC_LIST>

    Parameter

    Description

    <DC_LIST>

    Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp.

    Example: Display replication failures that KCC is aware of

    The example in figure 3.11.4 shows sample output from the repadmin /failcache command.

    Repadmin /failcache

    The output from the repadmin /failcache command is divided into two sections explained in the following table.

    KCC Link Failures

    Lists errors for existing connection links. The ISTG domain controller imports showreps ("repsfroms") data for every bridgehead server in its site. However, the ISTG domain controller does not list errors. The link failure cache is emptied at the beginning of every KCC run and refilled during the course of the current run.

    KCC Connection Failures

    Lists unsuccessful attempts to build connection objects between domain controllers ("reps from" or "reps to"). When you run the repadmin /failcache command from the ISTG domain controller, it lists entries that are imported from bridgeheads in the site. At the beginning of each KCC run, the KCC examines each entry in the connection failure cache and tries to DsBind to the failing server. If the bind succeeds, the entry is removed.

    In the example in figure 3.11.4.1, the failures are a result of some topology changes from the past and would continue to exist due to the value of the replTopologyStayOfExecution attribute, which determines how long domain controller metadata is retained in Active Directory after a domain controller has been removed.

    Example: Output when there are no failures

    When there are no failures, the output should appears as it does in figure 3.11.4.2.

    Repadmin /failcache

    Note

    The repadmin /failcache command differs from the repadmin /showrepl command in two ways:

    • The repadmin /showrepl command shows the naming context that is failing.

    • The repadmin /failcache command does not.

    Repadmin /KCC

    Repadmin /KCC forces the KCC to recalculate replication topology for a specified domain controller. By default, this recalculation occurs every 15 minutes.

    Syntax

    repadmin /kcc <DC_LIST> [/async]

    Parameter

    Description

    <DC_LIST>

    Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp.

    /async

    Specifies that replication will be asynchronous. This means that repadmin starts the replication event, but it does not expect an immediate response from the destination domain controller. Use this parameter to start the KCC and not wait for it to finish.

    Example 1: Running the KCC on the local domain controller

    Repadmin /kcc

    Example 2: Running the KCC against the ISTG of the HUB site

    Repadmin /kcc /fsmo_istg

    Example 3: Running the KCC against all the global catalog servers in the forest

    Repadmin /kcc /gc

    Example 4: Running the KCC against all the domain controllers in the BRANCH2 site

    Repadmin /kcc /site

    Repadmin /ISTG

    Repadmin /ISTG returns the server name of the ISTG server for a specified site.

    Syntax

    repadmin /istg <DC_LIST> [/verbose]

    Parameters

    Descriptions

    <DC_LIST>

    Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp.

    /verbose

    Lists detailed information.

    Example: Display ISTGs in my environment

    Repadmin /istg /verbose

    In the example in figure 3.11.6, the ISTGs are listed from the perspective of the local domain controller from which the command was run. It is important to note that this information may be different from the perspective of each domain controller, depending on the forest-wide Active Directory convergence time and replication status.

    Repadmin /querysites

    Repadmin /querysites use routing information to determine cost of a route from a specified site to another specified site or sites.

    Syntax

    repadmin /querysites <FromSiteRDN> <ToSite1RDN> <ToSite2RDN>...]

    Parameter

    Description

    <FromSiteRDN>

    Specifies the relative distinguished name of the site from which the cost is calculated.

    <ToSite1RDN>

    Specifies the relative distinguished name of the site to which the cost is calculated.

    Example 1: Display cost between BRANCH1 and HUB

    Repadmin /querysites

    Example 2: Display cost between BRANCH1 and BRANCH2

    Due to site link transitivity, the cost from BRANCH1 to BRANCH2 is aggregated by adding the cost from BRANCH1 to HUB (100) with the cost from HUB to BRANCH2 (100).

    Repadmin /querysites

    Example 3: Display cost between BRANCH1 and Branch2

    Note that the relative distinguished name of the site is case sensitive and hence the error.

    Repadmin /querysites

    Note

  • The relative distinguished name of the site is case sensitive.

  • The repadmin /querysites parameter does not allow the use of alternate credentials.

  • Repadmin /queue

    Repadmin /queue displays tasks that are waiting in the replication queue.

    Syntax

    repadmin /queue <DC_LIST>

    Parameter

    Description

    <DC_LIST>

    Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp.

    Example: Display the queue length against the local domain controller

    Under normal circumstances this list should always be empty and the command should be run outside of the replication window when troubleshooting domain controller overload was caused due to replication requests.

    Repadmin /queue

    Example: Queue contains one item

    Repadmin /queue returns one item

    Repadmin /bridgeheads

    Repadmin /bridgeheads lists the bridgehead servers for a specified site.

    Syntax

    repadmin /bridgeheads [<DC_LIST>] [/verbose]

    Parameter

    Description

    <DC_LIST>

    Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp.

    /verbose

    Lists detailed information.

    For clarity:

    • The following example shows only bridgeheads only for the HUB site.

    • The following example shows the normal and verbose modes to help compare them.

    • “The RPC service is unavailable” status is abbreviated as RPC.

    • “The operation completed successfully” status is abbreviated as status.

    Example 1: Repadmin /bridgeheads rootdns

    Bridgeheads for site HUB (rootdns.contoso.com):
    Source Site    Local Bridge  Trns         Fail. Time    #    Status
    ===========    ============  ====     ==============   ===   ======
    BRANCH2       BRANCH-HUB-BH   IP 2005-02-14 14:18:52    3    RPC.
    Configuration research
    BRANCH1       BRANCH-HUB-BH   IP             (never)    0    Success.
    Configuration ForestDnsZones DomainDnsZones research
    BRANCH3       BRANCH-HUB-BH   IP             (never)    0    Success.
    Configuration DomainDnsZones ForestDnsZones research
    

    Example 2: Repadmin /bridgeheads rootdns /verbose

    Bridgeheads for site HUB (rootdns.contoso.com):
    Source Site    Local Bridge  Trns         Fail. Time    #    Status
    ===========    ============  ====     ==============   ===   ======
    BRANCH2        BRANCH-HUB-BH  IP  2005-02-14 14:18:52   3    RPC.
    Naming Context    Attempt Time         Success Time  #Fail  Last Result
    ==============    ============         ============  =====  ===========
    Configuration  2005-02-14 14:51:41  2005-02-14 14:18:51  3   RPC.
    research       2005-02-14 14:53:15  2005-02-14 14:18:52  2   RPC.
    Source Site    Local Bridge  Trns         Fail. Time    #    Status
    ===========    ============  ====     ==============   ===   ======
    BRANCH1        BRANCH-HUB-BH  IP             (never)    0   Success
    Naming Context    Attempt Time         Success Time  #Fail  Last Result
    ==============    ============         ============  =====  ===========
    Configuration  2005-02-14 14:51:41  2005-02-14 14:51:41  0   Success.
    ForestDnsZones 2005-02-14 14:52:37  2005-02-14 14:52:37  0   Success.
    DomainDnsZones 2005-02-14 14:53:15  2005-02-14 14:53:15  0   Success.
    research       2005-02-14 14:52:37  2005-02-14 14:52:37  0   Success.
    Source Site    Local Bridge  Trns         Fail. Time    #    Status
    ===========    ============  ====     ==============   ===   ======
    BRANCH3        BRANCH-HUB-BH  IP             (never)    0    Success.
    Naming Context    Attempt Time         Success Time  #Fail  Last Result
    ==============    ============         ============  =====  ===========
    Configuration  2005-02-14 14:51:42  2005-02-14 14:51:42  0   Success.
    DomainDnsZones 2005-02-14 14:53:15  2005-02-14 14:53:15  0   Success.
    ForestDnsZones 2005-02-14 14:52:37  2005-02-14 14:52:37  0   Success.
    research       2005-02-14 14:53:15  2005-02-14 14:53:15  0   Success.
    

    How to interpret the data

    Repadmin /bridgeheads is run remotely against a domain controller in the HUB site and the output is the perspective of the topology for ROOTDNS. In these examples, we are seeing local bridgehead server BRANCH-HUB-BH is having replication problems with the remote bridgehead server in the BRANCH2 site.

    Fields of interest

    Explanation

    Source Site

    Source site from where the local bridge head (inbound) is pulling data. Remember replication is always inbound.

    Local Bridge

    Local Bridge head server for the site for which the tool is displaying results. In the example in figure 3.11.9.2, BRANCH-HUB-BH is the bridgehead server of the HUB site.

    Trns

    In the example in figure 3.11.9.2, the transport is IP.

    Fail time

    This is the last successful replication time.

    #

    Number of failures since the last successful replication time.

    Status

    Replication status.

    Naming Context

    Directory partition. Remember Bridgeheads are partition specific.

    Attempt time

    Last replication attempt time with the remote bridgehead.

    Success time

    Last successful replication time with the remote bridgehead.

    #Fail

    Number of attempts since the failure per partition.

    Last result

    Latest replication status.

    Note

    Replication is performed for each partition. But sometimes we do not see the Schema partition listed in the previous example as a naming context (partition) and hence there are no bridgeheads listed. This is not a limitation of the tool; it has to do with the how information is stored in the connection object that is queried to determine the bridgehead. If you see the configuration partition in the output, it is implied that schema is also included because the KCC calculates the configuration and schema partitions to have the same replication topology.

    Repadmin /showmsg

    Repadmin /showmsg displays the error message for a given error number.

    Syntax

    repadmin /showmsg <Win32Error> | <DSEventID> /NTDSMSG}

    Parameter

    Description

    <Win32Error>

    Returns a short description of the given Win32 error code.

    <DSEventID>/NTDSMSG

    Returns the actual event log text for the specified event ID.

    Example: Display the error message for the win32error 1722 and DS event ID 1404

    Repadmin /showmsg

    Repadmin /viewlist

    By default, this subcommand is used to displays a list of domain controllers. It could also be used to form an Lightweight Directory Access Protocol (LDAP) query to list only objects in the directory.

    Syntax

    repadmin /viewlist <DC_LIST> <OBJ_LIST>

    Parameter

    Description

    <DC_LIST>

    Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp.

    <OBJ_LIST>

    This parameter takes a distinguished name (DN) or a special keyword that expands into a DN. The keywords are:

    • Ncobj:config: This keyword is the Configuration directory partition for the forest.

    • Ncobj:schema: This keyword is the Schema directory partition for the forest.

    • Ncobj:domain: This keyword is the domain partition DN of the home server.

    • Dsaobj: This keyword is the NTDS settings object of the home server.

    Example 1: Display all the DC’s in the forest

    Repadmin /viewlist

    Example 2: Display all the Group Policy objects in the domain directory partition for the domain of the domain controller that repadmin is running against

    Repadmin /viewlist

    Note the usage of OBJ_LIST and OBJ_LIST OPTIONS. For details please refer to repadmin /listhelp.

    Open sessions with the domain controller

    The repadmin /showctx command displays a list of computers that have opened sessions with a specified domain controller.

    Syntax

    repadmin /showctx <DC_LIST> [/nocache]

    Parameter

    Description

    <DC_LIST>

    Specifies the host name of a domain controller, or a list of domain controllers separated by a space. For details about <DC_LIST>, see repadmin /listhelp.

    /nocache

    Specifies that globally unique identifiers (GUIDs) are left in hexadecimal form. By default, GUIDs are translated into strings.

    Example: Show open sessions with a DSA

    Repadmin /showctx