Site Server - Membership Directory Group Scalability

August 1999

Introduction

The Membership Directory is the central data repository for Microsoft® Site Server version 3.0, which is a feature of Microsoft® Commercial Internet System (MCIS) version 2.0. You can use the Membership Directory to register users and manage user data, protect and share user data, verify users' identities, and control access to site content.

This paper discusses issues associated with managing the Membership Directory when a complex structure of users or products is required. It presents the internal Membership Directory architecture used to support large numbers of groups or containers or complex structures. It also describes limitations of Personalization & Membership (P&M) RTM code, identifies hot fixes to the code to remove limitations, and presents architectural solutions for these demanding user applications.

Note Personalization & Membership (P&M) RTM code is the software that shipped with Site Server 3.0 and MCIS 2.0.

Background

Complex Web applications are being developed to take advantage of the flexibility of the Membership Directory. Large product families can be modeled in the Membership Directory. It can also be used to organize users into multiple small communities for applications such as family accounts, business-to-business commerce, large numbers of user-interest communities, and controlled numbers of personal Web pages.

Administration of portions of the Membership Directory can be delegated securely. This is advantageous when the Membership Directory is organized to model a real-world structure like those described earlier. For example, an individual product manager may be responsible for a subset of products. In the case of family accounts, a family member will likely determine how to apply parental controls. And in business-to-business commerce, a company employee may be designated to determine who may place orders, as well as their associated purchase limits. The Membership Directory makes it possible to delegate these sorts of responsibilities.

Membership Directory Architecture

The Membership Directory architecture supports the user applications described in the previous section by creating a secure hierarchical Directory structure. The key components used to accomplish this are containers and Membership Directory access control lists (ACLs). This section describes those structures and how they are used to create a directory to support the outlined user applications.

Containers

The hierarchy of the Site Server Membership Directory is organized using structures called containers. There are two pre-defined container objects in the Directory: organizationalUnits and mgroups.

• OrganizationalUnit containers are the basic building blocks of the Membership Directory. They may include more organizationalUnit containers to extend the hierarchy, child leaf objects where user (member) information is stored, and mgroups (discussed in the following section). An organizationalUnit container may consist of zero or more of any combination of organizationalUnit containers, child leaf objects and mgroups. All organizationalUnit containers and child leaf objects under a particular organizationalUnit container must be uniquely named.

  In the Membership Directory, containers can be created under ou=Members,o=Root or under ou=Groups,o=Root containers. 

• Mgroup* containers are used to build lists similar to Microsoft® Windows NT® groups, in that they can be mapped to Membership Directory ACLs. Mgroups *identify all members belonging to a particular group. Where administration of small communities is delegated, an mgroup can be used to contain a community's administrators. An mgroup may consist only of MemberOf * *child leaf objects (MemberOf objects store the relationships between group objects and member objects that belong to those groups). All child leaf objects under a particular mgroup must be uniquely named. 

  By default, a parallel Windows NT group is created by the Site Server Authentication Service each time an mgroup is created. This is useful if Windows NT ACLs will be used on Windows NT file system content (such as Web pages, data files, or ASP pages) or on your site's Web pages. If administration of very large numbers of communities will be delegated on your site, refer to the section titled Windows NT Group Creation later in this document. 

  In the Membership Directory, mgroups may be found anywhere under the ou=Groups,o=Root container branch of the Directory Tree. It is also possible to create mgroups under the ou=Members,o=Root container branch of the Directory Tree, but features such as automatic Windows NT group creation are not available. 

Both organizationalUnit and mgroup containers are structures in the Membership Directory Information Tree (DIT). The DIT structure is considered to have changed whenever one of these containers is added, deleted, or renamed.

Membership Directory Access Control Lists

By default, each object in the Membership Directory has an access control list (ACL), which is a list of security IDs (SIDs) and their permissions for that object. ACLs are added to Membership Directory objects when it is necessary to secure the objects. To secure an object in the Membership Directory, an mgroup is created under the ou=Groups,o=root container. The mgroup is then added to the ACLs on an organizationalUnit container to provide appropriate access control. Administrators are then added as a MemberOf object to the mgroup.

By default, all objects created in the Membership Directory inherit the ACL placed on the root node. This is convenient if security is needed on Directory objects. However, there are performance implications associated with proliferation of ACLs in the MembershipDirectory. If this inherited ACL is not needed, it should be removed using Microsoft® Management Console. See the section titled DIT Caching Performance later in this document for more information.

Windows NT Group Creation

By default, the Membership Directory creates a Windows NT group on the server running the Site Server Authentication Service each time an mgroup is added to the ou=Groups,o=Root container. Automatic creation of Windows NT groups is particularly useful if your application uses ACLs on content, because Windows NT ACLs are used to protect content.

The Membership Directory can maintain security principals on objects within the Directory without creation of Windows NT groups. Administrators of sites with very large numbers of mgroups may want to disable automatic creation of Windows NT groups to avoid the performance impact of having correspondingly very large number of Windows NT groups. In extreme cases, the number of Windows NT groups can become so large that they exceed the limits of the Windows NT directory database.

LDAP Service Cache

The Lightweight Directory Access Protocol (LDAP) service uses a memory cache to optimize performance when finding objects and determining group memberships for the authentication service to use in building security contexts. The cache consists of:

• The DIT structure, including all organizationalUnits and ACLs associated with those containers.

• The group membership structure, including nested group memberships, incorporating mgroups only. 

Due to the fact that P&M groups are represented as mgroup container objects, all groups are cached in both the DIT container cache and the groups cache. However, the DIT cache includes all container objects in the directory (including any customer-defined object classes), not just group objects. Note that all container-class objects are cached, regardless of whether they actually contain any child leaf objects.

The DIT information is cached when the LDAP service instance starts. It is recached whenever a change to the DIT structure is made by the local LDAP service, unless re-caching is disabled using the appropriate registry key. Refer to Appendix C: Registry Key Information for more information.

At a site with a single LDAP server, any change to the DIT is automatically and immediately reflected in the LDAP cache. At a site with multiple LDAP servers, DIT changes can be made by any of the LDAP servers. To ensure up-to-date DIT information in the cache, each LDAP service periodically polls a timestamp in the Directory database to see if any DIT changes have been made by another LDAP server since the last local DIT cache. If so, then it will recache the DIT information to bring it up to date.

The table below summarizes re-caching behavior on the LDAP server.

RE-CACHE MATRIX | RECACHE BEHAVIOR OF MULTIPLE LDAP SERVERS

Operation

Mgroup

orgUnit

GroupMemberOf

MemberOf

Add

Offside DIT & Group

Offside DIT

All

None

Modify

All

Offside DIT

All

None

Delete

All

Offside DIT

All

None

ACL Change*

Offside DIT & Group

Offside DIT

Offside

None

* Any change to security (adding or removing an ACL on a container)

The Directory Assembled

A hierarchical Membership Directory with many small communities is built by adding containers under the ou=Members,o=root container. Figure 1 depicts a hierarchical Membership Directory with an individual (Lefty), a single level organization (Comp1), and a multi-level company (Comp2). (The example companies, organizations, products, people and events depicted herein are fictitious. No association* with *any real company, organization, product, person or event is intended or should be inferred.)

Figure 1 Membership Directory Communities 

In Figure 1, Lefty is an individual or child leaf object. Comp1 is an organizationalUnit container representing a sub-community containing several child leaf objects (Joe, Mary, and Frank). Comp2 is a nested organizationalUnit container with two additional organizationalUnit containers, each representing a sub-community (Comp2a and Comp2b). Comp2a is an empty organizationalUnit container. Comp2b has child leaf objects (Sally and Sue). Any of the organizationalUnit containers included in Figure 1 may have additional organizationalUnit containers added at a future date.

Administration of these accounts can be delegated by setting ACLs on the organizationalUnit containers under ou=Members,o=Root to control use of administrative functionality. To do this, using Directory ACLs, first create an ACL for each organizationalUnit container you want to administer separately. By default, the Membership Directory creates ACLs on all organizationalUnit containers under ou=Members,o=Root. (See the section titled Number of Access Control Lists later in this document for a discussion of performance implications.) Then mgroups are created under ou=Groups,o=Root. Mgroups are added to ACLs on the organizationalUnit containers under ou=Members,o=Root. Finally, users are placed in the mgroups as MemberOf objects.

In Figure 2, Frank is designated as the administrator of Comp1 and Sue is the administrator of Comp2b. (If there is only a single administrator for each organizationalUnit container, it is more efficient to add the member directly to the container's ACL rather than doing this through a group.) Frank and Sue have been added to an mgroup on the ACL, which allows them to gain access and perform administrative tasks for their respective groups.

Figure 2 Membership Community Administration 

Searching the Directory

When retrieving user data for authentication and personalization operations, the LDAP service normally performs a direct lookup of a user object from the full distinguished name (DN) of the object. The DN is similar to a fully-pathed file name in that it includes the complete hierarchical path information from the root of the directory to the specific object. The DN for Lefty in Figure 1 is cn=Lefty,ou=Members,o=Root, while the DN for Sue is cn=Sue,ou=Comp2b,ou=Comp2,ou=Members,o=Root.

In addition to the direct lookups, the LDAP service also supports one-level and subtree searches where the exact DN of the object is not available, but one or more attributes of the object are known. If the path to the object's parent container is known, the search is called a one-level search. If only some attributes of the desired object, but not the parent container, are known, the search is called a subtree search.

A subtree search can be initiated either from the root of the directory, in which case the entire directory is searched for objects meeting the designated criteria, or from some lower node (group or container) in the directory and search only in the starting group/container and any containers below that. A subtree search starting at some container below the root is called a partial subtree search because it does not include the entire directory tree.

If a site has a large hierarchy (more than 5,000 containers), partial subtree searches that include these containers will be slow and should generally be avoided. Subtree searches from the root provide better performance.

The following are examples of subtree searches from the root:

• All member objects where the first-name attribute is Mary. 

• All member objects with an email address attribute of Joe@Comp1.Com 

• All mgroups in the directory.

The following are examples of partial subtree searches:

• All member objects within or below the ou=Comp2,ou=members,o=root organizationalUnit container where the first-name attribute is Sue. 

• All MemberOf objects within or below the ou=groups container. 

Current Situation

This section describes the behavior of the Personalization & Membership RTM code. This is the software that shipped with Site Server 3.0 and MCIS 2.0.

Partial Subtree Searches

The original Membership Directory RTM code limited the number of subcontainers that can be included in a directory OU container when doing a partial subtree search. A full subtree search (a search that starts at the root directory) is not affected by this limitation.

The technical reason for the limitation is that, in the case of a partial subtree search, the LDAP service must specify the portion of the directory to be searched. Only the LDAP service understands the hierarchical tree structure of the directory. The representation of the Membership Directory in the SQL Server database only contains a parent object ID to indicate where in the directory an object exists. Formulating an SQL query for a partial subtree search requires the LDAP server to add a clause to the SQL query specifying all of the possible parent containers that the target object might have (for example, WHERE ParentContainerID=x1 or ParentContainerID=x2 or ParentContainerID=x3€¦).

The LDAP service uses a set of pre-allocated buffers to build the SQL queries. The original RTM code had a buffer length that only allowed a clause specifying approximately 200 parent objects.

Many customers who have large numbers of groups in their directory are not affected by the partial subtree search issue because they use custom tools or do not do partial subtree searches.

DIT Caching Performance

There are three issues in Membership Directory RTM code related to DIT and group caching performance:

• The number of mgroup containers for which memory could be allocated in creating the group cache on an LDAP server is limited to 5,000. With RTM code, it is not possible to have more than 5,000 mgroups. A hot fix is available that removes this limitation. 

• A Microsoft® Internet Information Services (IIS) timeout value limited how long an LDAP server could spend populating the DIT and Group caches at service startup. With RTM code, the IIS timeout value will be exceeded at approximately 5,000 groups (depending on the speed of the LDAP server). A hot fix is available that removes this limitation. 

• At sites with multiple LDAP servers and large numbers of organizationalUnit or mgroup containers, the amount of time it takes to recache the DIT when changes occur can affect performance (see Appendix B: Performance Data). If recache performance exceeds the requirements of the MCIS Installation at your site, see the Architectural Solutions section of this document. 

Windows NT Groups

The automatic creation of Windows NT groups with each DIT mgroup is a useful feature if you need to protect content on your site. However, if your site has a very large number of groups or containers, be aware there is a limit on the number of groups in the Windows NT directory database. That limit is approximately 80,000 groups per server. If you anticipate having more than 80,000 groups or containers in your site, see the Content Access Rights in Large Communities section of this document.

Hot Fixes

Hot fixes are available to remove the previously-described limitations. Contact Microsoft Product Support Services at https://support.Microsoft.com/support for information about how to obtain the hot fixes. Note that in order to download all hot fixes, you must have already installed Microsoft Site Server 3.0, Service Pack 1.

The following hot fixes will be available to address limitations:

• Phase 1 (available now) allows a site to scale beyond 100,000 groups or containers. The upper limit is under investigation, but will be affected by the number of Windows NT groups created. See the Windows NT Groups section of this document for more information. Additionally, Phase 1 allows configuration of the amount of memory allocated by the LDAP server for DIT caching. It also allows configuration of the IIS timeout value for DIT caching at startup, and for re-caching after a DIT structure change. These hot fixes require Site Server 3.0, SP1. 

• Phase 2 (scheduled to ship with Site Server SP2) will significantly improve caching performance on the LDAP server. This is particularly important for sites with complex DIT structures where the DIT structure changes regularly at runtime. 

All hot fixes will be included with Microsoft® Site Server, Service Pack 2 when it is released in late 1998. Site Server, Service Pack 2 will ship with the Microsoft® Commercial Internet System (MCIS) 2.5 in Q1CY99.

Tuning Your Directory

With Membership Directory RTM code, several factors aside from hardware configuration and total system load affect performance. These factors are:

• Total number of containers 

• Rate of change of the DIT 

• Number, size, and complexity of ACLs on directory objects 

This section discusses performance impacts associated with each factor.

Total Number of Containers

As noted earlier, there are two types of LDAP cache. The entire DIT is cached on each LDAP server in the MCIS installation. The LDAP Service also separately caches all mgroups. LDAP Server memory requirements go up in proportion to the number of containers in the DIT, as well as the number of mgroup containers. In addition, startup time is adversely affected by the number of containers that are being loaded into the two caches. See Appendix B: Performance Data for more information.

The size of the LDAP Service cache can be reduced using Active Server Pages (ASP) code. See the Writing Custom ASP Code section of this document for more information.

Rate of Change of the DIT

Each LDAP service periodically polls a timestamp in the Membership Directory database to see if any DIT changes have been made by another LDAP service since the last local DIT cache. If so, then it will recache the DIT information to bring it up to date. A registry key setting (dsPollThdRate) controls the polling interval. The interval is reset when the LDAP Service cache has been reloaded.

While re-caching is underway, the DIT can be read, but it is not possible to make changes such as adding an organizationalUnit container or an mgroup container. If the structure of the DIT changes frequently (as might be the case at a site where users can self-register their community), LDAP services could spend a significant amount of time re-loading DIT and group caches. This effectively prevents other users from registering while re-caching is underway. If this behavior is undesirable, alternate solutions such as pre-creation of containers should be considered (see the Architectural Solutions section of this document).

It is also possible to use a single LDAP server for adding containers to your Membership Directory. This would prevent the latency that takes place when re-caching occurs, because re-caching would only occur on the other (offside) LDAP servers and not the server managing structure modifications.

Number of Access Control Lists

The Membership Directory makes it easy to use ACLs for authorization control. Membership Directory ACLs are stored in the LDAP cache to make it possible to quickly determine who is authorized to access organizationalUnit and mgroup containers. Large numbers of ACLs in a directory significantly increase memory requirements on the LDAP server and lengthen the time spent by the LDAP server loading its cache. If your site has very large numbers of containers, consider writing custom ASP code to control authorization, which will minimize caching and re-caching time. This is particularly important at sites where the structure of the DIT changes frequently. See Appendix B: Performance Data for performance information.

Architectural Solutions

This section discusses several approaches that can be used to enhance Membership Directory performance for sites with large numbers of communities or complex product hierarchies.

Writing Custom ASP Code

Generally, the best way to avoid performance problems associated with large numbers of organizationalUnit and/or mgroup containers is to reduce the number of containers. ASP code can accomplish the same grouping and authorization functions provided by the Membership Directory.

Writing ASP code can provide one or both of the following benefits:

• Reducing mgroup containers by controlling access to administrative functions using logic to check a member's attribute. This reduces the size of the DIT cache and the group cache on all LDAP servers at the site. 

• Reducing organizationalUnit containers by grouping members according to the setting of a designated member attribute. Locating members of a community using a sub-tree search reduces the size of the DIT cache on all LDAP servers at the site. This approach is slower, however, and if the information will be needed often, it may not be an optimal solution. 

  There are many ways that ASP code, as software, can be used to reduce the number of containers in a DIT. The following is one possibility for providing delegated administration:

  • Identify possible administrators by creating a mandatory attribute type for each member called user-type. Values for this attribute will be drawn from an enumerated list that includes Admin.

  • Identify community members by creating an attribute type for each member called CommunityName. Values for this attribute may correspond to company name (if company is the community being administered), family name (if a family unit is being administered), and so on. If this is a product hierarchy, consider naming this attribute something appropriate such as ProductFamily.

Note Anyone who connects directly to the Membership Directory can alter values in this attribute for their member record using the SELF security principal. A user security principal is a placeholder that grants/denies access for the current user to that user's own attributes. The security principal is used to determine which user attributes a user will be allowed to see and/or change. For instance, you could set the ACL on the user password attribute for an entire organizationalUnit container to read/write for SELF and no access for anyone else. Then, without having to set individual ACLs for each user's profile, each user could change their own password. Care should be taken to ensure that only ASP pages that enforce the delegated administration are allowed to access the Directory. Alternatively, access to these attributes can be explicitly denied.

The following sections describe the sequence of operations for registering a company, logging on, and administering a system using this architecture

Register a Company (Assumes Delegated Administration) 

1. The user accesses the registration page, provides registration information, including company and administrator, and then presses Submit. 

2. Custom ASP code is used to perform a subtree search to confirm the company name is unique. The system must also perform a subtree search to ensure a unique user name. If both are unique, a child leaf object is created for the member under ou=Members,o=Root using information provided on the registration page. The user-type attribute is assigned a value of Admin. The company name is placed as the value in the CommunityName attribute. 

3. The administrator can now log on. If other users will be accessing the system, their accounts can be created now.

User Logon 

1. The user accesses the logon page, enters user name and password, and then presses Submit.

2. The logon page verifies password with cn=UserName,ou=Members,o=Root. 

3. The user conducts activities using custom ASP code. All work is assumed to be performed on behalf of the community specified in CommunityName.

Administrator Tasks 

1. The administrator logs in as described earlier. 

2. Using custom ASP code, the system restricts access to administrative functionality to users with Admin as the value for the user-type attribute. 

3. The system uses custom ASP code to ensure that only other users with the same value in CommunityName can be added, modified, or deleted. 

Real-time Creation of Communities

If you anticipate hosting a large number of separate user communities, and the business of your site requires that communities be allowed to register in real time, system performance may degrade significantly due to the amount of time spent re-loading the LDAP cache. To alleviate this situation, consider either writing ASP code to administer the communities, dedicating a single LDAP server to registration, or creating an architectural solution similar to the one described in the following paragraphs.

This solution is based on the assumption that you want to avoid re-caching whenever a new community registers at your site, and that you are willing to pay service startup penalties to cache the DIT. You should also consider disabling automatic creation of Windows NT groups to avoid Windows NT directory database limits on the number of Windows NT groups. See the topic "PMAdmin Set AuthSvc" in the P&M section of the Site Server version 3.0 documentation for information about how to turn off automatic creation of Windows NT groups.

With this solution, you pre-create batches of containers before they are needed, at times when re-loading DIT changes will have minimum impact on Membership Directory performance (such as during scheduled downtime). You will need to create the following system constructs:

• Pre-create organizationalUnit containers under the ou=Members container and if delegated administration is a feature of the site, pre-create mgroup containers under the ou=Groups container. Names must be unique and, perhaps, should be sequentially assigned. If delegating administration, it should either be possible to derive the distinguished name of the mgroup from the organizationalUnit container or a mapping table should be maintained. 

• Maintain a list of unused containers. You must be sure that each entry on the list is assigned to no more than one company. A table maintained in SQL Server is ideal for this purpose. 

• Maintain a map that relates company name to its assigned container. You must be able to derive a company's container from a company name entered at logon time. 

After the system constructs have been created and a single company named ABC ihas been added, the directory and the list of unused containers might look like the diagram in Figure 3.

Figure 3 Membership Directory Structures to Avoid Run-time Recache 

After the containers have been pre-created and the list of unused containers established, the run-time procedure is as follows:

Register a Company (Assumes Delegated Administration) 

1. The user accesses the registration page, provides registration information including company and administrator, and then presses Submit. 

2. Using custom ASP code, the system confirms the company name is unique. If so, it locks the Container List and grabs the next available container ID, adding the company information as a way of indicating the record is in use. 

3. A DN is created to the container using the same container ID. The system makes an LDAP call to populate the container ID with company information. 

4. The system creates a child leaf object for the administrator under the container ID and populates it with information provided on the registration page.

5. A DN is created to the mgroup associated with the same container ID. The system makes an LDAP call to add the administrator as a MemberOf object to the mgroup container. 

6. The administrator can now log on. If other users will be accessing the system, accounts must be created. 

User Logon 

1. The user accesses the custom login page, enters company name, user name, and password, and then presses Submit. Note that this approach does not support the Distributed Password Authentication (DPA) protocol used in MCIS 1.0. If security is a concern, you may want to consider performing the logon over a Secure Sockets Layer (SSL) channel. 

2. Using custom ASP code, the Container List is accessed using the company name to retrieve the container ID. 

3. The system verifies the password using cn=UserName,ou=RetrievedContainerId,ou=Members,o=Root 

4. The system grants the user access based on results of password verification. 

Administrator Activities 

1. The administrator logs in as described earlier. 

2. The system grants access to administrative pages. 

3. The administrator adds, modifies, and removes users as necessary.

Content Access Rights in Large Communities

Normally, the Membership Directory uses ACLs to selectively control access to content based on community membership. However, if a very large number of Windows NT groups is needed to protect content on a site, there is a risk of exceeding the maximum number of groups supported by the Windows NT directory database. To alleviate this situation, consider either writing ASP code to control access to content or creating an architectural solution similar to the one described in the following paragraphs.

This solution is based on the assumption that you want to provide selected content access rights to very large numbers of sub-communities. It is also assumed that you want to use Windows NT ACLs to simplify coding and operations, but expect to extend beyond the Windows NT directory database group limit of approximately 80,000 for a single computer.

With this solution, you create multiple, separate Membership Directories running on separate computers, and use a custom mapping table to direct users to the correct logon page. The key constructs are:

• Multiple Membership Directories running on separate computers. 

• A mapping table that equates company name to logon URL. The mapping table will contain a company name, a URL, and perhaps a Container ID. 

• Pre-created containers, as in the solution in the "Real-time Creation of Communities" section of this document. 

After the constructs have been created, the MCIS installation and Membership Directory may look like the diagram in Figure 4.

Figure 4 Large Numbers of Communities with Access Control Lists 

Once the containers have been pre-created and the list of unused containers established, the following run-time behavior takes place:

Register a Company (Assumes Delegated Administration) 

1. The user accesses the registration page, provides registration information including company and administrator, and then presses Submit. 

2. Using custom ASP code, the system confirms the company name is unique. If so, it locks the Container List and grabs the next available container ID, adding the company information as a way of indicating the record is in use. The record also specifies one of the data center's Membership Directories. 

3. A DN is created to the container using the same container ID for the designated Membership Directory. The system makes an LDAP call to populate the container ID with company information. 

4. The system creates a child leaf object for the administrator under the same container ID and populates it with information provided on the registration page. 

5. A DN is created to the mgroup associated with the same container ID. The system makes an LDAP call to add the administrator as a MemberOf object to the mgroup container. 

6. The administrator can now log on. If other users will be accessing the system, accounts must be created. 

User Logon 

1. The user accesses the custom login page, enters company name, user name and password, and then presses Submit. Note that this approach does not support the DPA protocol. If security is a concern, you may want to consider performing the logon over an SSL channel. 

2. The system accesses the Company Map using the company name to retrieve the computer on which the Membership Directory is located, and redirects it to the appropriate ASP page for that Membership instance. 

3. From that ASP page, the system verifies the password using cn=UserName,ou=RetrievedContainerId,ou=Members,o=root. 

4. The system grants the user access to the system based on the results of the password verification. 

Administrator Activities 

1. The administrator logs in as described earlier. 

2. The system grants access to administrative pages. 

3. The administrator adds, modifies, and removes users as necessary. 

Appendix A: Quick Reference

Use this appendix to help determine problems associated with Membership Directory Group Scalability.

• Do you want to delegate administration to large numbers of small user communities? If so, consider minimizing the number of groups and containers by writing ASP pages that use logic based on some user attribute to grant administrative privilege. You should also consider turning off inheritance of ACLs in the Membership Directory. Propagation of ACLs is a significant hit on LDAP caching at start time and during recaching. 

• Do you need to delegate content administration to individual user communities? If not, consider turning off the automatic creation of Windows NT groups. For information about how to do this, see the topic "PMAdmin Set AuthSvc" in the P&M section of the Site Server version 3.0 documentation.

• How much memory does your LDAP server have? The amount of memory required is a function of the number of groups and containers cached. Be sure you configure dsMaxDitNodes and dsStoreCpoolMaxGroups entries in the registry for each LDAP server. 

• How much time is required for re-caching? Be sure to configure dsLdapStartupTimeOut to support the amount of time needed to cache/recache the DIT on your hardware.

Appendix B: Performance Data

The tables in this appendix show preliminary results of testing performance with the Membership Directory hot fixes in place. The following hardware and software was used to obtain the results listed.

LDAP Servers:	Dual Intel Pentium II processors, 400 MHz, 128 MB of RAM, Microsoft® Windows NT® Server version 4.0, Windows NT Service Pack 3, Microsoft® Internet Explorer version 4.01, Windows NT 4.0 Option Pack
SQL Server Computer:	Quad Intel Pentium Pro processors, 200 MHz, 512 MB of RAM, RAID 0 disk array, Windows NT Server version 4.0, Windows NT 4.0 Service Pack 3, Internet Explorer version 4.01, Windows NT 4.0 Option Pack, Microsoft® SQL Server version 6.5, SQL Server Service Pack 4

Note Your performance numbers may vary from those provided here based on your hardware and software platform.

Add Time (Clean Database)

This table shows the amount of time required to add containers to a clean database. This is a one-time cost. Time values represent the number of seconds it takes to create the specified number of containers.

NUMBER OF CONTAINERS	ORG UNIT CONTAINERS--Time	ORG UNIT CONTAINERS--Adds/sec	Mgroup CONTAINERS--Time	Mgroup CONTAINERS--Adds/sec
10,000	275 sec	36.4	304 sec	32.9
25,000	960 sec	26.1	932 sec	26.8
50,000	1983 sec	25.2	2193 sec	22.7
100,000	4363 sec	22.9	5119 sec	19.5

Container Cache Time

This table shows the time it takes to load the indicated number of containers/groups into the memory cache. This is a recurring cost at service startup and at each recache. Separate columns show the impact of loading ACLs.

NUMBER OF CONTAINERS	ORG UNIT CONTAINERS--Mem Usage w/o ACLs	ORG UNIT CONTAINERS--;Mem Usage with ACLs	Mgroup CONTAINERS--Mem Usage w/o ACLs	Mgroup CONTAINERS--Mem Usage with ACLs	ORG UNIT/Mgroup COMBINED * --Mem Usage w/o ACLs	ORG UNIT/Mgroup COMBINED * --Mem Usage with ACLs
10,000	8 sec	23 sec	9 sec	12 sec
25,000	11 sec	56 sec	14 sec	25 sec
50,000	18 sec	64 sec	20 sec	46 sec	21 sec	56 sec
100,000	34 sec	80 sec	46 sec	104 sec	41 sec	199 sec

* Load time for 25,000 organizationalUnit containers and 25,000 mgroup containers (50,000 total containers), without any ACLs is 21 sec. Load time for 50,000 organizationalUnit containers and 50,000 mgroup containers (100,000 total containers) with ACLs is 199 seconds.

Memory Usage

This table shows the amount of LDAP server memory needed to cache containers in the DIT. Tests were performed with default ACLs. Performance will vary based on length and complexity of ACLs.

NUMBER OF CONTAINERS	ORG UNIT CONTAINERS--Mem Usage w/o ACLs	ORG UNIT CONTAINERS--Mem Usage with ACLs	Mgroup CONTAINERS--Mem Usage w/o ACLs	Mgroup CONTAINERS--Mem Usage with ACLs	ORG UNIT/Mgroup COMBINED*--Mem Usage w/o ACLs	ORG UNIT/Mgroup COMBINED*--Mem Usage with ACLs
10,000	14.7 MB	41.7 MB	15.1 MB	19.3 MB
25,000	21.8 MB	73.0 MB	22.4 MB	34.2 MB
50,000	33.5 MB	83.0 MB	34.7 MB	54.4 MB	43.8 MB	78.8 MB
100,000	56.9 MB	88.1 MB	59.4 MB	63.9 MB	76.9 MB	87.3 MB

* Memory usage for 25,000 organizationalUnit containers and 25,000 mgroup containers (50,000 total containers), without ACLs is 43.8 MB. Memory usage for 50,000 organizationalUnit containers and 50,000 mgroup containers (100,000 total containers) with ACLs is 87.3 MB.

Appendix C: Registry Key Information

The following three registry keys are new with Quick Fix Engineering (QFE) hot fixes 47502 / 47505 / 47597. All are found under HKLM\system\currentcontrolset\services\ldapsvc\parameters on the LDAP server, and any changes to the key values are effective only when the LDAP service instance is started or restarted.

• dsMaxDitNodes
  DWORD
  default = 10000
  This is the maximum total number of nodes (containers) of any type or location supported in the directory tree structure. This needs to include both mgroups and organizational unit containers. The total DIT Nodes = number of groups + number of ou containers + total number of all other objects in the directory that can have child objects.

• dsStoreCpoolMaxGroups
  DWORD
  default=10000
  This is the maximum total number of groups (mgroup objects) supported in the directory.

• dsLdapStartupTimeOut
  DWORD
  default=300 (seconds)
  This is the time (in seconds) allowed at startup of an LDAP server instance before it will be timed out. The timeout value required will depend on the hardware processor speed, the number of other services or processes running on the same computer, and the total number and mix of ou containers and mgroups in the directory. Depending on hardware, the default setting will need to be increased to go beyond approximately 100,000 groups/containers. Note that the Phase 2 hot fixes significantly improve cache load time. Until these fixes are available, you should increase the default setting to get beyond 5,000 to 10,000 containers. 

The following are other existing related registry keys that are of interest to the issues of group/container scalability and performance of the LDAP service:

• dsPollThdRate
  DWORD
  default=120 (seconds)
  This is the period (in seconds) between times that the LDAP service will poll the directory to determine whether there have been changes to the DIT structure (that is, additional groups or other containers added to the directory tree) requiring recaching. The lower this number, the more frequently the LDAP server will repoll and recache if there have been DIT changes made through a different LDAP server instance.

• dsPollThdEnabled
  DWORD
  default=1 (1=true, 0=false)
  This switch controls whether or not the LDAP service will poll the directory to determine whether there have been changes to the DIT structure requiring recaching.

• dsRecacheEnabled
  DWORD
  default=1 (1=true, 0=false)
  This switch controls whether the LDAP service will recache the DIT information if changes are made to the directory tree structure through the local LDAP instance. This should be turned off (and the LDAP service instance restarted) if, for example, a large number of groups or other containers need to be created, so that the LDAP service does not try to recache the DIT after every change. Note, however, that if this key is turned off, objects cannot be created inside any new containers (including adding members to any new groups) until after the LDAP service instance is restarted, so that the LDAP service is aware of the new containers.