Post MortemMigrating from Exchange 5.5

Whitney Roberts

THE PROJECT

The Kentucky Department of Education needed to migrate from its existing Microsoft® Exchange Server 5.5 implementation to Exchange Server 2003. The system supports approximately 600,000 faculty, staff, and students during any given school year.

THE PLAYERS

The project was tackled by the Kentucky Department of Education's Office of Education Technology (OET) assisted by KiZAN Technologies consultants. The OET staff provided management oversight of the existing Exchange 5.5 implementation. The day-to-day running of the Exchange servers was the responsibility of the local staff in the school districts.

THE CHALLENGES

The Department of Education provides technical direction for all 178 school districts in the Commonwealth of Kentucky. These school districts are connected to each other via a WAN but each hosts its own locally managed Exchange 5.5 server, which needed to be upgraded individually. This upgrade would bring support for Outlook® 2003 cached mode clients, an improved Outlook Web Access (OWA) experience, a standardized e-mail addressing scheme, and more.

In addition to rolling out a new e-mail server and client infrastructure, many of the school districts were adding e-mail support for students. Both state and federal policy required that student information be protected and not visible outside of the respective district.

THE PLAN

Since an Active Directory® Connector (ADC) could not be used, we created a new Exchange Organization and migrated into it. Standardizing e-mail addresses meant creating both new identifying subdomains for each school district and over 400 new recipient policies. Protection of student addresses required constructing a system of Global Address Lists (GALs) and custom address lists for each of the school districts.

The actual migration involved staging new hardware, creating and configuring mailboxes based on district membership and user type, and setting extension attributes based on district membership and user type so that the user would show up in the appropriate address lists.

We created a Visual Basic®-based application to detect and correct misconfigured account attributes or group memberships and to provision new mailboxes properly to make sure visibility requirements were maintained. The app was deployed to all 178 districts and runs on a timed basis from the school district servers.

Background

On the surface, this may not sound like a challenging or technically difficult job. However, as I have been humbly reminded, business requirements rule, and the business requirements of this project were eye-opening.

Over the years, decentralized management and a mish-mash of standards created numerous support "silos" that most often had to be managed by OET. In addition, OET also had to make sure the Exchange 5.5 implementation met the requirements of the school districts and the Department of Education at the same time. This meant that, for the districts that did utilize student e-mail, the student information was protected and not visible outside of the respective district as dictated by state and federal guidelines (see ed.gov/policy/gen/guid/fpco/ferpa/index.html).

In Exchange 5.5, these guidelines meant student information within the district could not replicate out to the other school districts or other state agencies that shared GAL data. Specifically, this necessitated going through a manual export/import process for each of the 178 Exchange 5.5 sites: hiding the students, replicating the list, un-hiding the students, and finally importing the address list back in. In this manner, other districts would see only the staff data, and the owning school district would see all of its own data but only staff data from the other school districts.

Given the strengths and weaknesses of the existing system, upgrading the messaging infrastructure had to satisfy some specific criteria. First, the Kentucky Department of Education wanted to standardize on a common, readable SMTP naming scheme for the entire enterprise. The scheme needed to provide separate SMTP address spaces for faculty and students.

The new system also needed to provide a centrally managed service that would remove the messaging management and responsibility for availability from the district. This meant centralizing management of disaster recovery, mail flow, and helpdesk call resolution. As such, the new system had to provide a centralized method for standardizing user configuration across all districts, regardless of geographic location or district affiliation of a given user.

Further, the new system had to provide secure Web-based and mobile-device access to e-mail across a common, unified namespace for all districts.

Given these requirements—many of which were not already implemented in the existing Exchange 5.5 system—the upgrade to Exchange 2003 looked quite daunting.

Solution Building Blocks

To satisfy the business requirements, several Exchange-specific options were decided on for the deployment.

Due to the lack of standardization and the sheer magnitude of the Exchange 5.5 environment, an ADC would not be used. Instead, we created a new Exchange Organization and migrated into it.

Cleaning up and standardizing e-mail addresses for staff and students in each district led to the creation of over 400 new recipient policies. The requirements mandated a unique identifying subdomain for each district, including an identifying district designation for students. For example, in the case of Shelby County, the primary subdomain would be shelby.kyschools.us. The students in each district received a stu. subdomain designation, so the students in Shelby county would have stu.shelby.kyschools.us addresses.

Another constraint was that student addresses should be visible only within the students' district, while staff addresses should be visible across districts. This meant the construction of an elaborate system of GALs and custom address lists, as shown in Figure 1 and Figure 2. One benefit of moving to Exchange 2003 was that the address lists could be secured with permissions, and access to each individual address list could be controlled by group membership. We made good use of this feature and created two district-level security groups (one for staff and one for students), then secured the corresponding district staff and student GALs and address lists so that only user accounts within a specific district had access to certain address lists. By removing the default GAL and address lists and then making sure that the Everyone and Authenticated Users groups were not listed on the access control list (ACL) for each address list, we could preselect the address list a specific user type in a specific district would see.

Figure 1** Custom Address Lists Based on Role **(Click the image for a larger view)

The permissions, however, only solved a portion of the visibility problem. We had to come up with a way to control who was visible in what address list. A base Lightweight Direct Access Protocol (LDAP) query was created that allowed us to query for staff and student visibility; the query was then customized slightly for each address list and GAL. For example, the LDAP query for the Shelby staff GAL would show only the appropriate staff and students in Shelby and all of the staff (but no students) from every other district in the state. The Shelby student GAL would show only the staff and students from Shelby and no other staff or students from any other district in the state. We repeated this query for all of the districts in the organization. Figure 3 shows examples of some LDAP queries used in this complex migration.

Figure 3 Address List LDAP Queries

Staff GAL LDAP Query

(&(&(objectCategory=*)(mailnickname=*)
(!objectCategory=ExchangeAdminService)(!objectCategory=msExchPublicMDB)
(|(extensionattribute15=15)(&(extensionattribute14=<districtNumber>)
(extensionattribute15=20))(&(extensionattribute14=<districtNumber>)
(extensionattribute15=14))))) 

District Address List LDAP Query

(&(&(&(objectCategory=*)(mailnickname=*)
(!objectCategory=ExchangeAdminService)(!objectCategory=msExchPublicMDB)
(&(extensionattribute14=<districtNumber>)(extensionattribute15=15)))))

District Students Address List LDAP Query

(&(&(&(objectCategory=*)(mailnickname=*)
(!objectCategory=ExchangeAdminService)(!objectCategory=msExchPublicMDB)
(&(extensionattribute14=<districtNumber>)(extensionattribute15=20)))))

District Staff Offline Address List LDAP Query

(&(&(objectCategory=*)(mailnickname=*)
(!objectCategory=ExchangeAdminService)(!objectCategory=msExchPublicMDB)
(|(extensionattribute15=15)(&(extensionattribute14=<districtNumber>)
(extensionattribute15=20))(&(extensionattribute14=<districtNumber>)
(extensionattribute15=14)))))

Figure 2** Global Address Lists Segregating Staff and Students **(Click the image for a larger view)

Custom values for the Exchange user properties extensionAttribute14 (Custom Attribute 14) and extensionAttribute15 (Custom Attribute 15) were the distinguishing mechanism behind the LDAP queries. All staff and students state-wide used a specific value for extensionAttribute14 to designate user type and a unique value in extensionAttribute15 to show district affiliation. By using unique values for each district and the same values for each user-type, the address lists were able to show exactly the information required for each user type in each district.

These address lists worked quite well for users accessing e-mail through Outlook in non-cached mode or through OWA. (We directed each user to the correct address list in OWA using the msExchQueryBaseDN attribute.). But users running Outlook in cached mode presented another problem. Because Outlook will not honor a customized GAL as the designated Offline Address Book (OAB), we had to create additional address lists that were duplicates of the GALs. Each district mailbox store was then configured with the appropriate OAB address list for its cached-mode users. All in all, this solution topped out at over 1,500 address lists to provide full functionality to the various types of users across all 178 districts.

Ensuring Compliance

To make sure the structure that was being put into place would work correctly, adherence to the newly established standards was paramount. There were several high-level tasks that had to occur for every Exchange-enabled user in the organization. The reason for this is that each user has to be properly located across all of the district Exchange servers, as well as be visible in the correct address lists.

The user needed to be a member of the proper security group within the district's domain because the security group grants permissions to open the appropriate address list. The user also needed the msExchQuerybaseDN attribute set correctly based on district and organizational unit (OU) membership. The user's OU determined the user type.

The next step was to create the user's mailbox on the correct store on the proper server based on district membership and user type. This affected service levels, mailbox size limits, and offline address book address lists so that Outlook cached-mode would function correctly. The extension attributes were set based on district membership and user type so the user would show up in the appropriate address lists and not be visible in others.

Finally, we needed to provide a means to automatically correct any account's attributes or group memberships if they were misconfigured (or changed) to inappropriate values. If a user moves from one OU to another within the district domain, this signifies a user classification change, and mailbox location, GAL selection, and address list visibility must be updated.

We needed a provisioning system that would accomplish these tasks and assure Active Directory consistency and accuracy. We created a console-based Visual Basic-based app that runs on a timed basis from the school district Exchange 2003 servers. The application, which checks policy and standards adherence and also handles new object creation and provisioning, was deployed to all 178 districts.

We felt that the tasks at hand were too complex for a series of scripts to execute quickly and reliably. As a result, the Visual Basic-based application was seen as the appropriate solution since it was efficient, had structured error and exception handling support, and its internal code could be easily adapted to Microsoft Identity Integration Server (MIIS) extension code in the future with minimal rewriting.

Because we were migrating to a new organization, some method of coexistence had to be found. There were more than 178 legacy SMTP domains to support in the new environment—some districts had preexisting student e-mail domains and others did not—but we had the luxury of migrating districts one at time and bringing their legacy messages and addresses over to the new system without affecting the other districts. However, we needed to avoid disrupting e-mail flow as a district was migrated from Exchange 5.5 to Exchange 2003. This meant implementing some type of directory synchronization between the Exchange 2003 Active Directory and the legacy Exchange 5.5 directory. Microsoft granted the Kentucky Department of Education limited usage of MIIS for the duration of the migration only, so customized extension code was developed and used to synchronize both Exchange organizations via MIIS before, during, and after the migration.

Moving to a new Exchange organization naturally affects MAPI client configuration. The Department of Education was faced with the fact that there were literally tens of thousands of MAPI profiles at the district level that would have to be reconfigured during the migration. Fortunately, Microsoft released the Exchange Profile Tool (ExProfRe.exe), which was used for profile migration at the districts. We created a packaged installer that ran ExProfRe.exe with preloaded values so that district-level profile migrations could be performed in an automated fashion. The program was invoked via Group Policy. We saw close to a 100 percent success rate on automated profile migrations between organizations.

Hardware Upgrades

Since the new Exchange 2003 servers would be taking the place of geographically dispersed Exchange 5.5 servers, the hardware aspect of this solution had to support the same number of users as the Exchange 5.5 deployment, plus the load of the additional student accounts. The servers would have to be hosted locally within the district because the WAN topology was designed in such a way that all inter-district traffic (Internet and internal) flowed over a private T1 line (in some districts, multiple T1 lines).

Due to budgetary, management, and equipment-hosting constraints, the Exchange servers would not have storage area network (SAN) attached storage, so all storage had to be either internal or direct-attached. Because most districts do not have datacenters with rack storage and centralized cooling, the servers had to be as self-contained as possible. Fortunately, most districts had fewer than 5,000 users so single-server deployments were possible. For the districts that had more users, more powerful servers (or multiple servers) and additional storage were used, sometimes with separate front-end servers to further segregate the student and staff traffic.

In most districts we were able to deploy well-equipped dual-processor servers that were configured with the maximum number of 15,000 RPM Ultra320 SCSI drives, multiple RAID controllers, and 4GB of RAM. Larger districts got one or more quad-processor servers, using smaller front-end servers where necessary. Benchmark and performance testing before deployment indicated no usage or load problems. Varying degrees of local usage patterns, though, would be the real determining factor on the server's overall performance.

Proof of Concept Testing

Although the production environment comprises a 178-domain Active Directory forest, the proof-of-concept (POC) lab was scaled down to 13 district domains and an empty root domain. All of the test domains were populated with exports from the production domains. Virtual Exchange 5.5 servers were built and configured with copies of the selected districts' databases so that we could test the migration process and build-out of the planned environment. The entire POC testing lab was constructed with virtual machines so that the environment could be baselined, frozen, and then redeployed when necessary.

A diverse combination of scripts (using VBScript), Systems Management Server (SMS) installer packages, and console applications were devised, coded, tested, and finalized. In the end, we had scripts that performed initial Exchange 2003 setup and configuration, including recipient policies, GALs and custom address lists, and permissions. Scripts also handled administrative group configuration, district domain configuration (security group creation and permissions delegation), and post-deployment configuration for migration.

Staging the Exchange 2003 servers presented some significant challenges. We needed to get all of the Exchange servers installed successfully in all 178 domains and do so in such a way that didn't jeopardize the rest of the migration. Exchange 2003 differs from Exchange 2000 in that the domain-specific Exchange Domain Servers group is not added to the ACL of the organization until the first Exchange 2003 server in the domain is installed. (In Exchange 2000, the group is added when running the DomainPrep process.) To make sure we had all of the necessary security groups listed on the organization object in Active Directory, we had to install in a given domain, manually replicate Active Directory from the domain to the Active Directory replication hub site, and then pull that replication down to the next district to be installed. Only after the ACL on the organization object was confirmed to have the Exchange Domain Servers group from the previous district could the installation of the next district begin safely. If this process was completed out of order in even one domain, that district would have to be reinstalled (at the minimum) or other districts would not be able to communicate between districts due to an organization object ACL that didn't list all of the other districts correctly.

We tested the entire deployment, configuration, server staging, and migration numerous times until we were satisfied that the process worked well within the POC environment.

Migration and Growing Pains

Once the POC procedures and results were approved by the project team, the migration was ready to begin. OET chose to migrate themselves first so that they could experience the process and after-effects firsthand. At the time of migration, all of the servers had already been staged and the initial environment had been verified. All of the front-end protocol servers and mailbox servers were in place and ready to go into production with users.

Execution of the migration plan went well and, after a lengthy 21 hours to transfer content from one server to another, OET was up and functional on Exchange 2003. Fortunately, there were no significant issues with migrating the Outlook profiles from Exchange 5.5 to Exchange 2003. We packaged a batch file that called ExProfRe.exe in different configurations based on some simple logic. We then distributed the batch file and ExProfRe.exe to the Netlogon share of a domain controller (DC) in each district and used Group Policy to run the batch for all clients at the conclusion of a migration. This worked well at OET and we were confident of the process going forward.

At this point, the migration plan was reviewed and reapproved based on our experience with the OET deployment. Moving forward with the migration, we chose six districts to be the first production sites. Before each district was migrated, MIIS would read the latest info about the district's mailboxes from Exchange 5.5 and update the corresponding users in the district domain. We used the Exchange Migration Wizard (Mailmig.exe) to actually move the mailbox data between organizations. As the wizard moved the mailbox data, all of the proxy addresses were brought over and updated from what MIIS originally exported to Active Directory. During the migration, mail would be queued at the front-end SMTP servers and then released to the district after the migration, post-migration server configuration, and user testing had completed; the district was prepared to perform wide-scale client migration using ExProfRe.exe.

Wrangling the RUS

For Outlook clients to work correctly, we needed to ensure that the provisioning solution was the ultimate authority on setting attribute values. We also need the Recipient Update Service (RUS) to finish the job, so to speak, on account setup, but the RUS should run only after provisioning is finished. To achieve this, the RUS instances for the districts (all 178 of them) were set to Never for their schedules and the provisioning solution used code that triggered the RUS to start after provisioning had completed.

Most districts had fewer than 5,000 users and far fewer updates on a daily basis, so the RUS finished quickly since the provisioning system kicked off an update instead of a rebuild at the end of its daily run. However, there were several districts with more than 10,000 users. Doing updates in those environments was more time-consuming, and if a full RUS rebuild was needed in these large domains, it could run for a week or more. Since the RUS evaluates all object types when it runs, the effect of doing a rebuild in such a large domain was debilitating.

Although we were not able to come up with a mitigation strategy (and we worked with members of the Exchange product group to try to come up with a solution), we enhanced the provisioning code to handle much of what the RUS normally does. Users get showInAddressBook and proxyAddresses attributes populated upon initial provisioning so that they can log in immediately instead of waiting for the RUS to complete its work. We also created a manual task that runs on one of the front-end Exchange 2003 protocol servers. It does a timed LDIFDE import and clears the RUS gatewayProxy address so that if a recipient policy is accidently applied in the environment, it is not put on the to-do list of every RUS in the enterprise.

Categorizer Issues

One interesting aspect of keeping student data from getting outside the school district was implementing controls to prevent students from e-mailing users outside their own district or Internet addresses outside the system. Instead of working through an elaborate system of connectors and related items, we decided to implement a lesser-known feature of Exchange 2003 and use connector restrictions on the existing Routing Group Connector from the district to the central hub where the front-end protocol servers sit. Two security groups were created in each domain—one for staff and one for students—and these groups were added to the connector restrictions.

Once we enabled connector restriction checking in the registry, we started to notice some very debilitating performance problems. Messages would be stuck in the categorizer and local delivery queues for hours—even for messages destined to recipients on the same server. These problems got worse over time as additional districts migrated. As a short-term solution we turned off connector restriction checking and the mail delivery performance improved. Further inspection revealed that the categorizer was checking not only the group memberships of the two groups within its own domain upon message submission, but also the group memberships of the same two groups in the other 177 domains as well. This was done for every message going into or out of the server.

We subsequently opened a support case with Microsoft. After evaluation, Microsoft is currently in the process of generating a hotfix to alleviate the issue by allowing the categorizer to work in a reduced topology complexity mode. Further details on the hotfix are not available as we go to press, but the goal is to provide a setting that allows the categorizer to make certain assumptions about the routing topology and not check group membership across all of the connectors.

Lessons Learned

If there's one thing I've learned all over again from this project, it's that business requirements drive everything we do as consultants. There were many times where the feasibility of the business requirement had to be weighed against the complexity of the solution. The decisions and choices I've described in this column are those we felt most appropriate given our environment (including elements I haven't had time to discuss here), our resources, and our business requirements. For you, the decisions may be different, but hopefully this column has given you some things to think about. Fortunately for us, Exchange 2003 and Active Directory allowed plenty of flexibility to meet all of the requirements in this complex migration. Once you get past the shocking scale of the Kentucky Department of Education deployment, the actual solution is rather simple and built on what I feel are underutilized features of Active Directory and Exchange 2003.

The second valuable lesson is to practice an even-keeled approach toward troubleshooting and risk/problem management. There are times when local troubleshooting is appropriate, and other times when calling on the advanced resources of Microsoft Product Support Services is required.

Looking Forward

The Kentucky Department of Education environment is certainly unique. In the course of this deployment, almost every feature in Exchange has been utilized or customized to satisfy the department's requirements. Now that the entire environment has been deployed and migrated successfully, the stage is set for implementing additional functionality. The Department of Education has already seen school district interest in the Windows Mobile® platform and expects the mobile device movement to continue.

There is significant value to be gained from server consolidation. The Kentucky Department of Education is currently looking at virtualizing some aspects of the Active Directory environment. And with Exchange 2007 on the horizon, the department has started looking at the requirements for an upgrade of their messaging systems.

The Kentucky Department of Education is also looking at additional collaboration offerings in Live Communication Server, Live Meeting Server, and so on. Microsoft Operations Manager 2005 is also currently under deployment for management and monitoring of the Exchange and Active Directory servers.

The Results

Every child in the Kentucky public school system will be affected by the quality and pervasiveness of this implementation—including my own children and the children of the people with whom I worked on this project. This implementation has poised the Kentucky Department of Education to go further faster with technology, and in a secure and standardized way, than ever before. I'm proud to have been a part of such an undertaking.

Whitney Roberts is a Principal Consultant with KiZAN Technologies (www.kizan.com) and has been working with Exchange since the release of Exchange 4.0.

© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.