Inside SharePointTroubleshoot Messaging Integration

Pav Cherny

Code download available at: SharePoint2008_08.exe(416 KB)

Contents

General Troubleshooting Approach
SharePoint Messaging Architecture
Obtaining Diagnostic Information
Outbound Message Transfer
Inbound Message Transfer
Directory Management
Conclusion

In a well-designed SharePoint environment, information workers do not need to change their communication habits or work routines to collaborate. Instead of manually checking collaboration sites on an ongoing basis, SharePoint can deliver documents and other information, such as alerts and reminders, directly to their mailboxes.

Users can also participate in workflows by replying to these messages or by sending new items as e-mail messages to document libraries and lists.

However, integrating SharePoint® into a secure messaging environment poses challenges, and it introduces an opportunity for non-SharePoint components to impact SharePoint-based collaboration. The challenge is to isolate and eliminate root causes of problems in the integrated environment efficiently wherever they occur.

In this column, I discuss SharePoint messaging integration based on a proven troubleshooting strategy that delivers fast results. The idea is to locate the trouble spot efficiently and then zoom in with more detailed and targeted troubleshooting steps.

In a large organization, this phased investigation might mean handing over the case to another specialist after an initial analysis, especially if the problem is related to non-SharePoint components. In small and medium organizations, on the other hand, it is not uncommon that a single IT administrator must be able to troubleshoot all involved systems directly, which points even more to the need for using structured approaches.

To keep this column practice-oriented, I use a test environment with Active Directory®, Exchange Server 2007, and WSS 3.0. Step-by-step instructions to build this test environment, as well as the troubleshooting tools discussed in this column, in the companion material, available in the Code Downloads section of the TechNet Magazine Web site.

General Troubleshooting Approach

Perhaps even more so than in other interoperability scenarios, troubleshooting SharePoint messaging integration requires a highly structured approach. There are several factors that complicate this endeavor.

Obviously you have to deal with complex technologies, and while you are struggling with message transfer issues, message-format conversions, security aspects, and directory-management problems, you will find that some SharePoint error messages can be confusing. Internet newsgroups are filled with unresolved discussion threads and suggestions that can lead you down the wrong path—like running all Web applications under the application pool identity for SharePoint Central Administration in order to make messaging integration work.

But there are also positive aspects. SharePoint is very flexible. In fact, it can provide you with detailed troubleshooting information if you know where to look. With a few basic scripts, you can greatly improve your troubleshooting efficiency.

The first troubleshooting objective is to understand the particular issue with which you're dealing. You need to identify where the issue is located and what components it involves. You might also try to reproduce the problem in varying system configurations and study Windows® event logs and text-based log files. But this can be tricky as some issues only occur sporadically. Still, the better you can reproduce an issue, the faster you can identify and eliminate its root cause. Another important objective, often overlooked, is to document all configuration changes and fixes and to make sure they are applied on all relevant systems in the environment, such as on all front-end servers in a Web farm.

SharePoint Messaging Architecture

Whenever you are confronted with an issue related to messaging integration, it is a good idea to get a cup of coffee or tea first and then review the SharePoint messaging integration architecture before engaging in any troubleshooting. Otherwise you might end up fixing the wrong problem or component. For example, an "Access Denied" error that prevents you from creating a contact object in Active Directory is not necessarily an indicator that you need to fix access permissions in Active Directory, as you'll see later.

In any case, it is vital to understand the role of each component involved in messaging integration and how the individual components interact with each other. If you look at Figure 1 you'll see the important components in a SharePoint-Exchange connectivity scenario, and this is covered in detail in subsequent sections.

fig01.gif

Figure 1 SharePoint messaging integration architecture (Click the image for a larger view)

Obtaining Diagnostic Information

One of the most important troubleshooting tools is reliable diagnostic information. The all-time classic is the Windows event log. Among other things, you can control the level of detail SharePoint writes to the Windows event log on Web servers by using SharePoint Central Administration (on the Operations page, under Logging and Reporting, click on Diagnostic Logging). You can specify the least critical event to report to the event log and to the trace log. The trace log (located by default under %CommonProgramFiles%\Microsoft Shared\Web Ser­ver Extensions\12\Logs) is very useful if you are looking for low-level information, but can fill very quickly so use this feature sparingly.

Another way to obtain diagnostic information is to enable debugging for the affected Web app by configuring the corresponding web.config file, as outlined in Web Application Config.pdf in the companion material. SharePoint displays detailed trace information directly in the user interface. An advantage of this approach is that you can see the relevant error information without having to parse through megabytes of tracing data. A disadvantage is that you must change the system configuration of your Web application. Don't forget to back up the original web.config file and then revert back to the original configuration when you have finished troubleshooting.

You can also configure the SMTP service to write detailed processing information to the SMTP protocol log (in IIS Manager, select the Enable logging checkbox on the General tab). Make sure you install the ODBC Logging module along with the SMTP Service feature as outlined in SharePoint Messaging Integration.pdf in the companion material, even if you do not plan to use ODBC Logging. Otherwise, the SMTP service will not write the protocol log. By default, the SMTP protocol log is located at %SystemRoot%\System32\LogFiles\SmtpSvc1. It is a text file that enables you to analyze the SMTP communication in detail and determine whether a message has left the SharePoint server.

On the Hub Transport server communicating with the SMTP service, you can also enable protocol logging in the Send Connector and Receive Connector configuration (see the SharePoint Messaging Integration.pdf). You can use a variety of other tools to follow the path messages take in the Exchange Server 2007 environment across Hub Transport servers and Mailbox servers, such as Message Tracking, Queue Viewer, and Pipeline Tracing. If you want detailed information about troubleshooting Exchange Server 2007 mail flow issues, refer to the topic "Transport and Mailflow Issues" at go.microsoft.com/fwlink/?LinkID=120145.

On Active Directory domain controllers, you can gather detailed troubleshooting information by enabling diagnostic logging for Directory Access and other categories by setting corresponding registry entries under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics. A value of 0x0 for a category means that only critical events are logged whereas a value of 0x5 stands for all events, including debug information. Using a logging level greater than 0x3 can degrade server performance and quickly fill the Windows event log. During normal operations, all values should be set to 0x0.

Gathering detailed diagnostic information on SharePoint, Exchange, and Active Directory servers in a troubleshooting situation can help you locate a trouble spot quickly and reliably. Additional troubleshooting tools, such as TCP/IP tools (ping, tracert, telnet, and so on) and Network Monitor, might also come in handy, as most of the communication between these systems is carried out over the computer network. Microsoft® Network Monitor 3.1 is available at go.microsoft.com/fwlink/?LinkId=120143.

Outbound Message Transfer

Resources

Equipped with event logs, trace logs, protocol logs, message-tracking logs, and network traces, I am ready to troubleshoot SharePoint messaging integration. Let's start with the easy part: outbound message transfer. SharePoint supports outbound message transfer in a variety of configurations and doesn't require a local SMTP service on the Web servers. However, in a complete messaging integration scenario, it is the recommended configuration to use the local SMTP service. Figure 2 shows the key components in the scenario.

fig02.gif

Figure 2 Outbound message transfer (Click the image for a larger view)

This messaging scenario includes four stages: SharePoint must transfer the message to the SMTP service, the SMTP service must process the message, the Hub Transport server must receive the message, and Exchange Server 2007 must transfer the message to the final destination.

If SharePoint cannot transfer the message to the SMTP service, you will typically receive an error in the user interface, such as that an e-mail notification cannot be sent. The cause may be that the SMTP service is not running. Or perhaps the SMTP protocol log is telling you that the SMTP service rejected the submission attempt with error code 550 (Requested action not taken: mailbox unavailable), which indicates that the issue is located at the SMTP service.

To verify that it is not a SharePoint issue, you can use a basic script (see SMTP.vbs in the companion material) to submit a test message via SMTP with the same sender and recipient information. If it is an SMTP service issue, the script will not succeed. In fact, this script might give you valuable clues about the root cause of the problem, which SharePoint unfortunately does not disclose even with debugging enabled, as illustrated in Figure 3.

fig03.gif

Figure 3 Unable to relay messages (Click the image for a larger view)

By granting the Web server relay permissions in the SMTP service configuration and restarting the SMTP service, you can solve the SharePoint message submission issue. Now the message can reach the SMTP service, and the next important question is whether the SMTP service can route the message to the Hub Transport server.

If the message remains in the Queue folder, the SMTP service is unable to contact the Hub Transport server. That's because either the Microsoft Exchange Transport service is not running or because of a network communication or configuration issue. By using the Telnet client, you can verify quickly and easily whether you can connect to the destination port of a Receive Connector configured for internal communication.

This isn't necessarily TCP port 25. In fact, if the SMTP service uses this port, you might transfer the messages to the default Receive Connector for Hub Transport servers, which is configured to block anonymous message submissions. In this case, you'll see a message file in the Drop folder. (The Windows SharePoint Services Timer service must be stopped; if not, the message disappears after a few seconds.) The message file is a delivery status notification stating that the message was rejected with "Diagnostic-code: smtp;530 5.7.1 Client was not authenticated."

To solve this issue, you should create a dedicated Receive Connector for your SharePoint servers. Because your SharePoint servers are internal systems, choose the Externally Secured authentication option. In this way, you do not need to grant extended rights to the ANONYMOUS LOGON account. Also, consider using IPsec to protect the communication between the SharePoint servers and the Hub Transport server.

If messages leave the SMTP Queue folder and the SMTP protocol logs on the SharePoint server and on the Hub Transport server (check out the %ProgramFiles%\Microsoft\Exchange Server\TransportRoles\Logs\ProtocolLog\SmtpReceive folder) indicate a successful message transfer, you can consider the job done, provided Exchange Server 2007 can route the message to the destination. As mentioned earlier, use mail flow troubleshooting tools included in Exchange Server 2007 to investigate any issues if messages do not reach the intended Exchange recipients. Note that incorrect recipient information, spam filters, and message size restrictions can prevent a final delivery at this stage.

Inbound Message Transfer

In the opposite direction, message transfer is slightly more complicated because potential issues are more subtle. For example, the SharePoint product documentation recommends configuring MX records in DNS for SharePoint e-mail domains, which is a good way to misroute messages in an Exchange organization.

Exchange Server 2007 uses Send Connectors with associated address spaces to transfer messages. It might route your SharePoint messages to Edge Transport servers for transfer over the Internet even though your SharePoint servers are internal. For internal message transfer, use Send Connectors with detailed address spaces instead and specify your SharePoint servers running the SMTP service as smart hosts in the connector configuration (see the SharePoint Messaging Integration.pdf in the companion download). Establishing a well-defined routing topology with dedicated bridgehead servers helps to achieve reliable message transfer from Exchange to SharePoint.

In order to verify that messages arrive at your SharePoint servers, stop the Windows SharePoint Services Timer service. Messages will pile up in the Drop folder because it is the Timer service that processes the messages and places file attachments in document libraries associated with the recipient e-mail addresses, as indicated in Figure 4. If the messages do not arrive in the Drop folder, use the Queue Viewer on your Hub Transport server to see if Exchange routes the messages correctly. Then investigate network communication issues that might prevent the Hub Transport server from communicating with the SMTP service on the SharePoint server.

fig04.gif

Figure 4 Inbound message transfer (Click the image for a larger view)

When you restart the Timer service, you should see the message items disappear from the Drop folder. However, if message attachments do not end up as documents in the destination libraries even though SharePoint indicates successful message processing in the Windows event log, it means that Exchange has sent the messages in Exchange RTF format. To investigate this problem, shut down the Timer service again, send another message with an attachment from your Outlook® client, and then open the message in Notepad after it arrives in the Drop folder. Now search for the string "winmail.dat." If you find it, Exchange Server is sending the messages in the wrong format.

SharePoint requires message attachments to be encoded in MIME or UUENCODE format. Also, SharePoint does not show any winmail.dat-related processing issues in the Windows event log (see Figure 5).The file attachments simply won't appear in the document library. Use Notepad as a troubleshooting tool and eliminate the formatting issue by configuring a Remote Domain definition in Exchange Management Console for the SharePoint e-mail domain. On the Format of original message sent, as attachment to journal report tab, under Exchange rich-text format, select Never use.

fig05.gif

Figure 5 Winmail.dat message processing (Click the image for a larger view)

Directory Management

One of the more useful Timer service events is event 6873, which states that an error occurred while processing an incoming e-mail file because of an unknown alias. This happens if an Exchange user specifies an incorrect SharePoint e-mail address when sending messages, such as a nonexistent document library.

You can mitigate this issue by configuring the Directory Management Service in the Incoming e-mail settings in SharePoint Central Administration to create recipient objects for mail-enabled document libraries in Active Directory. Exchange users can then conveniently select these recipient objects with correct address information from the address book.

Be aware, however, that you are now opening a whole new can of troubleshooting worms. The Directory Management Service fails in a secured system configuration based on the principle of least privilege because this feature has elevated permission requirements on the Web server. What's more, the product documentation (such as technet.microsoft.com/library/cc262947.aspx) somewhat overstates the situation when it suggests that you must grant the Central Administration application pool account the "Create, delete, and manage user accounts" right in the OU you specified for the SharePoint recipient objects in Active Directory.

The Directory Management Service creates contact and group objects, and, therefore, the Central Administration application pool account requires full control over contact and group objects in this OU, but it doesn't require administrative permissions for user accounts. If you set up the Directory Management Service in a Web farm as outlined and try to mail-enable a document library, chances are very good that you will encounter an "Access is denied" error.

The error information points toward Active Directory permission issues, and SharePoint admins are quick to assign full control for Everyone to the SharePoint OU. However, this not only weakens Active Directory security, it also does not solve the problem.

Structured troubleshooting requires that you locate the trouble spot first and then apply targeted configuration changes. If you follow this approach and check the Windows event log on the domain controller and perhaps trace network communication by using Network Monitor, you may find out that SharePoint does not access Active Directory when this problem occurs. So it is unlikely that Active Directory configuration changes will fix the problem. The problem is occurring on the SharePoint server.

Unfortunately it is exceptionally hard to obtain more useful information about this permission issue. SharePoint does not record any additional information in the Windows event log. However, if you enable SharePoint debugging, you can see the call stack (as shown in Figure 6), which suggests that the Directory Management Service uses the CreateContact method of a SharePoint Web service. The SharePoint SDK will tell you that this is the Email Integration Web Service (<WSS_server>:<admin_port>/_vti_bin/SharepointEmailWS.asmx).

Figure 6 Debug output

Server was unable to process request. ---> Access is denied.
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message,      WebResponse response, Stream responseStream, Boolean asyncCall) 
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[]      parameters) 
   at Microsoft.SharePoint.DirectorySoap.SPDirectoryManagementProxy.CreateContact(String Alias,      String FirstName, String LastName, String ForwardingEmail, ContactFlags Flags) 
   at Microsoft.SharePoint.SPList.UpdateDirectoryManagementService(String oldAlias, String      newAlias) 
   at Microsoft.SharePoint.SPList.Update(Boolean bFromMigration) 
   at Microsoft.SharePoint.SPList.Update() 
   at Microsoft.SharePoint.ApplicationPages.EmailSettingsPage.SubmitButton_Click(Object sender,      EventArgs args) 
   at System.Web.UI.WebControls.Button.OnClick(EventArgs e) 
   at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) 
   at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String      eventArgument) 
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean      includeStagesAfterAsyncPoint)

Let's take a look at SharepointEmailWS.asmx in a Web browser to see the list of supported operations. You can see the CreateContact method, and clicking on the CreateContact link reveals the SOAP messages you must send to this Web service if you want to create a contact in Active Directory. This is great because now you can create a script-based troubleshooting tool (see SharepointEmailWS.vbs in the companion material) to work outside the SharePoint user interface, making it easier to specify different user accounts during test runs. Figure 7 shows the information returned if you run this script under the Central Administration application pool account (left) and under the SharePoint Administrator account (right).

fig07.gif

Figure 7 Two different "Access is denied" errors (Click the image for a larger view)

The error messages differ! Both messages state that access is denied, but the error on the left indicates a processing issue and the error on the right doesn't. So what is the difference between application pool accounts and the SharePoint Administrator account?

The SharePoint Administrator is a local Administrator on the SharePoint server while application pool accounts are not. Adding the application pool accounts of your Web app to the local Administrators group and restarting the Web server solves the problem, but this is not great news. I consider running application pool accounts with administrative privileges on production Web servers unacceptable.

Why does an application pool account require these elevated permissions? Again, the script can reveal the answer. If you grant all users local Administrator permissions on your Web server for testing purposes, you'll find out that application pool accounts can use the Email Integration Web Service while access remains denied for all other accounts, including SharePoint administrators. This means that the Email Integration Web Service uses the application pool configuration as the basis for access-control decisions.

The application pool configuration is part of the IIS metabase (or applicationHost.config file in IIS 7.0), and access to the metabase is restricted to local Administrators by default. Fortunately it is possible to grant non-administrator accounts access to the metabase using the Metabase Explorer included in the IIS 6.0 Resource Kit tools. On IIS 7.0, it is even easier to grant full control to the applicationHost.config file; simply run the following commands:

CACLS "%SystemDrive%\Windows\System32\inetsrv\config\applicationHost.config" 
/G "<application pool account>" :F /E iisreset /noforce

To sum up, in order to create contact objects in Active Directory, the Central Administration application pool requires full control over contact and group objects in the target OU. And the pool accounts of the Web applications require full access to the metabase on the SharePoint servers in the Web farm. Now you are ready to create contact objects using the SharePoint user interface.

But wait, there's more! Creating recipient objects in Active Directory is only half of the directory integration story. The other half is generating Exchange-related recipient attributes and updating server-based address books. For example, if you update the global address list on your Exchange server by using the Exchange Management Shell command Update-GlobalAddressList "Default Global Address List," you can find newly created recipient objects for SharePoint document libraries in the Outlook address book. But sending messages to these recipients results in nondelivery reports because of incorrect address information, as illustrated in Figure 8.

fig08.gif

Figure 8 Undeliverable message to a document library (Click the image for a larger view)

The IMCEAEX acronym stands for Internet Mail Connector Encapsulated Address for Exchange. It indicates an encapsulated non-Exchange recipient address in a format that the old Exchange Internet Mail Connector can handle. But we are dealing with Exchange Server 2007 and native SMTP-based Send Connectors nowadays.

The point is that the SharePoint Email Integration Web Service does not write the address attributes that Exchange Server 2007 requires for message transfer. It only writes the given name, surname, display name, and target address (and, optionally, the ms-Exch­RequireAuthToSendTo) attributes. But it doesn't set the mail nickname, or the legacy Exchange DN or proxy addresses, and it doesn't associate the recipient object with an Exchange recipient policy.

To solve this issue, use the Get-Mailbox cmdlet in the Exchange Management Shell to obtain all recipient objects with a target address that points to the SharePoint e-mail domain. And pipe the output to the Set-MailContact cmdlet to activate Exchange recipient policies, as follows:

Get-Contact | where { $_.WindowsEmailAddress –like
  '*sharepoint.contoso.com' } | Set-MailContact
  -EmailAddressPolicyEnabled:$True

Alternatively, use the Exchange Man­agement Console and select the checkbox "Automatically update e-mail addresses based on e-mail address policy" in the contact object properties.

WSS 3.0 and MOSS 2007 support full messaging integration out of the box to enable e-mail based alerts and notifications, workflows, and document submissions. While it is not a trivial task to configure the systems correctly, message integration can increase worker productivity. Specifically, you must make sure that both inbound and outbound message transfer works properly. You should also ensure that directory integration works.

SharePoint error messages are not always intuitive or informative. However, I've shown you ways to get to the bottom of integration issues, locate root causes, and eliminate them reliably without having to jeopardize security on your SharePoint servers.

Pav Cherny is an IT expert and author specializing in Microsoft technologies for collaboration and unified communication. His publications include white papers, product manuals, and books with a focus on IT operations and system administration. Pav is President of Biblioso Corporation, a company that specializes in managed documentation and localization services.

© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.