Configure cloud hybrid search - roadmap

We are in the process of combining the SharePoint Server 2013 and SharePoint Server 2016 content into a single content set. We appreciate your patience while we reorganize things. See the Applies To tag at the top of each article to find out which version of SharePoint an article applies to.

 

Applies to: Office 365, SharePoint Online, SharePoint Server 2013

Topic Last Modified: 2017-07-18

Learn how to configure cloud hybrid search for SharePoint Server by setting up a cloud Search service application in your SharePoint Server 2013 environment and connecting it to your search index in Office 365.

This article describes how you set up cloud hybrid search in an environment with SharePoint Server and SharePoint Online for Office 365 for enterprises. With the cloud hybrid search solution, you add crawled metadata from all your content, including on-premises content, to your search index in Office 365. When users search in Office 365, they get search results from both on-premises and from Office 365 content.

NoteNote:
If you are an Office 365 Dedicated customer, setting up cloud hybrid search requires engagement of SharePoint Service Engineering staff. Contact your Microsoft Service Delivery Manager for assistance. If you aren’t sure what type of customer you are, you can safely disregard this note.

To complete the configuration steps you’ll need these items:

  • The hardware and software that’s needed in a SharePoint Server hybrid environment.

  • A server or virtual machine for cloud hybrid search that has:

    • Minimum 100 GB storage, 16 GB RAM, and four 1.8 GHz CPUs.

    • SharePoint Server 2013 with Service Pack 1 and January 2016 cumulative update installed.

  • The accounts that are needed in a SharePoint Server hybrid environment, a search account for cloud hybrid search in SharePoint Server 2013, and a managed account for default content access in SharePoint Server. Ensure that the account has at least read access to the content to crawl.

  • The URL of your Office 365 tenancy.

  • The CreateCloudSSA.ps1 and Onboard-CloudHybridSearch.ps1 PowerShell scripts, find them in the Microsoft Download Center.

  • The search architecture plan you made for cloud hybrid search.

If you already completed step 1 when you configured a different hybrid solution, skip that step and go to the next.

 

Step

Description

1. Configure Office 365 for SharePoint hybrid

Configure your Office 365 tenant for a hybrid environment, including registering your domain, configuring UPN suffixes, and synchronizing your on-premises user accounts with Office 365.

2. Create a cloud Search service application in SharePoint Server 2013

Use the CreateCloudSSA.ps1 PowerShell script to create the cloud Search service application (cloud SSA) on the search farm that’s running SharePoint Server 2013.

3. Set up search architecture in SharePoint Server 2013 for cloud hybrid search (Optional)

If you planned a search architecture that’s different from the default one, set up the planned search architecture.

4. Connect your cloud Search service application to your Office 365 tenant

Use the Onboard-CloudHybridSearch.ps1 PowerShell script to onboard your cloud SSA and Office 365 tenant to cloud hybrid search. The script sets up the cloud SSA to interact with the Office 365 tenant and also sets up server-to-server authentication.

5. Create a content source to crawl for cloud hybrid search

We recommend adding a small file share first, you can add more on-premises content later.

6. Set up a separate Search Center to validate hybrid search results in Office 365 (Optional)

We recommend that you validate and tune a new search experience in a separate Search Center, while keeping the original search experience unchanged.

7. Start a full crawl of on-premises content for cloud hybrid search

When the crawl completes, your on-premises content shows up in the search results in your validation Search Center in Office 365 and in Office Delve.

8. Verify that cloud hybrid search works

Go to your Search Center in SharePoint Online in Office 365 and enter this query: "IsExternalContent:true". The results you get should show content from the on-premises content source that you've crawled.

9. Tune cloud hybrid search

Set up and tune the search experiences you’ve planned for your users.

10. Remove the validation Search Center and expose all users to hybrid search results.

Set your Search Center and any site search in Office 365 to use the default result source and set up the default result source with the search experiences that you’ve tuned. Your on-premises content shows up in the search results in your Search Center in Office 365, site search in Office 365, and in Office Delve.

The cloud SSA lets you crawl and add metadata from on-premises content to the search index in Office 365. Each search farm can have only one cloud SSA, but can have multiple SSAs in combination with the cloud SSA. You can’t convert an existing SSA to a cloud SSA.

You can create a cloud SSA in the SharePoint Central Administration website, just like an SSA. Here we describe how to create a cloud SSA by running the CreateCloudSSA.ps1 Windows PowerShell script from the Microsoft Download Center. This script installs the cloud SSA and the default search architecture for cloud hybrid search.

NoteNote:
If your organization restricts computers from connecting to the internet, you need to allow access to the endpoints (FQDNs) that cloud hybrid search uses. Include the endpoints in your outbound allow lists. The endpoints are listed in the SharePoint Online section of the article Office 365 URLs and IP address ranges and are marked for use with Hybrid Search.

On the search server in SharePoint Server 2013, follow these steps:

  1. Run the CreateCloudSSA.ps1 PowerShell script.

  2. When prompted, type:

    1. The host name of the search server in SharePoint Server 2013.

    2. The Search service account in this format: domain\username.

    3. A name of your choice for the cloud SSA.

    4. The name of the database server in SharePoint Server 2013

  3. Verify that you see the message that the cloud SSA was successfully created.

If you want to make your own Windows PowerShellscript for creating a cloud SSA, first study the CreateCloudSSA.ps1 Windows PowerShell script we’ve provided. Notice that the difference between creating a cloud SSA and an SSA is the value of the property CloudIndex. You set CloudIndex:true when you create a cloud SSA (you can’t change this value later). When CloudIndex is true, crawled metadata is not added to the on-premises search index. However, this doesn’t mean that the metadata is added to the Office 365 search index, you have to onboard the cloud SSA to cloud hybrid search for that to happen (see Connect your cloud Search service application to your Office 365 tenant). Ensure that your Windows PowerShell script:

  • Tests that the Search service account is a managed account, and makes it a managed account if it isn’t.

  • Includes -CloudIndex $true as an argument when it uses the New-SPEnterpriseSearchServiceApplication PowerShell cmdlet.

If you planned to use the default search architecture that you get when creating a cloud SSA, you can skip this step.

Otherwise, ensure that you have prepared the servers you need for your planned search architecture for cloud hybrid search, and follow the guidance for setting up your search architecture. This guidance is also applicable for cloud hybrid search.

This section guides you how to onboard your cloud SSA and Office 365 tenant to cloud hybrid search and covers:

  • Connecting your cloud SSA and your Office 365 tenant - When your cloud SSA and your Office 365 tenant are correctly connected, the cloud hybrid search solution is ready to add crawled metadata from on-premises content to the search index in Office 365. When you’ve onboarded your cloud SSA, check to see that your cloud SSA has the value 1 for the property IsHybrid. You check by running this Windows PowerShell command: $ssa.GetProperty(“CloudIndex”).

  • Configuring server-to-server authentication - Server-to-server authentication allows servers to access and request resources from one another on behalf of users.

On the search server in SharePoint Server 2013 follow these steps:

  1. Ensure that the date and time of the server is synchronized with the other servers in the SharePoint Server farm.

  2. Download and install the Microsoft Online Services Sign-In Assistant for IT Professionals RTW from the Microsoft Download Center.

  3. Download and install version 1.1.166.0 or later of the Azure Active Directory Module for Windows PowerShell, click Run to run the installer package.

  4. Download the OnBoard-CloudHybridSearch.ps1Windows PowerShell script from the Microsoft Download Center.

  5. If your environment is Office 365 Business, Office 365 Enterprise, Office 365 Education, Office 365 operated by 21Vianet, Office 365 Germany, or Office 365 US Government Defense, open an elevated Windows PowerShell prompt, and run the OnBoard-CloudHybridSearch.ps1 Windows PowerShell script as follows:

    Import-Module MSOnline
    
    .\OnBoard-CloudHybridSearch.ps1 -PortalUrl <SPOTenantPortalUrl> -CloudSsaId <CloudSSANameCreatd>
    

    SPOTenantPortalUrl is the URL of your company's or organization’s UNRESOLVED_TOKEN_VAL(SharePoint_Online) portal, and CloudSsaID is the name of the cloud SSA that you created earlier.

  6. If your environment is Office 365 US Government Communication, open an elevated Windows PowerShell prompt, and run the OnBoard-CloudHybridSearch.ps1 Windows PowerShell script as follows:

    Import-Module MSOnline
    
    .\OnBoard-CloudHybridSearch.ps1 -PortalUrl <SPOTenantPortalUrl> -CloudSsaId <CloudSSANameCreatd> -IsPortalForUSGovernment $true
    

    SPOTenantPortalUrl is the URL of your company's or organization’s UNRESOLVED_TOKEN_VAL(SharePoint_Online) portal, and CloudSsaID is the name of the cloud SSA that you created earlier.

  7. When prompted, type the global admin credentials for your Office 365 tenant.

We recommend that you start with a small on-premises content source, for example a small file share, to test. You can add more on-premises content sources later.

  1. Verify that the user account that is performing this procedure is an administrator for the cloud SSA.

  2. On the home page of Central Administration, in the Application Management section, click Manage service applications.

  3. On the Manage Service Applications page, click the cloud SSA.

  4. On the Search Administration Page, in the Crawling section, click Content Sources.

  5. On the Manage Content Sources page, click New Content Source.

  6. On the Add Content Source page, in the Name section, in the Name box, type a name for the new content source.

  7. In the Content Source Type section, select the type of content that you want to crawl.

  8. In the Start Addresses section, in the Type start addresses below (one per line) box, type the URLs from which the crawler should begin crawling.

  9. In the Crawl Settings section, select the crawling behavior that you want.

  10. In the Crawl Schedules section, to specify a schedule for full crawls, select a defined schedule from the Full Crawl list. A full crawl crawls all content that is specified by the content source, regardless of whether the content has changed. To define a full crawl schedule, click Create schedule.

  11. To specify a schedule for incremental crawls, select a defined schedule from the Incremental Crawl list. An incremental crawl crawls content that is specified by the content source that has changed since the last crawl. To define a schedule, click Create schedule. You can change a defined schedule by clicking Edit schedule.

  12. To set the priority of this content source, in the Content Source Priority section, on the Priority list, select Normal or High.

  13. Click OK.

After you’ve set up cloud hybrid search and completed a full crawl of your on-premises content, your Search Center in Office 365 automatically shows both on-premises and online search results. However, we recommend that you keep the original search experience unchanged while you validate and tune the new search experience in a separate Search Center.

Follow these steps to set up a separate Search Center in Office 365 to validate and tune of cloud hybrid search:

  1. Create a result source that retrieves search results from the search index of the tenant, but limits search results to Office 365 content by using a Query Transform. Change the default query transform to “(?(searchTerms) NOT IsExternalContent:true)”. This works because content that has the managed property IsExternalContent set to true (see About the IsExternalContent managed property) in the SharePoint Online search schema is on-premises content.

  2. Modify the Search Results Web Part in your Office 365 Search Center to use the result source that you just created. Your users get the original search experience in this Search Center.

  3. Create a second Office 365 Search Center that uses the default result source. This Search Center has hybrid search results when you’ve run a full crawl. Validate and tune your new search experience in this Search Center.

  4. Set up access so only testers and administrators have access to this site.

Here’s an example of a validation environment:

The illustration shows how content enters the Office 365 index from both a SharePoint Server content farm and from Office 365. The standard search center in Office 365 only retrieves Office 365 results from the search index, while the validation search ce
  1. On-premises content. During crawl, content is added to the Office 365 index.

  2. Office 365 content. During crawl, content is added to the Office 365 index.

  3. Default (or existing) Office 365 Search Center. This Search Center uses the custom result source that limits search results to only Office 365 content.

  4. Second Office 365 Search Center where you validate and tune how hybrid search results are shown. This Search Center uses the default result source and shows search results from both on-premises and Office 365 content.

An important part in this environment is the custom result source you use in the default or existing Office 365 Search Center. This result source keeps the search experience unchanged while you validate and tune how hybrid search results are displayed. An important piece in this custom result source is the IsExternalContent managed property in the SharePoint Online search schema. Before you set up cloud hybrid search, this managed property is empty. But, after you’ve set up cloud hybrid search and crawled your on-premises content, this property is set to true for all on-premises content. You to limit search results to show only Office 365 content with NOT IsExternalContent:true.

Start a full crawl of the content source. See Start, pause, resume, or stop a crawl in SharePoint Server 2013 or follow these steps:

  1. Verify that the user account that is performing this procedure is an administrator for the Cloud Search service application.

  2. On the home page of the SharePoint Central Administration website, in the Application Management section, click Manage service applications.

  3. On the Manage Service Applications page, click the cloud Search service application.

  4. On the Search Administration page, in the Crawling section, click Content Sources.

  5. On the Manage Content Sources page, in the list of content sources, point to the name of the content source that you want to crawl, click the arrow and then click Start Full Crawl. The value in the Status column changes to Crawling Full for the selected content source.

After the full crawl completes, verify that your on-premises content shows up in the search results in your validation Search Center in Office 365.

  1. Log in to Office 365 with your work or school account. Make sure that:

    • You have access to the validation Search Center.

    • You have access to the content in the content source that you have crawled. If you've done step 1 of this roadmap, you should have access.

    • Your organization hasn’t assigned user access rights to the on-premises content by using one of the default security groups in Windows Server Active Directory (AD), for example the Domain Users security group, learn more in Plan cloud hybrid search for SharePoint.

  2. Search for IsExternalContent:1 in the validation Search Center.

  3. Verify that your on-premises content shows up in the search results.

After you’ve set up cloud hybrid search and verified that you get search results from on-premises content in your validation Search Center in Office 365, set up the search experiences that you planned.

You might find this guidance useful:

After setting up and validating the planned search experiences, you might want to clear your search index in Office 365 for metadata from the on-premises content you’ve used during this work. This works differently from what you might be familiar with from SharePoint Server.

In the SharePoint Central Administration website you can use the option “Index reset” for an SSA to remove all content from the search index. This option does not work for cloud hybrid search because there is no direct communication between the cloud SSA in SharePoint Server 2013 and the search index in Office 365. If you only want to remove some on-premises metadata, remove that on-premises content source, or create a crawl rule that doesn’t crawl the URL of a file. If you need to remove all metadata from on-premises content from the search index in Office 365, open a ticket with Microsoft Support.

Show: