Monitor clusters with the Health Service

Applies to: Azure Stack HCI, versions 23H2 and 22H2; Windows Server 2022, Windows Server 2019, Windows Server 2016

The Health Service, first released in Windows Server 2016, improves the day-to-day monitoring and operational experience for clusters running Storage Spaces Direct.

Prerequisites

The Health Service is enabled by default with Storage Spaces Direct. No additional action is required to set it up or start it. To learn more about Storage Spaces Direct, see the Storage Spaces Direct overview.

Cluster performance history

Get live performance and capacity information from your Storage Spaces Direct cluster. See Get cluster performance history.

Health Service faults

Display any current faults to easily verify the health of your deployment. See View Health Service faults.

Health Service actions

Track the progress of Health Service actions that are performed autonomously. See Track Health Service actions.

Automation

This section describes workflows which are automated by the Health Service in the disk lifecycle.

Disk lifecycle

The Health Service automates most stages of the physical disk lifecycle. Let's say that the initial state of your deployment is in perfect health - which is to say, all physical disks are working properly.

Retirement

Physical disks are automatically retired when they can no longer be used, and a corresponding Fault is raised. There are several cases:

• Media Failure: the physical disk is definitively failed or broken, and must be replaced.
• Lost Communication: the physical disk has lost connectivity for over 15 consecutive minutes.
• Unresponsive: the physical disk has exhibited latency of over 5.0 seconds three or more times within an hour.

If the retired disk was serving as the cache for many other physical disks, these will automatically be reassigned to another cache disk if one is available. No special user action is required.

Restoring resiliency

Once a physical disk has been retired, the Health Service immediately begins copying its data onto the remaining physical disks, to restore full resiliency. Once this has completed, the data is completely safe and fault tolerant anew.

Blinking the indicator light

If possible, the Health Service will begin blinking the indicator light on the retired physical disk or its slot. This will continue indefinitely, until the retired disk is replaced.

Physical replacement

You should replace the retired physical disk when possible. Most often, this consists of a hot-swap - i.e. powering off the node or storage enclosure is not required. See the Fault for helpful location and part information.

Verification

When the replacement disk is inserted, it will be verified against the Supported Components Document (see the next section).

Pooling

If allowed, the replacement disk is automatically substituted into its predecessor's pool to enter use. At this point, the system is returned to its initial state of perfect health, and then the Fault disappears.

Supported Components Document

The Health Service provides an enforcement mechanism to restrict the components used by Storage Spaces Direct to those on a Supported Components Document provided by the administrator or solution vendor. This can be used to prevent mistaken use of unsupported hardware by you or others, which may help with warranty or support contract compliance. This functionality is currently limited to physical disk devices, including SSDs, HDDs, and NVMe drives. The Supported Components Document can restrict on model, manufacturer (optional), and firmware version (optional).

Usage

The Supported Components Document uses an XML-inspired syntax. We recommend using your favorite text editor, such as the free Visual Studio Code or Notepad, to create an XML document which you can save and reuse.

Sections

The document has two independent sections: Disks and Cache.

If the Disks section is provided, only the drives listed (as Disk) are allowed to join pools. Any unlisted drives are prevented from joining pools, which effectively precludes their use in production. If this section is left empty, any drive will be allowed to join pools.

If the Cache section is provided, only the drives listed (as CacheDisk) are used for caching. If this section is left empty, Storage Spaces Direct attempts to guess based on media type and bus type. Drives listed here should also be listed in Disks.

Example

<Components>

  <Disks>
    <Disk>
      <Manufacturer>Contoso</Manufacturer>
      <Model>XYZ9000</Model>
      <AllowedFirmware>
        <Version>2.0</Version>
        <Version>2.1</Version>
        <Version>2.2</Version>
      </AllowedFirmware>
      <TargetFirmware>
        <Version>2.1</Version>
        <BinaryPath>C:\ClusterStorage\path\to\image.bin</BinaryPath>
      </TargetFirmware>
    </Disk>
    <Disk>
      <Manufacturer>Fabrikam</Manufacturer>
      <Model>QRSTUV</Model>
    </Disk>
  </Disks>

  <Cache>
    <CacheDisk>
      <Manufacturer>Fabrikam</Manufacturer>
      <Model>QRSTUV</Model>
    </CacheDisk>
  </Cache>

</Components>

To list multiple drives, simply add additional <Disk> or <CacheDisk> tags.

To inject this XML when deploying Storage Spaces Direct, use the -XML parameter:

$MyXML = Get-Content <Filepath> | Out-String
Enable-ClusterS2D -XML $MyXML

To set or modify the Supported Components Document once Storage Spaces Direct has been deployed:

$MyXML = Get-Content <Filepath> | Out-String
Get-StorageSubSystem Cluster* | Set-StorageHealthSetting -Name "System.Storage.SupportedComponents.Document" -Value $MyXML

You can verify using the following PowerShell cmdlet:

Get-PhysicalDisk | Select Model, Manufacturer, FirmwareVersion

Health Service settings

Modify Health Service settings to tune the aggressiveness of faults or actions, turn certain behaviors on or off, and more. See Modify Health Service settings.

Additional References

• Get cluster performance history
• View Health Service faults
• Track Health Service actions
• Storage Spaces Direct overview