Deploying Applications to Azure Nodes in Windows HPC Server 2008 R2

The method that you use to deploy applications to Azure Nodes depends on what is required to install your application, as outlined in the following table.

Each Azure Node deployment is associated with a Windows Azure storage account (as specified in the node template). A cluster administrator can stage files (such as applications, SOA services, XLLs, cluster-enabled Excel workbooks, and utilities) to the storage account by using the hpcpack commands. You can use hpcpack create to package files or folders in a compressed format (.zip) that can be uploaded to Azure storage. Each application, SOA service, or XLL must be packaged separately, and the package must include any required dependencies such as DLLs or configuration files. You can then use hpcpack upload to upload the package to the storage account. You can run the hpcpack commands on the head node, or on a computer that has the HPC client utilities installed.

All packages in the storage account are automatically deployed to new Azure Node instances during the provisioning process. This happens when you deploy a set of Azure Nodes by using the HPC management utilities, and when your node instances are reprovisioned by the Window Azure system. The hpcsync command runs on each Azure Node and copies all packages from storage to the node, and then extracts the files. If you upload packages to storage after the Azure Nodes are started, you can deploy the packages by running the hpcsync command manually on each Azure Node.

Note
If you create multiple Azure Node templates that reference the same storage account, the same staged files will be deployed to all Azure Node sets. To deploy different files to different node sets, create a separate Azure storage account for each Azure Node template.

The following diagram illustrates the basic workflow and mechanisms for copying applications to Azure Nodes:

By default, hpcsync extracts files to a location that is specified by the %CCP_PACKAGE_ROOT% environment variable. This variable is set on Azure Nodes during the provisioning process. The extracted files are placed in a folder that is determined as follows: %CCP_PACKAGE_ROOT%\<packageName>\<uploadTimeStamp>. This is the expected location for SOA services, XLLs, and Excel workbooks. However, this is not convenient for applications that cluster users will be calling in their command lines. To simplify the folder structure for executable files, you can set the relative path property for the package when you upload it to storage. hpcsync applies the relative path when extracting the files, so that the path is determined as follows: %CCP_PACKAGE_ROOT%\<relativePath>. Users can then specify the path to their application as in the following example of a job submit command: job submit %CCP_PACKAGE_ROOT%\myRelativePath\myapp.exe

Important considerations about hpcsync and CCP_PACKAGE_ROOT:

• On Azure Worker Nodes, the %CCP_PACKAGE_ROOT% folder is created on a 10 GB disk partition. This means that all application files on a node instance cannot exceed 10 GB. If an application has considerable input and output files, you can use a startup script to grant user permissions on the C:\ drives so that users can write to all available scratch space on the node. On Azure Virtual Machine Nodes, the size limit depends on how you set up the disk partitions when you create the VHD.
  
  
• When you run hpcsync manually, you can override the default location (%CCP_PACKAGE_ROOT%). For example, you can create a folder on each Azure Node and then specify that location when you run hpcsync. All packages will be extracted to that folder. However, any new node instances that are deployed (or automatically reprovisioned) will not include that folder, and packages will be automatically deployed to the default location. Additionally, cluster users only have write permissions to folders in %CCP_PACKAGE_ROOT%. Unless you modify folder permissions on the Azure Nodes, only administrators can run applications outside of %CCP_PACKAGE_ROOT%.
  
  
• When hpcsync deploys a package, none of the extracted files can have a full path over 256 characters. The root directories where the extracted files are temporarily and then finally placed can take up to 136 characters, leaving 120 characters for the file name, subdirectories (if any), and the relativePath (if specified). If the path for the extracted files exceeds 256 characters, the package deployment fails.
  
  

The hpcsync mechanism is sufficient for deploying SOA services, XLL files, and applications that can be installed by simply copying files to a node. If you need to run an installer to install an application, or if the application requires additional configuration steps such as setting environment variables, adding firewall exceptions, modifying folder permissions, or creating folders, you can include a startup script in the node template. This script will run during the provisioning process after hpcsync runs, and can be used to configure the nodes and perform the required application installation steps. If the installation is complex, error prone, long-running, or requires manual input, you can deploy Azure Nodes by using the Virtual Machine (VM) option in the node template. This allows you to deploy Azure Nodes based on a VHD image that already includes installed applications. Startup scripts and Azure VM nodes are available in Microsoft HPC Pack 2008 R2 Service Pack 2 (and later).

For more information, see the following guides:

• Deploying Windows Azure VM Nodes in Windows HPC Server 2008 R2 Step-by-Step Guide
  
  
• Appendix 2: Configure a Startup Script for Windows Azure Nodes
  
  

This section provides information about how to package applications and stage them to Azure storage. Staged packages are automatically deployed to new Azure Node instances that you provision (or that are automatically reprovisioned by the Windows Azure system).

Note
You must be a cluster administrator or at least have the Azure subscription and storage account IDs to stage files to Azure Storage.

Requirements

---

If you are packaging a SOA service:

• The name of the package must be the name of the SOA service (that is, the service name that the SOA client specifies in the SessionStartInfo constructor). For example, serviceName.zip or serviceName_serviceVersion.zip.
  
  
• You must include the service DLL, any dependent DLLs, and the service configuration files in the package.
  
  
• The service configuration file must also be deployed to the head node. All settings are determined by the on-premises copy of the configuration file.
  
  
• Do not specify a relative path when you upload the package. SOA services must be unpacked to the default location.
  
  

If you are packaging an XLL file:

• The name of the package must be the name of the XLL file. For example, XLLName.zip.
  
  
• If the XLL has dependencies, place the XLL and supporting files in a folder and package the folder. The XLL must be in the top level of the folder (not in a subfolder).
  
  
• Do not specify a relative path when you upload the package. XLLs must be unpacked to the default location.
  
  

If you are packaging an Excel workbook:

• The name of the package must be the name of the workbook. For example, workbookName.zip.
  
  
• If the workbook has dependencies, place the workbook and supporting files in a folder and package the folder. The workbook must be in the top level of the folder (not in a subfolder).
  
  
• Do not specify a relative path when you upload the package. Workbooks must be unpacked to the default location.
  
  

If you are packaging an executable file (such as an MPI application), application installer, or utility that you will call from a startup script:

• You must include any dependent DLLs or files in the package.
  
  
• When you upload the package, specify the relative path property.
  
  

If you are packaging a startup script:

• The name of the package must be the name of the startup script. For example, startup.bat.zip.
  
  
• Do not specify a relative path when you upload the package. The startup script must be unpacked to the default location.
  
  
• If your startup script calls installers or utilities, ensure that you package and stage the required files separately.
  
  

Steps

---

As examples, the following procedures illustrate how to stage various types of application files to Azure storage.

Note
You do not need an elevated command prompt (run as Administrator) to run hpcpack create, but hpcpack upload does require elevation. To perform the following procedures, run the commands in an elevated command prompt window.

To package and stage a SOA service to Azure storage

---

1. If the SOA service is not already registered and deployed to the on-premises cluster, register the SOA service by placing a copy of the service configuration file in the service registration folder on the head node (typically this is C:\Program Files\Microsoft HPC Pack 2008 R2\ServiceRegistration). For detailed information, see Deploy and Edit the Service Configuration File.

2. Copy the service configuration file, the service assembly, and any dependent DLLs to an empty folder. For example, to a folder named C:\myFiles\myServiceFiles.

3. At an elevated command prompt window, run HpcPack create and specify a name for your package and specify the folder that contains your service files.

   Important
   The name of the package must be the name of the SOA service (that is, the service name that the SOA client specifies in the SessionStartInfo constructor).

   For example, to package the content of C:\myFiles\myServiceFiles as myServiceName.zip (and save the package to a folder called AzurePackages):

   hpcpack create C:\AzurePackages\myServiceName.zip C:\myFiles\myServiceFiles

4. Run HpcPack upload to upload the package to Azure storage by using the following command, where myHeadNode is the name of your head node, and myAzureTemplate is the name of the template that you used to deploy the Azure Nodes. For example:

   hpcpack upload C:\AzurePackages\myServiceName.zip /nodetemplate:myAzureNodeTemplate /scheduler:myHeadNode

To package and stage an XLL file or Excel workbook to Azure storage

---

1. If the XLL or workbook has dependencies on DLLs or other files, copy the XLL or workbook and its dependencies to a folder. For example, to a folder named c:\myFiles\myExcelFiles.

2. At an elevated command prompt window, run HpcPack create to package your XLL or workbook. Specify a name for the package and specify the XLL or workbook. The name of the package must be the name of the XLL file or of the Excel workbook.

   For example, if your XLL or workbook has dependencies, package the entire folder (and save the package to a folder called AzurePackages):

   hpcpack create C:\AzurePackages\myXLL.zip C:\myFiles\myExcelFiles

   If your XLL or workbook has no dependencies, you can package it directly. For example, to package C:\myFiles\myXLL.xll as myXLL.zip:

   hpcpack create C:\AzurePackages\myXLL.zip C:\myFiles\myXLL.xll

3. Run HpcPack upload to upload the package to Azure storage by using the following command, where myHeadNode is the name of your head node, and myAzureTemplate is the name of the template that you used to deploy the Azure Nodes. For example:

   hpcpack upload C:\AzurePackages\myXLL.zip /nodetemplate:myAzureNodeTemplate /scheduler:myHeadNode

   Note
   Workbook offloading jobs do not run on Azure Worker nodes. Ensure that you stage your workbooks to a storage account that is associated with an Azure VM node template, and that the VHD image that you use to deploy the nodes has Microsoft Excel installed.

To package and stage an executable file to Azure storage

---

1. Copy the executable and any dependencies or DLLs to a folder. For example, to a folder named C:\myFiles\myAppFiles.

2. At an elevated command prompt window, run HpcPack create to package your application files. Specify a name for your package and specify the folder that contains your application files.

   For example, to package the content of c:\myFiles\myAppFiles as myApp.zip (and save the package to a folder called AzurePackages):

   hpcpack create c:\AzurePackages\myApp.zip c:\myFiles\myAppFiles

3. Upload the package to Azure storage by using the following command, where myHeadNode is the name of your head node, and myAzureTemplate is the name of the template that you used to deploy the Azure Nodes. Specify a relative path for the application files. For example:

   hpcpack upload c:\AzurePackages\myApp.zip /scheduler:myHeadNode /nodetemplate:myAzureTemplate /relativepath:myApp

Packages that are staged to Azure storage are automatically deployed to new node instances. You can manually deploy packages, for example to verify that you have all the necessary dependencies in a package before automating the deployment to all new nodes, or to deploy packages to nodes that are already running. You can use clusrun and hpcsyncto deploy the files from the Windows Azure storage account to the Windows Azure nodes.

For example:

clusrun /nodegroup:AzureWorkerNodes hpcsync

To see a list of folders or files that have been deployed to the Windows Azure nodes, you can run the following command:

clusrun /nodegroup:AzureWorkerNodes dir %CCP_PACKAGE_ROOT% /s

• Deploying Windows Azure Nodes in Windows HPC Server 2008 R2
  
  
• How to deploy Azure Worker Nodes video
  
  
• Run an MPI job on Azure Nodes
  
  
• Offload Excel workbooks to Azure Nodes
  
  
• Windows HPC with Burst to Windows Azure: Application Models and Data Considerations