Deploying Applications to Windows Azure Nodes in a Windows HPC Cluster

The method that you use to deploy applications to Windows Azure nodes depends on what is required to install your application, as outlined in the following table.

Each Windows Azure node deployment is associated with a Windows Azure storage account (as specified in the node template). A cluster administrator can stage files (such as applications, SOA services, XLLs, cluster-enabled Excel workbooks, and utilities) to the storage account by using the hpcpack commands. You can use hpcpack create to package files or folders in a compressed format (.zip) that can be uploaded to Azure storage. Each application, SOA service, or XLL must be packaged separately, and the package must include any required dependencies such as DLLs or configuration files. You can then use hpcpack upload to upload the package to the storage account. You can run the hpcpack commands on the head node, or on a computer that has the HPC client utilities installed.

All packages in the storage account are automatically deployed to new Windows Azure node instances during the provisioning process. This happens when you deploy a set of Windows Azure nodes by using the HPC management utilities, and when your node instances are reprovisioned by the Window Azure system. The hpcsync command runs on each Windows Azure node and copies all packages from storage to the node, and then extracts the files. If you upload packages to storage after the Windows Azure nodes are started, you can deploy the packages by running the hpcsync command manually on each Windows Azure node.

Note
If you create multiple Windows Azure node templates that reference the same storage account, the same staged files will be deployed to all Windows Azure node sets. To deploy different files to different node sets, create a separate Windows Azure storage account for each Windows Azure node template.

The following diagram illustrates the basic workflow and mechanisms for copying applications to Windows Azure nodes:

By default, hpcsync extracts files to a location that is specified by the CCP_PACKAGE_ROOT environment variable. This variable is set on Windows Azure nodes during the provisioning process. The extracted files are placed in a folder that is determined as follows: %CCP_PACKAGE_ROOT%\<packageName>\<uploadTimeStamp>. This is the expected location for SOA services, XLLs, and Excel workbooks. However, this is not convenient for applications that cluster users will be calling in their command lines. To simplify the folder structure for executable files, you can set the relative path property for the package when you upload it to storage. hpcsync applies the relative path when extracting the files, so that the path is determined as follows: %CCP_PACKAGE_ROOT%\<relativePath>. Users can then specify the path to their application as in the following example of a job submit command: job submit %CCP_PACKAGE_ROOT%\myRelativePath\myapp.exe

Important considerations about hpcsync and CCP_PACKAGE_ROOT:

• On Windows Azure worker nodes, the %CCP_PACKAGE_ROOT% folder is created on a 10 GB disk partition. This means that all application files on a node instance cannot exceed 10 GB. If an application has considerable input and output files, you can use a startup script to grant user permissions on the C:\ drives so that users can write to all available scratch space on the node.
  
  
• When you run hpcsync manually, you can override the default location (%CCP_PACKAGE_ROOT%). For example, you can create a folder on each Azure Node and then specify that location when you run hpcsync. All packages will be extracted to that folder. However, any new node instances that are deployed (or automatically reprovisioned) will not include that folder, and packages will be automatically deployed to the default location. Additionally, cluster users only have write permissions to folders in %CCP_PACKAGE_ROOT%. Unless you modify folder permissions on the Windows Azure nodes, only administrators can run applications outside of %CCP_PACKAGE_ROOT%.
  
  
• When hpcsync deploys a package, none of the extracted files can have a full path length longer than 256 characters. The root directories where the extracted files are temporarily and then finally placed can take up to 136 characters, leaving 120 characters for the file name, subdirectories (if any), and the relativePath (if specified). If the path for the extracted files exceeds 256 characters, the package deployment fails.
  
  

The hpcsync mechanism is sufficient for deploying SOA services, XLL files, and applications that can be installed by simply copying files to a node. If you need to run an installer to install an application, or if the application requires additional configuration steps such as setting environment variables, adding firewall exceptions, modifying folder permissions, or creating folders, you can include a startup script in the node template. This script will run during the provisioning process after hpcsync runs, and can be used to configure the nodes and perform the required application installation steps.

This section provides information about how to package applications and stage them to Windows Azure storage. Staged packages are automatically deployed to new Windows Azure node instances that you provision (or that are automatically reprovisioned by the Windows Azure system).

Note
You must be a cluster administrator or at least have the Windows Azure subscription and storage account IDs to stage files to Windows Azure storage.

Steps

---

As examples, the following procedures illustrate how to stage various types of application files to Windows Azure storage.

Note
You do not need an elevated command prompt (run as Administrator) to run hpcpack create. However, hpcpack upload requires elevation. To perform the following procedures, run the commands in an elevated command prompt window.

To package and stage a SOA service to Windows Azure storage

---

1. If the SOA service is not already registered and deployed to the on-premises cluster, register the SOA service by placing a copy of the service configuration file in the service registration folder on the head node (typically this is %CCP_HOME%\ServiceRegistration). For detailed information, see Deploy and Edit the Service Configuration File.

2. Copy the service configuration file, the service assembly, and any dependent DLLs to an empty folder. For example, co;y the files to a folder named C:\myFiles\myServiceFiles.

3. At an elevated command prompt, run HpcPack create and specify a name for your package and the folder that contains your service files.

   Important
   The name of the package must be the name of the SOA service (that is, the service name that the SOA client specifies in the SessionStartInfo constructor).

   For example, to package the content of C:\myFiles\myServiceFiles as myServiceName.zip (and save the package to a folder called AzurePackages):

   hpcpack create C:\AzurePackages\myServiceName.zip C:\myFiles\myServiceFiles

4. Run hpcpack upload to upload the package to Windows Azure storage by using the following command, where myHeadNode is the name of your head node, and myAzureTemplate is the name of the template that you used to deploy the Windows Azure nodes. For example:

   hpcpack upload C:\AzurePackages\myServiceName.zip /nodetemplate:myAzureNodeTemplate /scheduler:myHeadNode

To package and stage an XLL file or Excel workbook to Windows Azure storage

---

1. If the XLL or workbook has dependencies on DLLs or other files, copy the XLL or workbook and its dependencies to a folder. For example, to a folder named c:\myFiles\myExcelFiles.

2. At an elevated command prompt, run hpcpack create to package your XLL or workbook. Specify a name for the package and specify the XLL or workbook. The name of the package must be the name of the XLL file or of the Excel workbook.

   For example, if your XLL or workbook has dependencies, package the entire folder (and save the package to a folder called AzurePackages):

   hpcpack create C:\AzurePackages\myXLL.zip C:\myFiles\myExcelFiles

   If your XLL or workbook has no dependencies, you can package it directly. For example, to package C:\myFiles\myXLL.xll as myXLL.zip:

   hpcpack create C:\AzurePackages\myXLL.zip C:\myFiles\myXLL.xll

3. Run hpcpack upload to upload the package to Windows Azure storage by using the following command, where myHeadNode is the name of your head node, and myAzureTemplate is the name of the template that you used to deploy the Windows Azure nodes. For example:

   hpcpack upload C:\AzurePackages\myXLL.zip /nodetemplate:myAzureNodeTemplate /scheduler:myHeadNode

To package and stage an executable file to Windows Azure storage

---

1. Copy the executable and any dependencies or DLLs to a folder. For example, to a folder named C:\myFiles\myAppFiles.

2. At an elevated command prompt, run hpcpack create to package your application files. Specify a name for your package and specify the folder that contains your application files.

   For example, to package the content of c:\myFiles\myAppFiles as myApp.zip (and save the package to a folder called AzurePackages):

   hpcpack create c:\AzurePackages\myApp.zip c:\myFiles\myAppFiles

3. Upload the package to Windows Azure storage by using the following command, where myHeadNode is the name of your head node, and myAzureTemplate is the name of the template that you used to deploy the Windows Azure nodes. Specify a relative path for the application files. For example:

   hpcpack upload c:\AzurePackages\myApp.zip /scheduler:myHeadNode /nodetemplate:myAzureTemplate /relativepath:myApp

Packages that are staged to Windows Azure storage are automatically deployed to new node instances. You can manually deploy packages - for example, to verify that you have all the necessary dependencies in a package before automating the deployment to all new nodes, or to deploy packages to nodes that are already running. You can use clusrun and hpcsync to deploy the files from the Windows Azure storage account to the Windows Azure nodes.

For example:

clusrun /nodegroup:AzureWorkerNodes hpcsync

To see a list of folders or files that have been deployed to the Windows Azure nodes, you can run the following command:

clusrun /nodegroup:AzureWorkerNodes dir %CCP_PACKAGE_ROOT% /s

• Deploying Windows Azure Nodes with Microsoft HPC Pack
  
  
• How to deploy Azure Worker Nodes video
  
  
• Run an MPI job on Windows Azure nodes
  
  
• Offload Excel workbooks to Windows Azure nodes
  
  
• Windows HPC with Burst to Windows Azure: Application Models and Data Considerations