.png){kind=spacer}
hpcpack
Updated: December 21, 2011
Applies To: Windows HPC Server 2008 R2
In Microsoft® HPC Pack 2008 R2 SP1 and later releases, you can join Windows Azure nodes to your on-premises HPC cluster. hpcPack helps you package, upload, and download files (such as SOA service files, applications, and data) to and from a Windows Azure storage account. hpcPack also helps you mount a VHD drive from the storage account to the Azure nodes. When you provision or start a set of Azure Nodes with HPC server, any OPC files that that are on the storage account in the hpcpackages container are automatically deployed to the Azure nodes (even when node instances are automatically reprovisioned by the Window Azure system). If you upload packages to the hpcpackages container after the Azure Nodes are started, you can use hpcsync to manually deploy them to the Azure Nodes. Alternately, you can create different containers in your storage account for files that you want to manage manually or with scripts.
 Note |
|---|
Service Pack 3 for Microsoft HPC Pack 2008 R2 includes many additions to hpcPack functionality. For an overview of the changes, see What's New in Windows HPC Server 2008 R2 Service Pack 3. |
In this topic:
-
HpcPack create
-
HpcPack upload
-
HpcPack remove
-
HpcPack list
-
HpcPack view
-
HpcPack download
-
HpcPack mount
-
HpcPack unmount
-
End to end examples
Packages the specified folder or files in a format that can be uploaded to an Azure storage account. The files are packaged in an Open Packaging Convention (OPC) format or as a ZIP file.
If you are packaging a SOA service:
-
You must include the service DLL, any dependent DLLs, and the service configuration files in the package.
-
The service assembly attribute in the service configuration file must specify the service DLL filename only (not the path).
-
The name of the package must be the name of the SOA service (that is, the service name that the SOA client specifies in the
SessionStartInfoconstructor). For example, serviceName.zip or serviceName_serviceVersion.zip.
If you are packaging an XLL file:
-
You must include the XLL and any dependent DLLs or files in the package.
-
The name of the package must be the name of the XLL file. For example, XLLName.zip.
 Important |
|---|
When hpcsync deploys a package to the Azure Nodes, none of the extracted files can have a full path over 256 characters. The root directories where the extracted files are temporarily and then finally placed can take up to 136 characters, leaving 120 characters for the file name, subdirectories (if any), and the relativePath (if specified). If the path for the extracted files exceeds 256 characters, the package deployment fails. |
hpcPack create <packageName> <sourceFolder> [/d] [/0|/1|/9]
hpcPack create <packageName> <sourceFile1, file2, file3,…> [/d] [/0|/1|/9]
| Parameter | Description |
|---|---|
|
<packageName> |
Specifies the name of the package (the output OPC file). This must be the same as the service name when packaging SOA services, or as the XLL when packaging XLLs. The file is created relative to the current folder unless you specify an absolute path. For example: C:\AzurePackages\myPackage.zip. If you specify the path, file name, and extension of an existing file, that file is replaced. |
|
<folder> or <file list> |
Specifies a folder or a comma-separated list of files that you want to include in the package. |
|
/d |
Use this parameter if you specify a folder and do not want to include the subfolders. |
|
[/0|/1|/9] |
Specifies the level of compression. Compression options are as follows:
|
To create a package for the Lizard files and save it to C:\AzurePkgs. The executable, supporting files, and input file are in a folder named C:\Lizard:
hpcPack create C:\AzurePkgs\lizard.zip C:\Lizard
To create a package for a SOA service named “myService.dll” and save it to C:\AzurePkgs, the service DLL and configuration files are in a folder C:\Services\myService:
hpcpack create C:\AzurePkgs\myService.zip C:\Services\myService
To create a package for an Excel XLL file named “myXLL.xll”. The XLL is in the current directory that you see in the command prompt (for example, C:\Users\myName>), and the package will be saved to the current directory:
hpcPack create myXLL.zip myXll.xll
Uploads a package, file, or VHD to a Windows Azure storage account. You can specify an account by using the node template name, the head node and node template name, or the Windows Azure storage account name and storage key. By default, files and packages are uploaded to a container named hpcpackages in the storage account. The hpcsync command copies OPC packages from this container to the Azure nodes (when you run hpcsync manually, or as part of the provisioning process when Azure nodes are added to an HPC cluster). Because of the automatic sync, the hpcpackages container can be used for packages that you want to ensure are always present on the Azure nodes. Note that only OPC files will be synced automatically, other files types will not. Optionally, for VHDs or other files that you want to handle differently, such as job input or output files, you can upload them to a different container in the storage account.
|
Note |
|---|
If you upload a VHD, it must be of fixed size. It cannot be dynamically sized, which is the default. The hpcpack mount command only works for fixed size VHDs. |
hpcPack upload <packageName> /nodetemplate:<nodeTemplateName> [/container:<containerName>] [/relativePath:<path>] [/description:<”my description”>] [/blocksize:<blockSize>] [/blobtype:<PageBlob|BlockBlob>] [/usehttp]
hpcPack upload <packageName> /scheduler:<headNodeName> /nodetemplate:<nodeTemplateName> [/container:<containerName>] [/relativePath:<path>] [/description:<”my description”>] [/blocksize:<blockSize>] [/blobtype:<PageBlob|BlockBlob>] [/usehttp]
hpcPack upload <packageName> /account:<storageAccountName> /key:<storageKey> [/container:<containerName>] [/relativePath:<path>] [/description:<”my description”>] [/blocksize:<blockSizeInt>] [/blobtype:<PageBlob|BlockBlob>] [/usehttp]
| Parameter | Description | ||
|---|---|---|---|
|
<packageName> |
Specifies the package that you want to upload. The package must be a valid OPC package (you can use The expected location is relative to the current folder. You can specify an absolute path. For example: C:\AzurePackages\myPackage.zip. |
||
|
/nodetemplate |
Specifies an Azure Worker Role node template. HpcPack uses the Windows Azure account name and storage key that is associated with the node template. |
||
|
/scheduler |
Specifies the head node of the cluster. You can use this parameter along with the |
||
|
/account |
Specifies the public name of the Azure storage account (the same storage name that you specify in the Azure Node Template). This is the value of <storageAccountName> in the public URL of the account: http://<storageAccountName>.blob.core.windows.net. If you specify |
||
|
/key |
Specifies the access key for the Azure storage account. You can view this value in the Summary page for the storage account (in the Azure portal). If you specify |
||
|
/container |
Specifies the name of the container in the Azure storage account that you want to upload the file to. If you do not specify this parameter, files and packages are uploaded to the default location: a container named hpcpackages in the storage account. The |
||
|
/relativePath |
Specifies a path relative to the root directory under which the packages are installed on the Azure Nodes by the The relative path also simplifies the target Azure directory, which is useful when you need to specify the command to run, for example:
You cannot install more than one package to the same relative path or to the same parent folder within a relative path. |
||
|
/description |
Specifies a description of the package. |
||
|
/blocksize |
Specifies the size of the block, in bytes, that the file will be separated into when uploading. The default block size is 4MB (4194304 bytes). 4MB is also the maximum block size. You might want to specify a smaller block size if you are experiencing an unstable network connection. |
||
|
/blobtype |
Specifies the type of blob to upload. The valid options are For more information, see Understanding Block Blobs and Page Blobs. |
||
|
/usehttp |
Specifies that you want to override the default https channel over port 443 and force the communication to http over port 80. |
To upload lizard.zip to the default container on the storage account that is associated with a node template named “AzureWorker”, and specify a relative path so that the command to run the executable is mpiexec %CCP_PACKAGE_ROOT%lizard\xhplmkl.exe (if you sync to the nodes using the default root folder):
hpcPack upload C:\AzurePkgs\lizard.zip /nodetemplate:AzureWorker /relativePath:lizard /description:”input file included, needs 32 cores”
To upload myService.zip to the default container on the storage account that associated with a node template named “AzureWorker”:
hpcPack upload C:\AzurePkgs\myService.zip /nodetemplate:AzureWorker
To upload myXLL.zip to the default container on the storage account that associated with a node template named “Azure Worker”:
hpcPack upload myXLL.zip /nodetemplate:”Azure Worker” /description:”64 bit XLL”
To upload myFile.txt to a container named “myInputFiles”:
hpcPack upload C:\myJobData\Job1\myFile.txt /account:<myStorageAccount> /key:<myStorageKey> /container:myInputFiles
Removes a package from a Windows Azure storage account. You can specify an account by using the node template name, the head node and node template name, or the Windows Azure storage account name and storage key.
|
Note |
|---|
This does not remove files from currently deployed Azure Nodes. Removing a package from storage ensures that the package will not be deployed to Azure Nodes when hpcsync runs. HpcSync runs automatically when Azure Nodes are deployed or when a node instance is reprovisioned by the Windows Azure system. You can also run hpcSync manually. |
hpcPack remove <packageName> /nodetemplate:<nodeTemplateName> [/container:<containerName>] [/usehttp]
hpcPack remove <packageName> /scheduler:<headNodeName> /nodetemplate:<nodeTemplateName> > [/container:<containerName>] [/usehttp]
hpcPack remove <packageName> /account:<storageAccountName> /key:<storageKey> > [/container:<containerName>] [/usehttp]
| Parameter | Description |
|---|---|
|
<packageName> |
Specifies the name of the package that you want to remove from the Windows Azure storage account. For example: myPackage.zip. |
|
/nodetemplate |
Specifies an Azure Worker Role node template. HpcPack uses the Windows Azure account name and storage key that is associated with the node template. |
|
/scheduler |
Specifies the head node of the cluster. You can use this parameter along with the |
|
/account |
Specifies the public name of the Azure storage account (the same storage name that you specify in the Azure Node Template). This is the value of <storageAccountName> in the public URL of the account: http://<storageAccountName>.blob.core.windows.net. If you specify |
|
/key |
Specifies the access key for the Azure storage account. You can view this value in the Summary page for the storage account (in the Azure portal). If you specify |
|
/container |
Specifies the name of the container in the Azure storage account that you want to remove the file from. If you do not specify this parameter, |
|
/usehttp |
Specifies that you want to override the default https channel and force the communication to http over port 80. |
hpcPack remove lizard.zip /nodetemplate:AzureWorker
hpcPack remove myXLL.zip /nodetemplate:”Azure Worker” /scheduler:headnode
Lists the packages that have been uploaded to the specified Windows Azure storage account. You can specify an account by using the node template name, the head node and node template name, or the Windows Azure storage account name and storage key.
hpcPack list /nodetemplate:<nodeTemplateName> [/container:<containerName>] [/usehttp]
hpcPack list /scheduler:<headNodeName> /nodetemplate:<nodeTemplateName> [/container:<containerName>] [/usehttp]
hpcPack list /account:<storageAccountName> /key:<storageKey> [/container:<containerName>] [/usehttp]
| Parameter | Description |
|---|---|
|
/nodetemplate |
Specifies an Azure Worker Role node template. HpcPack uses the Windows Azure account name and storage key that is associated with the node template. |
|
/scheduler |
Specifies the head node of the cluster. You can use this parameter along with the |
|
/account |
Specifies the public name of the Azure storage account (the same storage name that you specify in the Azure Node Template). This is the value of <storageAccountName> in the public URL of the account: http://<storageAccountName>.blob.core.windows.net. If you specify |
|
/key |
Specifies the access key for the Azure storage account. You can view this value in the Summary page for the storage account (in the Azure portal). If you specify |
|
/container |
Specifies the name of the container in the Azure storage account that you want to list the file in. If you do not specify this parameter, |
|
/usehttp |
Specifies that you want to override the default https channel over port 443 and force the communication to http over port 80. |
hpcPack list /nodetemplate:AzureWorker
hpcPack list /nodetemplate:”Azure Worker” /scheduler:headnode
Displays the attributes for a package that has been uploaded to a Windows Azure storage account. Attributes include the date it was uploaded, description (optionally defined during upload), and the target directory that you can use to access the files after they are deployed to the Azure Nodes.
You can specify an account by using the node template name, the head node and node template name, or the Windows Azure storage account name and storage key.
hpcPack view <packageName> /nodetemplate:<nodeTemplateName> [/container:<containerName>] [/usehttp]
hpcPack view <packageName> /scheduler:<headNodeName> /nodetemplate:<nodeTemplateName> [/container:<containerName>] [/usehttp]
hpcPack view <packageName> /account:<storageAccountName> /key:<storageKey> [/container:<containerName>] [/usehttp]
| Parameter | Description |
|---|---|
|
<packageName> |
Specifies the name of the package that you want to view. For example: myPackage.zip. |
|
/nodetemplate |
Specifies an Azure Worker Role node template. HpcPack uses the Windows Azure account name and storage key that is associated with the node template. |
|
/scheduler |
Specifies the head node of the cluster. You can use this parameter along with the |
|
/account |
Specifies the public name of the Azure storage account (the same storage name that you specify in the Azure Node Template). This is the value of <storageAccountName> in the public URL of the account: http://<storageAccountName>.blob.core.windows.net. If you specify |
|
/key |
Specifies the access key for the Azure storage account. You can view this value in the Summary page for the storage account (in the Azure portal). If you specify |
|
/container |
Specifies the name of the container in the Azure storage account that you want to view the files in. If you do not specify this parameter, |
|
/usehttp |
Specifies that you want to override the default https channel over port 443 and force the communication to http over port 80. |
hpcPack view lizard.zip /nodetemplate:AzureWorker
hpcPack view myXLL.zip /nodetemplate:”Azure Worker” /scheduler:headnode
Downloads a package or file from a Windows Azure storage account. You can specify an account by using the node template name, the head node and node template name, or the Windows Azure storage account name and storage key. By default, files and packages are downloaded from a container named hpcpackages in the storage account.
hpcPack download <packageName> /nodetemplate:<nodeTemplateName> [/container:<containerName>] [/path:<targetPath>] [/blocksize:<blockSize>] [/unpack] [/usehttp]
hpcPack download <packageName> /scheduler:<headNodeName> /nodetemplate:<nodeTemplateName> [/container:<containerName>] [/path:<targetPath>] [/blocksize:<blockSize>] [/unpack] [/usehttp]
hpcPack upload <packageName> /account:<storageAccountName> /key:<storageKey> [/container:<containerName>] [/path:<targetPath>] [/blocksize:<blockSize>] [/unpack] [/usehttp]
| Parameter | Description |
|---|---|
|
<packageName> |
Specifies the package or file that you want to download. |
|
/nodetemplate |
Specifies an Azure Worker Role node template. HpcPack uses the Windows Azure account name and storage key that is associated with the node template. |
|
/scheduler |
Specifies the head node of the cluster. You can use this parameter along with the |
|
/account |
Specifies the public name of the Azure storage account (the same storage name that you specify in the Azure Node Template). This is the value of <storageAccountName> in the public URL of the account: http://<storageAccountName>.blob.core.windows.net. If you specify |
|
/key |
Specifies the access key for the Azure storage account. You can view this value in the Summary page for the storage account (in the Azure portal). If you specify |
|
/container |
Specifies the name of the container in the Azure storage account that you want to download the file from. If you do not specify this parameter, files and packages are downloaded from default location: a container named hpcpackages in the storage account. |
|
/targetPath |
Specifies where you want to save the downloaded files. By default files are downloaded to the current directory. Path can be relative to the current directory, or an absolute path. For example: C:\AzurePackages\myPackage.zip. |
|
/blocksize |
Specifies the size of the block, in bytes, that the file will be separated into when downloading. The default block size is 4MB (4194304 bytes). 4MB is also the maximum block size. You might want to specify a smaller block size if you are experiencing an unstable network connection. |
|
/unpack |
Specifies that you want |
|
/usehttp |
Specifies that you want to override the default https channel over port 443 and force the communication to http over port 80. |
To download myPackage.zip from the default storage container to c:\myFiles on the local computer.
HpcPack download myPackage.zip /scheduler:headnode /nodetemplate:"nodetemplate name" /path:c:\myFiles
To download myData.zip from a storage container named myInputData and save the file, already unzipped, to the e:\approot folder (you can run this from an Azure Node to prefetch data for a job).
HpcPack download myData.zip /account:accountName /key:storageKey /container:myInputData /blocksize:<blocksize> /path:e:approot\myInputData /unpack
Run this command on an Azure node to mount a VHD file as a drive directly from the Windows Azure storage account, or to list all currently mounted drives on that node. You can specify an account by using the Windows Azure storage account name and storage key.
|
Note |
|---|
| The VHD that you mount must be of fixed size. It cannot be dynamically sized, which is the default. |
hpcPack mount <VHDname> /account:<storageAccountName> /key:<storageKey> > [/container:<containerName>] [/force|/snapshot]
hpcPack mount /list
| Parameter | Description |
|---|---|
|
<VHDname > |
Specifies the name of the VHD file that you want to mount from the Windows Azure storage account. For example: myDrive.vhd. |
|
/account |
Specifies the public name of the Azure storage account (the same storage name that you specify in the Azure Node Template). This is the value of <storageAccountName> in the public URL of the account: http://<storageAccountName>.blob.core.windows.net. If you specify |
|
/key |
Specifies the access key for the Azure storage account. You can view this value in the Summary page for the storage account (in the Azure portal). If you specify |
|
/container |
Specifies the name of the container in the Azure storage account that contains the VHD file. |
|
/force |
Force mount a VHD that is already mounted on another instance. |
|
/snapshot |
Mounts a snapshot of the specified VHD. This will mount the drive as read-only. Changes to the VHD in the storage account will not be visible to the Azure Node until the drive in unmounted and then mounted again. |
|
/list |
Lists all of the currently mounted drives on the Azure Node. |
To mount a read-only snapshot of myVHD.vhd from the specified storage account and container to the Azure Node.
hpcPack mount myVHD.vhd /account:accountName /key:storageKey /container:myVHDs /snapshot
To list the currently mounted drives on the Azure Node:
hpcPack mount /list
Run this command on an Azure node to unmount a VHD file that was previously mounted using hpcpack mount. You can specify a drive letter on the Azure node, or the VHD source (VHD name and storage account details).
hpcPack unmount <VHDname> /account:<storageAccountName> /key:<storageKey> > [/container:<containerName>]
hpcPack unmount <driveLetter>
| Parameter | Description |
|---|---|
|
<VHDname > |
Specifies the name of the VHD file that you want to unmount from the Azure Node. For example: myDrive.vhd. |
|
/account |
Specifies the public name of the Azure storage account (the same storage name that you specify in the Azure Node Template). This is the value of <storageAccountName> in the public URL of the account: http://<storageAccountName>.blob.core.windows.net. If you specify |
|
/key |
Specifies the access key for the Azure storage account. You can view this value in the Summary page for the storage account (in the Azure portal). If you specify |
|
/container |
Specifies the name of the container in the Azure storage account that contains the VHD file. |
|
<driveLetter> |
Specifies the drive letter on the Azure Node to which the VHD is mounted. |
To unmount drive F:\ from the Azure Node.
hpcPack unmount F:
To unmount myVHD.vhd from the Azure Node.
hpcPack unmount myVHD.vhd /account:accountName /key:storageKey /container:myVHDs
To help understand the workflow, the following steps outline the command sequence you can run to create, upload, and sync an application to a set of Azure Nodes that are already deployed. The fourth sample command demonstrates how to open a firewall exception on the Azure Nodes for your application (by using hpcfwutil).
hpcPack create C:\AzurePkgs\myApp.zip C:\AppFiles
hpcPack upload C:\AzurePkgs\myApp.zip /nodetemplate:AzureWorker /relativePath:apps
clusrun /nodegroups:AzureNodes hpcsync
clusrun /nodegroups:AzureNodes hpcfwutil register myApp %CCP_PACKAGE_ROOT%apps\myApp.exe
To submit a job that runs myApp.exe on the Azure Nodes:
Job submit /nodegroup:AzureNodes %CCP_PACKAGE_ROOT%apps\myApp.exe
To submit a job that includes a Node Preparation task. The Node Preparation task runs on each allocated node and downloads myApp.zip to the allocated Azure Node before running the main workload of the job:
Job new /nodegroup:AzureNodes
Job add <jobID> /type:nodepreparation “hpcpack download myApp.zip /unpack /account:<accountName> /key:<primaryKey> /targetpath:e:\approot”
Job add <jobID> e:\approot\myApp.exe
Job submit <jobID>
For additional examples of uploading and running applications on Azure Nodes, see:
