Azure Batch client library for .NET
Azure Batch allows users to run large-scale parallel and high-performance computing (HPC) batch jobs efficiently in Azure.
Use the client library for to:
Create and manage Batch jobs and tasks
View and perform operations on nodes in a Batch pool
Source code | Package (NuGet) | API reference documentation | Product documentation
Getting started
Install the package
Install the client library for .NET with NuGet:
dotnet add package Azure.Compute.Batch --prerelease
Prerequisites
An Azure account with an active subscription. If you don't have one, create an account for free.
A Batch account with a linked Azure Storage account. You can create the accounts by using any of the following methods: Azure CLI | Azure portal | Bicep | ARM template | Terraform.
Visual Studio 2019 or later, or the .NET SDK version 6.0 or later.
Authenticate the client
Batch account access supports two methods of authentication: Shared Key and Microsoft Entra ID.
We strongly recommend using Microsoft Entra ID for Batch account authentication. Some Batch capabilities require this method of authentication, including many of the security-related features discussed here. The service API authentication mechanism for a Batch account can be restricted to only Microsoft Entra ID using the allowedAuthenticationModes property. When this property is set, API calls using Shared Key authentication will be rejected.
Authenticate using Microsoft Entra ID
Azure Batch provides integration with Microsoft Entra ID for identity-based authentication of requests. With Azure AD, you can use role-based access control (RBAC) to grant access to your Azure Batch resources to users, groups, or applications. The Azure Identity library provides easy Microsoft Entra ID support for authentication.
var credential = new DefaultAzureCredential();
BatchClient _batchClient = new BatchClient(
new Uri("https://<your account>.eastus.batch.azure.com"), credential);
Authenticate using Shared Key
You can also use Shared Key authentication to sign into your Batch account. This method uses your Batch account access keys to authenticate Azure commands for the Batch service. You can find your batch account shared keys in the portal under the "keys" section or you can run the following CLI command
az batch account keys list --name <your-batch-account> --resource-group <your-resource-group-name>
var credential = new AzureNamedKeyCredential("<your account>", "BatchAccountKey");
BatchClient _batchClient = new BatchClient(
new Uri("https://<your account>.eastus.batch.azure.com"),
credential);
Key concepts
Thread safety
We guarantee that all client instance methods are thread-safe and independent of each other (guideline). This ensures that the recommendation of reusing client instances is always safe, even across threads.
Additional concepts
Client options | Accessing the response | Long-running operations | Handling failures | Diagnostics | Mocking | Client lifetime
Examples
The Azure.Compute.Batch package supports synchronous and asynchronous APIs.
The following section provides several synchronous code snippets covering some of the most common Azure Batch related tasks:
- Create a pool
- Retrieve a pool
- List pools
- Retrieve a node
- List nodes
- Create a job
- Retrieve a job
- List jobs
- Create a task
- Retrieve a task
- Retrieve an output file from a task
Create a Pool
In an Azure Batch workflow, a compute node (or node) is a virtual machine that processes a portion of your application's workload. A pool is a collection of these nodes for your application to runs on. For more information see Nodes and pools in Azure Batch.
Use the CreatePool
method with a BatchPoolCreateContent
instance to create a BatchPool
.
BatchClient _batchClient = new BatchClient(
new Uri("https://<your account>.eastus.batch.azure.com"), new DefaultAzureCredential());
string poolID = "HelloWorldPool";
ImageReference imageReference = new ImageReference()
{
Publisher = "MicrosoftWindowsServer",
Offer = "WindowsServer",
Sku = "2019-datacenter-smalldisk",
Version = "latest"
};
VirtualMachineConfiguration virtualMachineConfiguration = new VirtualMachineConfiguration(imageReference, "batch.node.windows amd64");
BatchPoolCreateContent batchPoolCreateOptions = new BatchPoolCreateContent(
poolID, "STANDARD_D1_v2")
{
VirtualMachineConfiguration = virtualMachineConfiguration,
TargetDedicatedNodes = 2,
};
// create pool
_batchClient.CreatePool(batchPoolCreateOptions);
Retrieve a Pool
GetPool
can be used to retrieve created pools
BatchClient _batchClient = new BatchClient(
new Uri("https://<your account>.eastus.batch.azure.com"), new DefaultAzureCredential());
BatchPool batchPool = _batchClient.GetPool("poolID");
Console.WriteLine(batchPool.Id);
Console.WriteLine(batchPool.Url);
Console.WriteLine(batchPool.AllocationState);
List Pools
GetPools
can be used to list all pools under the Batch account
BatchClient _batchClient = new BatchClient(
new Uri("https://<your account>.eastus.batch.azure.com"), new DefaultAzureCredential());
foreach (BatchPool item in _batchClient.GetPools())
{
Console.WriteLine(item.Id);
}
Retrieve a Node
A node is an Azure virtual machine (VM) that is dedicated to processing a portion of your application's workload. The size of a node determines the number of CPU cores, memory capacity, and local file system size that is allocated to the node. For more information see Nodes and pools in Azure Batch.
GetNode
can be used to retrieve an allocated BatchNode
from a pool.
BatchClient _batchClient = new BatchClient(
new Uri("https://<your account>.eastus.batch.azure.com"), new DefaultAzureCredential());
BatchNode batchNode = _batchClient.GetNode("<poolId>", "<nodeId>");
Console.WriteLine(batchNode.Id);
Console.WriteLine(batchNode.Url);
Console.WriteLine(batchNode.State);
List Nodes
GetNodes
can be used to list all BatchNode
allocated under a pool
BatchClient _batchClient = new BatchClient(
new Uri("https://<your account>.eastus.batch.azure.com"), new DefaultAzureCredential());
foreach (BatchNode item in _batchClient.GetNodes("poolID"))
{
Console.WriteLine(item.Id);
}
Create a Job
A job is a collection of tasks. It manages how computation is performed by its tasks on the compute nodes in a pool.
A job specifies the pool in which the work is to be run. You can create a new pool for each job, or use one pool for many jobs. You can create a pool for each job that is associated with a job schedule, or one pool for all jobs that are associated with a job schedule. For more information see Jobs and tasks in Azure Batch.
Use the CreateJob
method with a BatchJobCreateContent
instance to create a BatchJob
.
BatchClient _batchClient = new BatchClient(
new Uri("https://<your account>.eastus.batch.azure.com"), new DefaultAzureCredential());
_batchClient.CreateJob(new BatchJobCreateContent("jobId", new BatchPoolInfo() { PoolId = "poolName" }));
Retrieve a job
GetJob
can be used to retrieve a created BatchJob
BatchClient _batchClient = new BatchClient(
new Uri("https://<your account>.eastus.batch.azure.com"), new DefaultAzureCredential());
BatchJob batchJob = _batchClient.GetJob("jobID");
Console.WriteLine(batchJob.Id);
Console.WriteLine(batchJob.State);
List jobs
GetJobs
can be used to list all BatchJob
allocated under a Batch Account
BatchClient _batchClient = new BatchClient(
new Uri("https://<your account>.eastus.batch.azure.com"), new DefaultAzureCredential());
foreach (BatchJob item in _batchClient.GetJobs())
{
Console.WriteLine(item.Id);
}
Create a task
A task is a unit of computation that is associated with a job. It runs on a node. Tasks are assigned to a node for execution, or are queued until a node becomes free. Put simply, a task runs one or more programs or scripts on a compute node to perform the work you need done. For more information see Jobs and tasks in Azure Batch.
Use the CreateTask
method with a BatchTaskCreateContent
instance to create a BatchTask
.
BatchClient _batchClient = new BatchClient(
new Uri("https://<your account>.eastus.batch.azure.com"), new DefaultAzureCredential());
_batchClient.CreateTask("jobId", new BatchTaskCreateContent("taskId", $"echo Hello world"));
Retrieve a task
GetTask
can be used to retrieve a created BatchTask
BatchClient _batchClient = new BatchClient(
new Uri("https://<your account>.eastus.batch.azure.com"), new DefaultAzureCredential());
BatchTask batchTask = _batchClient.GetTask("<jobId>", "<taskId>");
Console.WriteLine(batchTask.Id);
Console.WriteLine(batchTask.State);
Retrieve an output file from a task
In Azure Batch, each task has a working directory under which it can create files and directories. This working directory can be used for storing the program that is run by the task, the data that it processes, and the output of the processing it performs. All files and directories of a task are owned by the task user.
The Batch service exposes a portion of the file system on a node as the root directory. This root directory is located on the temporary storage drive of the VM, not directly on the OS drive.
Tasks can access the root directory by referencing the AZ_BATCH_NODE_ROOT_DIR environment variable. For more information see Files and directories in Azure Batch.
GetTasks
can be used to list all BatchTask
allocated under a BatchJob
. GetTaskFile
can be used to retrive files from a BatchTask
BatchClient _batchClient = new BatchClient(
new Uri("https://<your account>.eastus.batch.azure.com"), new DefaultAzureCredential());
var completedTasks = _batchClient.GetTasks("jobId", filter: "state eq 'completed'");
foreach (BatchTask t in completedTasks)
{
var outputFileName = t.ExecutionInfo.ExitCode == 0 ? "stdout.txt" : "stderr.txt";
Console.WriteLine("Task {0} exited with code {1}. Output ({2}):",
t.Id, t.ExecutionInfo.ExitCode, outputFileName);
BinaryData fileContents = _batchClient.GetTaskFile("jobId", t.Id, outputFileName);
using (var reader = new StreamReader(fileContents.ToStream()))
{
Console.WriteLine(reader.ReadLine());
}
}
Troubleshooting
Please see Troubleshooting common batch issues.
Next steps
View more https://github.com/Azure/azure-sdk-for-net/blob/Azure.Compute.Batch_1.0.0-beta.1/sdk/batch/Azure.Compute.Batch/samples here for common usages of the Batch client library: Batch Samples.
Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit Contributor License Agreements.
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.