LLI Wiki — My Digital Knowledge Base

Azure Batch pool autoscale formula

Azure Batch autoscale formula that sets the number of dedicated nodes from pending tasks, with fixed time windows.

This autoscale formula for an Azure Batch pool sets the number of dedicated compute nodes from the pending tasks. It never goes above four nodes and uses two time windows.

Current formula

$varMaxTargetDedicatedNodes = 4;
$varActiveTasks = max($ActiveTasks.GetSample(15));
$varRunningTasks = max($RunningTasks.GetSample(15));
$varPendingTasks = max($PendingTasks.GetSample(15));
$varTaskSlotsPerNode = max($TaskSlotsPerNode);
$varCurrentDedicatedNodes = $CurrentDedicatedNodes;
$varUsableNodeCount = max($UsableNodeCount.GetSample(15));
$varPreemptedNodeCount = max($PreemptedNodeCount.GetSample(15));
$varTargetDedicatedNodes = $TargetDedicatedNodes;
$CurTime = time();
$CurMinute = $CurTime.minute;
$CurHour = $CurTime.hour;
$IsMorningOrEveningTime = ($CurHour >= 3 && $CurHour < 5) || ($CurHour >= 20 && $CurHour < 24);
$HasActivity = $varPendingTasks > 0;
$TargetDedicatedNodes = $IsMorningOrEveningTime ? $varMaxTargetDedicatedNodes : ($HasActivity ? 2 : 0);
$NodeDeallocationOption = taskcompletion;

How it works

Condition Target dedicated nodes
Between 03:00 and 05:00, or between 20:00 and 24:00 4
Other hours, with pending tasks in the last 15 minutes 2
Other hours, no pending task 0

NodeDeallocationOption = taskcompletion means that nodes removed from the pool are released once their running task has completed.

Previous formula

This version has been replaced by the current formula. It computes the target from the sampled pending tasks, with a minimum of 0 and a ceiling of 4 nodes.

$CurTime = time() + (2 * TimeInterval_Hour);
$CurHour = $CurTime.hour;
$CurMinute = $CurTime.minute;
$IsNightTime = ($CurHour == 21 && $CurMinute >= 50) || ($CurHour >= 22) || ($CurHour < 3);
$IsDayTime = !$IsNightTime;
$minCapacity = $IsNightTime ? 0 : 0;
$samples = $PendingTasks.GetSamplePercent(TimeInterval_Minute * 15);
$LastSampledActiveTasks = $samples < 70 ? max(0, $PendingTasks.GetSample(1)) : max($PendingTasks.GetSample(1), avg($PendingTasks.GetSample(TimeInterval_Minute * 15)));
$needcompute = min($LastSampledActiveTasks, 1);
$TargetDedicatedNodes = $needcompute ? ($minCapacity + 1) : $minCapacity;
$TargetDedicatedNodes = min($TargetDedicatedNodes, 4);
$NodeDeallocationOption = taskcompletion;

Notes

Mastodon LLIMastodon SocialFosstodon