Orchestrate Model Training

Log one line, mlflow.log_metric("accuracy", 0.94), inside a job that runs on Azure Machine Learning, and that value is already saved to your workspace and plotted on the job's Metrics tab. That works because Azure Machine Learning uses MLflow^[1] as its native experiment-tracking framework, and the workspace itself is the tracking backend. MLflow calls a tracked execution a run; Azure Machine Learning calls the same thing a job, and the two words name the same record.

There is no separate MLflow tracking server to install or point at. A job that runs on Azure ML managed compute has its MLflow tracking URI (the address MLflow logs to) set to the workspace automatically, so jobs that use MLflow and run on Azure Machine Learning log their tracking information to the workspace^[2] with no configuration. The common trap is assuming you must stand up and configure your own MLflow server first; the workspace already is that backend.

Log automatically with autolog

You can hand-log each value, or let MLflow capture the standard ones. Insert a single call before your training code and MLflow logs the parameters, metrics, and the trained model artifact for supported frameworks such as scikit-learn, PyTorch, TensorFlow, and XGBoost.

Turn on automatic logging (place before training):

import mlflow
mlflow.autolog()   # capture params, metrics, and the model automatically

# ... your existing training code, unchanged ...
model.fit(X_train, y_train)

The mlflow.autolog() call decides what to track per framework^[2], so you do not write a log_metric line for every value.

Tracking from outside Azure ML compute

Code that runs off Azure ML managed compute, such as a script or notebook on your own laptop, does not inherit the workspace tracking URI. Point MLflow at the workspace once and the same logging calls flow to the same place.

Point MLflow at the workspace from a local run:

import mlflow
mlflow.set_tracking_uri(azureml_mlflow_uri)   # the azureml-mlflow URI of your workspace

For how to obtain that azureml_mlflow_uri, see Configure MLflow for Azure Machine Learning^[3]. On managed compute you skip this step; off it, the run is not recorded until you set it.

Compare runs to pick a model

Because every job logs to the workspace, comparing models is comparing runs. Select several runs in an experiment and view their logged metrics side by side^[4] in the studio, or query them with the MLflow API^[5]. Compare on the agreed validation metric, not on training loss: a model can post the lowest training loss and still generalize worse. This shared metric history is what later lets a sweep or AutoML job hand you a single best trial instead of a pile of runs to sort by hand.

Two kinds of work need two kinds of compute. Interactive exploration in a notebook wants a machine that stays up while you think; a submitted training job wants machines that appear for the run and disappear after. Azure Machine Learning splits these across a compute instance and a compute cluster. This page covers the two you train on; the full catalog of compute targets, including serverless and inference endpoints, lives in Machine Learning Workspace Resources.

Compute instance: your notebook workstation

A compute instance is a single-node managed cloud workstation, personal to one user, with Jupyter, JupyterLab, and VS Code^[6] preinstalled. It is where you author code and run notebooks interactively. Because it is one dedicated VM that keeps billing compute hours the whole time it runs, and it does not scale to zero the way a cluster does, its cost control is different: you enable idle shutdown and attach start/stop schedules.

• Idle shutdown: set idle_time_before_shutdown_minutes and the instance stops itself after that much inactivity. The window must be at least 15 minutes and at most 3 days (configure idle shutdown^[7]). It counts as inactive only when there are no running kernels, terminals, jobs, VS Code connections, or custom apps, so an idle-looking browser tab with a live kernel still bills.
• Schedules: attach up to four start/stop schedules per instance to bring it up before work hours and shut it down after (schedule automatic start and stop^[7]).

Compute cluster: where jobs run

A compute cluster, whose resource type is AmlCompute, is multi-node and job-oriented. It autoscales dedicated nodes for each submitted job^[8] and can scale back to zero nodes when idle, so between runs it can cost nothing while each job still gets its own isolated nodes. Training jobs, sweeps, and distributed jobs run here.

The distinction the exam tests sits right at the reconciliation of those two facts: a compute instance bills while running and does not scale to zero, whereas a compute cluster runs submitted jobs and does. Trying to open notebooks directly on a cluster is the mismatch, because clusters run jobs, not the interactive notebook environment. Picking a single-node compute instance for a distributed multi-node training job is the opposite mismatch.

The workhorse of Azure ML training is the command job: it runs one training script on a compute target and records the result as a tracked run whose outputs and MLflow-logged metrics land in the workspace (the same tracking described in Track and compare runs with MLflow above). You describe four things and submit, and Azure ML does the rest. A command job binds together:

• code — the folder holding your training script.
• command — what to run, for example python train.py --epochs 20.
• environment — the dependencies, as a curated or custom Docker/conda image. An environment is the software stack, not the hardware.
• compute — the compute target (the hardware) the job runs on.

The last two are the pair people conflate. The environment supplies libraries; the compute supplies CPUs or GPUs. A job needs both, and you set them independently (train models with Azure Machine Learning^[9]).

Submit with the CLI

A command job defined in YAML (job.yml):

$schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
code: ./src
command: python train.py --data ${{inputs.training_data}}
environment: azureml:AzureML-sklearn-1.5@latest   # curated environment
compute: azureml:cpu-cluster
inputs:
  training_data:
    type: uri_folder
    path: azureml://datastores/workspaceblobstore/paths/data/
# ...

Submit it:

az ml job create -f job.yml

Bind data with typed inputs and outputs

Notice the ${{inputs.training_data}} token in the command above. A command job's inputs mount or download data assets and URIs into the script, and its outputs write results back to workspace-managed storage. You reference them in the command with ${{inputs.<name>}} and ${{outputs.<name>}} placeholders instead of hardcoding a blob or datastore path (access data in a job^[10]). Hardcoding a storage path in the script breaks portability and severs the lineage Azure ML would otherwise record between the data asset and the job. Data assets and environments are workspace assets in their own right; see Machine Learning Workspace Assets.

When a model is too large or too slow to train on one machine, a command job can spread it across several. The lever is a distribution config added to the command job, and one rule governs it: the distribution type must match the framework your script uses (distributed GPU training guide^[11]).

Match the distribution type to the framework

• pytorch — for scripts using torch.distributed and DistributedDataParallel.
• tensorflow — for tf.distribute scripts; add worker_count, plus parameter_server_count for the parameter-server strategy.
• mpi — for MPI or Horovod^[11] launchers.

DeepSpeed is not a fourth type. You launch it through either the pytorch or the mpi distribution (DeepSpeed on Azure ML^[11]). Selecting mpi for a plain PyTorch DistributedDataParallel script, which needs the pytorch type, is the classic mismatch.

Size the job: nodes and processes

Two numbers size a distributed job:

• instance_count — the number of nodes (VMs).
• process_count_per_instance — the processes per node, typically set to the number of GPUs on each node^[11].

So a two-node job with four GPUs per node sets instance_count to 2 and process_count_per_instance to 4, for eight worker processes in total, and it requires a compute cluster whose VM SKU actually provides those GPUs.

A PyTorch distribution on a command job:

command: python train.py
environment: azureml:AzureML-acpt-pytorch@latest
compute: azureml:gpu-cluster
resources:
  instance_count: 2              # 2 nodes
distribution:
  type: pytorch
  process_count_per_instance: 4  # 4 GPUs per node -> 8 processes total
# ...

The trap to avoid: setting instance_count above one with no distribution block at all. The extra nodes then run the script independently, each training its own copy, instead of coordinating a single distributed run.

Your training script already works, but you want the best learning rate and batch size. Wrap it in a sweep job: it runs your command job many times over a range of hyperparameter values and reports the best. Three choices define a sweep, namely how it samples, what it optimizes, and when it gives up on weak trials.

Sample the search space

A search space is the set of values each hyperparameter can take. A sweep explores it with one of three sampling algorithms^[12]:

• Grid — every combination; only sensible for small, discrete spaces.
• Random — values drawn from the space; a strong default for large or continuous spaces.
• Bayesian — each new trial is chosen from the results of prior trials, converging toward good regions.

Reaching for grid on a large or continuous space is the wrong reflex; random or bayesian is far more efficient there.

Optimize a primary metric

You set a primary_metric (the exact metric name your script logs) and a goal of Maximize or Minimize (specify the objective of the sweep^[12]). Every trial is scored on that one metric, and the sweep ranks its trials by it.

Terminate weak trials early, except under bayesian

To cut cost, an early-termination policy cancels trials that are clearly losing before they finish (early termination policies^[12]):

• Bandit — ends a trial whose metric falls outside a slack_factor (a ratio) or slack_amount (an absolute value) of the best trial so far.
• Median stopping — stops a trial doing worse than the median of running averages across trials.
• Truncation selection — cancels a fixed percentage of the lowest performers at each interval.

evaluation_interval and delay_evaluation control how often the policy runs and how long to wait before it starts acting. Two criteria are easy to confuse: bandit measures distance from the best run (slack), while truncation selection cuts a fixed percentage of the worst.

Here is the exception the exam loves: bayesian sampling does not support early termination. Because each bayesian trial is chosen from earlier results, there is no partial-trial signal to prune on, so the policy is set to None. Random and grid sampling do support early termination.

When you would rather not choose the algorithm at all, an automated ML (AutoML) job does it for you. You supply labeled data, name the task type and a metric to optimize, and AutoML tries many algorithm-and-hyperparameter combinations in parallel, returning the best (what is automated ML?^[13]).

Five task types

AutoML covers five task types: classification, regression, time-series forecasting, computer vision, and natural language processing (NLP) (AutoML task types^[13]). Assuming it only does classification is a common underestimate. You pick the task type and a primary_metric, for example accuracy, AUC_weighted, or normalized_root_mean_squared_error, and AutoML optimizes that metric when ranking models.

Featurization runs by default

Featurization is the preparation that turns raw columns into model-ready features: scaling and normalization, missing-value handling, and text-to-numeric encoding. In an AutoML job this runs automatically by default^[13], and those steps become part of the produced model, so the same transformations are reapplied automatically at inference. You can set featurization to custom to override the defaults, but you do not have to engineer every feature by hand first.

Exit criteria and ensembles

An AutoML job does not run forever; it stops on the exit criteria you configure^[14], such as an experiment timeout, a maximum number of trials, or a metric score threshold. Toward the end, AutoML builds voting and stacking ensemble models^[13], which are enabled by default, and these combined models often appear as the final, best-scoring iterations. Because AutoML already ranks trials by your primary_metric, choosing a production candidate means taking that best trial rather than re-ranking by a different metric, which could pick a worse model.

A single command job trains one script, but real MLOps workflows have several stages, such as prepare data, train, and evaluate, that should run as one repeatable, versioned unit. That unit is a pipeline job, built from components (what are Azure Machine Learning pipelines?^[15]).

Components are the reusable building block

A component is a self-contained pipeline step with typed inputs and outputs, its own environment, and its own code (Azure ML component^[16]). Unlike a one-off command job, a component is a registered, versioned asset that can be shared and reused across pipelines, and each step can run on its own compute target. The shift the exam checks: the reusable, shareable unit is the component, not a standalone command job.

A pipeline chains components into steps

A pipeline job wires components into an ordered workflow and lets Azure ML manage the dependencies between steps (build a pipeline with the Python SDK^[17]). Different people can own different steps, a data engineer the prep component and a scientist the training component, then integrate them into one graph.

Reuse skips unchanged work

Pipelines improve efficiency partly by reusing outputs from unchanged steps^[15]. When a component's code and inputs have not changed since a prior run, Azure ML can reuse that step's cached output and skip recomputation, so re-running a pipeline after editing only the training step need not repeat data preparation. Components and environments are workspace assets; the deeper treatment lives in Machine Learning Workspace Assets.

Orchestrate-training questions are mostly "pick the right mechanism for this described requirement." Map the trigger phrase to the answer:

• Track metrics without running your own tracking server points to the workspace as the MLflow backend; on managed compute the tracking URI is already set, so just log, or call mlflow.autolog().
• Record runs from a script on my laptop points to mlflow.set_tracking_uri with the workspace azureml-mlflow URI, because off-compute code does not inherit it.
• Interactive notebook development is a compute instance (single node, idle shutdown). A compute cluster is wrong: it runs jobs, not the notebook environment.
• Stop paying for an idle notebook workstation is idle shutdown (15 minutes to 3 days) or a start/stop schedule; a compute instance does not scale to zero like a cluster.
• Tune hyperparameters efficiently over a large or continuous space is a sweep job with random or bayesian sampling, not grid.
• Cut sweep cost by stopping weak trials is an early-termination policy (bandit, median stopping, or truncation selection). If the sampling is bayesian, early termination is not supported, so the answer is random or grid, or no policy.
• Get a good model with minimal code is an AutoML job: set the task type and primary_metric; featurization is automatic.
• Train a deep-learning model across multiple GPUs or nodes is a command job with a distribution config matching the framework (pytorch, tensorflow, or mpi), instance_count nodes, and process_count_per_instance GPUs per node.
• Reusable, versioned multi-step workflow is a pipeline of components; the component is the reusable unit, and unchanged steps reuse cached output.
• Choose the production model from many runs means comparing on the primary_metric in MLflow; a sweep or AutoML job already surfaces the best trial. Never select on training loss.

The meta-pattern is to move up the automation ladder only as far as the requirement asks: write a command job when you own the script, wrap it in a sweep to tune it, reach for AutoML when you want the model chosen for you, and compose a pipeline when the workflow has reusable stages.