Just as our other pipelines so far, our Azure Pipeline will be set up as a yaml file.
In our git repository, this can be found in the root of the repository as
Let’s take a look at this yaml file and then we’ll explain what’s happening at each step, it’ll be following the same template as our last two pipelines:
- cron: "0 12 * * 0"
displayName: "Weekly Sunday training pipeline run"
- job: 'training_pipeline_job'
- group: KeyVault
- group: ProductionEnvVars
- task: UsePythonVersion@0
- script: |
python -m pip install --upgrade pip
pip install -r requirements.txt
displayName: 'Install requirements'
- script: |
displayName: 'Run AML Experiment and Register Model'
So if we take a dive into at what’s happening here:
First, the CI trigger is turned off, so that this Pipeline isn’t run every time the code is updated.
There is a cron schedule that runs at 00:00 every Sunday on the master branch. Just as with the data pipeline this pipeline is always run, regardless of whether there are code changes, as the data may have changed.
As with our previous pipelines, we have set up a pipeline with a single stage, with a single job on an Ubuntu 16.04 base image.
Variables are extracted from the variable groups we set up in our resource set up in Azure DevOps.
After setting the python version, there are only two steps here.
First we install the python package requirements on our pipeline compute.
After we’ve installed the requirements, we run
src/my_custom_package/create_aml_experiment.py, this will submit an Azure ML experiment run. Upon submitting the run, a training script is run on a training compute node and this will train the model and register the resulting model on Azure Machine Learning.
Run Azure ML Experiment
main function of our
create_aml_experiment.py python script, we run
This will submit an experiment run, the code for this is below.
experiment = Experiment(aml_interface.workspace, AML_EXPERIMENT_NAME)
src_dir = __here__
run_config = ScriptRunConfig(
run_config.run_config.target = aml_interface.get_compute_target(
aml_run_env = Environment.get(
run_config.run_config.environment = aml_run_env
run = experiment.submit(config=run_config)
First we instantiate an Azure ML
Experiment object, providing our Azure ML
Workspace object and experiment name.
We then define a run configuration. We will be running a script on our AML training compute node. The script we’ll be running is
train.py, which we will examine below, and the source directory for which this script is held is the same directory as
As part of this run configuration, we also define a compute node, we’ve chosen a standard D2_V2 machine here. The full range of virtual machines available on can be viewed here, including GPU-enabled machines.
We then retrieve the environment we created in our Environment Pipeline, with our custom package installed by providing the
Workspace object and the environment name.
Now we’re ready to submit the run, so we submit the experiment run and wait for it to complete. The console output will be displayed as the run waits for completion. Any metrics from the run are then printed once the run is complete.