Creating End-to-End MLOps pipelines using Azure ML and Azure Pipelines
In this 7-part series of posts we’ll be creating a minimal, repeatable MLOps Pipeline using Azure ML and Azure Pipelines.
The git repository that accompanies these posts can be found here.
In this series we’ll be covering:
Quick disclaimer: At the time of writing, I am currently a Microsoft Employee
Part 1 – Introduction
In this post we will cover:
- Who is this tutorial for?
- What is MLOps?
- Why Azure Pipelines?
- Why Azure Machine Learning?
This series of posts will assume the reader has some knowledge & experience of data science workflows.
Who is this tutorial for?
This tutorial is for data science practitioners/enthusiasts or data engineers, that would like to set up a minimal, robust, repeatable set of pipelines for productionising Machine Learning models in the cloud using Microsoft Azure.
There are lots of other resources that can help you learn Azure Machine Learning, from the basics through to the more complex nuances, not least the official Microsoft documentation. It is a very powerful tool and one I would recommend to any data scientist. However it can sometimes be a little overwhelming to know how to put everything together in your own production environment.
In this series of posts, we’ll help you set up pipelines from scratch that will allow us to extract and transform data, set up an environment, train and deploy machine learning models.
What is MLOps?
MLOps is the enablement of the automated management of the end-to-end machine learning lifecycle.
It helps data engineers, developers and data scientists work collaboratively to productionise machine learning models. It’s no use having an ML model that performs super well and could save a huge amount of money if it never goes into production and remains in pickle files or jupyter notebooks.
Also, once it’s in production, it won’t change with the times. As time progresses, the underlying data used to train the model becomes stale and the performance of the model drifts. We need to have a way of updating our production services with models trained on up-to-date data.
In this 7-part series of posts we’ll set up pipelines to create a minimal end-to-end MLOps pipelines to achieve the following using Azure Machine Learning and Azure Pipelines:
Across this series of posts, we will create 5 Azure Pipelines in this example project.
Each of these pipelines are on an automated trigger – triggered either by a change to code or on a schedule. However, if we make changes and want to trigger the pipelines manually, we can do so through Azure Pipelines.
The pipelines are as follows: