Machine Learning Pipelines (Pt. 1)
Machine Learning (ML) is centered around constructing models capable of automating diverse tasks. Such tasks can vary from detecting fraudulent transactions and identifying parking lots in satellite imagery, to translating text between languages or offering pricing suggestions for cargo transportation.
Although certain tasks may be performed better by humans, machines excel at executing them rapidly, continuously and without getting bored.
As a programmer, you can think of a ML model as a function that picks up arguments and returns with an answer, similar to traditional programming. The distinction lies in the fact that instead of being coded by a human it is rather a black box derived from extensive data. The model includes a significant amount of data encapsulated by code to interpret that information into a function with multiple variables.
This analogy draws a parallel between machine learning pipelines and continuous integration / delivery pipelines in software development. Both types of pipelines compile source code into executable artifacts.
There are exceptions to this though, such as:
● In software we mostly work with codebases, whereas in machine learning models and big amounts of data are being managed.
● Software can be tested well - the build either passes or fails. Machine learning models on the other hand are always inaccurate to some degree.
● Data can be wrong and can become outdated, as can models.
Let me give you an example of how models can become outdated. In 2020, all the models that suggested prices for delivering cargo between two locations started giving wrong answers. Why did this happen? It was because of Brexit. Businesses in the UK started stocking up on supplies before the borders were shut, which created a higher demand, leading to higher prices. Since the models were trained on past data, they didn't know how to handle this new situation.
As software engineers, we care about making sure our code works perfectly in every possible situation. On the other hand, data scientists must be okay with not knowing everything and dealing with differences in data. ML is about finding patterns in data, which can be challenging because it's not always clear what the data means. This makes machine learning more flexible for complex problems, but also harder to understand and fix if something goes wrong.
Most problems that software engineers solve today will continue to be solved using traditional programming in the future. However, ML can be used to solve new types of problems that couldn't be solved before and can be a great addition to software development. As long as sufficient amounts of data are available and it has been tagged with a desired outcome.
What are machine learning pipelines?
Machine learning is a way of teaching computers to do things that are normally done by humans. It's like a set of instructions that the computer can use to learn and make decisions based on data. Machine learning pipelines are a series of steps that turn data into a trained and tested machine learning model. The pipeline involves getting data, changing the data to be good for training, creating a model, and packaging it so it can be used for a long time. The model can then be made available to others through an API.
A machine learning pipeline is like a big machine that takes in data and gives out predictions. It's made up of different parts, like the data that comes in, the way the data is changed so the machine can understand it, the machine learning model that does the thinking, and the output that the machine gives. All these parts work together to make sure the machine can make good predictions.
This is how machine learning pipelines could look like in a simple form:
Download data from some source. Usually, it will be a set of datastore rows or records.
Convert data to a format suitable for training: select features (arguments for the model), remove noise and bad records. Some fields in the dataset will be used as input and others will be specified as the desired output that we want to predict.
Define the model format (smell the wind and say “this big equation with a lot of variables will get the job done”) and train the model on data (tweak formula variables in semi-random way until the model starts accurately guessing results given inputs).
Package the model into a durable format (e.g., a Docker container with some binary blob).
Optionally, deploy the model as a service with an API.
Normally these transformations are codified as workflows (workflow as code), versioned and deployed. In simpler projects, one can implement them with Bash or Python scripts. Larger projects and teams are well advised to use something that is better documented and based on conventions (for example via domain-specific language, or declaratives syntax).
Long story short: Machine learning pipelines are codified workflows that ingest data, transform, and derive reusable models from it. Their goal is similar to CI/CD pipelines in software engineering: automate, ensure repeatability and scale processes. Implementation details differ from CI/CD because machine learning pipelines work mostly with data.
Why are machine learning pipelines important?
In a regular system design, all the tasks would be performed together in a single program. This means that the same code would be used to collect, clean, model, and deploy the data. Because machine learning models generally have less code than other software programs, it makes sense to keep everything in one place.
In the ML pipeline, every step of your work process is made into its own separate service. This means that when you want to create a new workflow, you can select the specific parts you need and use them wherever you want. Any updates or changes made to a service will be done at a higher level, making it easier and more efficient to manage.
Machine learning pipelining can solve several problems. It allows for more efficient scaling of ML workflows. Rather than having to repeat the entire process for each new model, pipelining enables the reuse of the same data preparation and processing steps. Also, by allowing you to update individual components without affecting the rest of the pipeline, ML pipelining can help with version control.
Considering workflow efficiency: Breaking down a machine learning workflow into smaller, reusable components can save a lot of time. And last but not least, with machine learning pipelining, teams can collaborate on individual parts of a workflow without worrying about how their changes will affect the entire process.