Kubeflow Pipeline for Production systems


A typical ML process for training a model is as follows 

  • Start with data prepossessing  
  • Train the model  
  • Evaluate the model  
  • Iterate the above steps  


These steps are often executed manually by running scripts, notebooks, and applying human judgment to validate the model as successfully trained. After model training for production, the process of training an ML model is more likely to be automated with an ML model training pipeline. This way, the model training can be reviewed, reproduced, tuned, and debugged at any point. When triggered, the components in the above pipeline will run in sequential order, and only if they are all successful, a micro-service that serves the trained model will be created or updated.  

Most of the components in the above pipeline can be reused and have open-source implementations. There are all types of data pre-processing, model training, and validation components hosted on the AI Hub. You can go very far into production ML training and serving without the need for writing custom components.  

Let’s go through the basics before using any components. All of the components are applications in Docker containers. For example, to train an ML model, we would need the Docker registry URI that points to the container and pass arguments as training-data, learning-rate, and output-location. When we run that trainer image, the application will read the runtime arguments and use them to produce a trained model. There are many solutions for running docker containers on the Cloud. AI Platform training is a specialized service for running ML training containers. 

AI Platform training works better than other solutions because of the easy distributed training setup and the provision of TPUs.To serve inference from your model, you need to create a serving REST endpoint. That is easy if you let AI Platform serving host your model. AI Platform serving creates a REST API that returns predictions and scales automatically according to demand. This eliminates the need for setup and maintenance of complex infrastructure. AI Platform also simplifies the process of updating prediction models by providing a model name and version. Note that the process of training and deployment will be the same for all TensorFlow components as well. The orchestration is the final stage of any pipeline development, and you should be able to execute every component separately before you put them in a pipeline.  

kubeflow process

So how do we chain our Docker ML containers?  

Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers. To create a pipeline, we need to describe the DAG of components with the Kubeflow Pipelines Python SDK. KFP pipelines are composed of components, and components are Docker containers with the addition to some metadata as inputs, outputs, and Docker registry. That metadata is normally stored in a YAML file.  

ML Container + component.yaml = Component  

ML Containers are released with such a metadata file, and they are published under the “Kubeflow pipeline” category. Some ML containers don’t have a component.yaml file. They are just Docker containers that parse command line arguments.

Contact Our Kubeflow Experts

Get Free Consultation
Share on facebook
Share on twitter
Share on linkedin
Share on pinterest
Share on whatsapp
Share on email

Leave a Reply

Your email address will not be published. Required fields are marked *

Recent Post

Hadoop Vs. Hbase

Hadoop is an open-source framework of programs that is used to store and process big data. Hadoop uses multiple clusters of computers to analyze big data sets in parallel. The distributed processing of data sets can

Read More »
no sql databases

No SQL Databases : Types

No SQL databases are non-relational databases. It is an approach to database design which allows storage and retrieval of data in a non-tabular format as that found in relational database. NoSQL

Read More »

Contact us for a quote, help, or to join the team.




(732) 347-6245

About Us

iSmile Technologies is a global technology services company.

(732) 347-6245


+1 (732) 347-6245
241 Jonathan Way
Bolingbrook, IL 60490


2-3-285, Secunderabad Hyderabad 500003


3190 Stocksbridge Ave
Oakville, ON L6M 0A7