Open Source CNCF Kubeflow

Kubeflow Pipelines

Build and deploy portable, scalable machine learning workflows using containers on Kubernetes.

Author pipelines in Python with the KFP SDK, then run them on any KFP-conformant backend.

What is Kubeflow Pipelines?

Kubeflow Pipelines (KFP) is a platform for building and deploying portable, scalable machine learning workflows using containers on Kubernetes. Author pipelines in Python with the KFP SDK, compile them to a platform-neutral IR YAML, and run them unchanged on any KFP-conformant backend, such as the open source KFP backend or Google Cloud Vertex AI Pipelines.

Kubeflow Pipelines is the workflow orchestration component of Kubeflow, the open-source ML platform for Kubernetes.

Why Kubeflow Pipelines?

Pythonic Authoring

Turn a Python function into a pipeline step with a single decorator. Author components and pipelines with the kfp SDK — no YAML by hand.

Portable & Reproducible

Compile pipelines to a platform-neutral IR YAML that runs unchanged across environments, so results are consistent and easy to reproduce.

Reusable Components

Build once, share everywhere. Compose pipelines from your own reusable components and an ecosystem of existing ones.

Track Everything

Automatically track parameters, artifacts, runs, and experiments with built-in ML metadata and end-to-end lineage.

Efficient Execution

Run tasks in parallel and skip redundant work with caching, so pipelines finish faster and use fewer resources.

Pass Data Between Steps

Easily pass parameters and ML artifacts between components, so each step consumes the outputs of the steps before it.

How It Works

You author a pipeline in Python and compile it to a platform-neutral IR YAML, then submit it with the SDK, CLI, or UI. The KFP API Server records the run and Argo orchestrates it, running each step as a container while artifacts, metadata, and status stream back to the Pipeline UI.

How a Kubeflow Pipeline goes from Python to running pods. 1. Author: write a pipeline in Python with the KFP SDK. 2. Compile: the SDK compiles it to a platform-neutral IR YAML. 3. Submit: send it to the KFP API Server via the SDK, CLI, or UI. 4. Orchestrate: the Argo Workflow Controller runs the pipeline. 5. Track: a driver pod coordinates the run and executor pods run each step as a container, reporting status, artifacts, and metadata to ML Metadata and the object store, all viewable in the Pipeline UI.

Documentation

Everything you need, from your first pipeline to running in production and contributing back.

Overview

Learn what Kubeflow Pipelines is, who it is for, and why you would use it

Overview
Getting Started

Install KFP and run your first pipeline in minutes

Getting started
Interfaces

Author, run, and manage pipelines with the SDK, CLI, and web UI

Interfaces
Concepts

Understand pipelines, components, graphs, runs, experiments, and artifacts

Concepts
Operator Guides

Install, configure, and operate Kubeflow Pipelines on a cluster

Operator Guides
User Guides

Author components and pipelines, pass data between tasks, and control flow

User Guides
Python SDK

Install the kfp SDK, then look up the API and CLI reference

Python SDK
Reference

Look up API specifications, version compatibility, and component definitions

Reference
Contributor Guide

Set up your development environment and contribute to Kubeflow Pipelines

Contributor Guide

Build a Pipeline in Python

PYTHON
from kfp import dsl

@dsl.component
def say_hello(name: str) -> str:
    hello_text = f'Hello, {name}!'
    print(hello_text)
    return hello_text

@dsl.pipeline
def hello_pipeline(recipient: str) -> str:
    hello_task = say_hello(name=recipient)
    return hello_task.output

Compile it to IR YAML, then run it locally or submit to any KFP-conformant backend. See the full quickstart →