kfp.dsl

The kfp.dsl module contains domain-specific language objects used to compose pipelines.

Data:

Input

Type generic used to represent an input artifact of type T, where T is an artifact class.

PIPELINE_JOB_NAME_PLACEHOLDER

A placeholder used to obtain a pipeline job name within a task at pipeline runtime.

PIPELINE_JOB_RESOURCE_NAME_PLACEHOLDER

A placeholder used to obtain a pipeline job resource name within a task at pipeline runtime.

PIPELINE_JOB_ID_PLACEHOLDER

A placeholder used to obtain a pipeline job ID within a task at pipeline runtime.

PIPELINE_TASK_NAME_PLACEHOLDER

A placeholder used to obtain a task name within a task at pipeline runtime.

PIPELINE_TASK_ID_PLACEHOLDER

A placeholder used to obtain a task ID within a task at pipeline runtime.

PIPELINE_TASK_EXECUTOR_OUTPUT_PATH_PLACEHOLDER

A placeholder used to obtain the path to the executor_output.json file within the task container.

PIPELINE_TASK_EXECUTOR_INPUT_PLACEHOLDER

A placeholder used to obtain executor input message passed to the task.

PIPELINE_ROOT_PLACEHOLDER

A placeholder used to obtain the pipeline root.

PIPELINE_JOB_CREATE_TIME_UTC_PLACEHOLDER

A placeholder used to obtain the time that a pipeline job was created.

PIPELINE_JOB_SCHEDULE_TIME_UTC_PLACEHOLDER

A placeholder used to obtain the time for which a pipeline job is scheduled.

WORKSPACE_PATH_PLACEHOLDER

A placeholder used to obtain the path to the shared workspace within a component.

Classes:

InputPath([type])

Type annotation used in component definitions for indicating a parameter is a path to an input.

OutputPath([type])

Type annotation used in component definitions for indicating a parameter is a path to an output.

PipelineTaskFinalStatus(state, ...)

A final status of a pipeline task.

TaskConfig([affinity, tolerations, ...])

Configurations for a task.

Artifact([name, uri, metadata])

Represents a generic machine learning artifact.

ClassificationMetrics([name, uri, metadata])

An artifact for storing classification metrics.

Dataset([name, uri, metadata])

An artifact representing a machine learning dataset.

HTML([name, uri, metadata])

An artifact representing an HTML file.

Markdown([name, uri, metadata])

An artifact representing a markdown file.

Metrics([name, uri, metadata])

An artifact for storing key-value scalar metrics.

Model([name, uri, metadata])

An artifact representing a machine learning model.

SlicedClassificationMetrics([name, uri, ...])

An artifact for storing sliced classification metrics.

ContainerSpec(image[, command, args])

Container definition.

Condition(condition[, name])

Deprecated.

If(condition[, name])

A class for creating a conditional control flow "if" block within a pipeline.

Elif(condition[, name])

A class for creating a conditional control flow "else if" block within a pipeline.

Else([name])

A class for creating a conditional control flow "else" block within a pipeline.

OneOf(*channels)

For collecting mutually exclusive outputs from conditional branches into a single pipeline channel.

ExitHandler(exit_task[, name])

A class for setting an exit handler task that is invoked upon exiting a group of other tasks.

ParallelFor(items[, name, parallelism])

A class for creating parallelized for loop control flow over a static set of items within a pipeline definition.

Collected(output)

For collecting into a list the output from a task in dsl.ParallelFor loops.

IfPresentPlaceholder(input_name, then[, else_])

Placeholder for handling cases where an input may or may not be passed.

ConcatPlaceholder(items)

Placeholder for concatenating multiple strings.

PipelineTask(component_spec, args[, ...])

Represents a pipeline task (instantiated component).

PipelineConfig([workspace, ...])

PipelineConfig contains pipeline-level config options.

WorkspaceConfig(size[, kubernetes])

Configuration for a shared workspace that persists during the pipeline run.

KubernetesWorkspaceConfig([pvcSpecPatch])

Configuration for Kubernetes-specific workspace settings.

TaskConfigField(*values)

TaskConfigPassthrough(field[, apply_to_task])

Functions:

get_uri([suffix])

Gets the task root URI, a unique object storage URI associated with the current task.

run_notebook(**kwargs)

Execute the component's embedded Jupyter notebook with injected parameters.

component([func, base_image, target_image, ...])

Decorator for Python-function based components.

container_component(func)

Decorator for container-based components in KFP v2.

pipeline([func, name, description, ...])

Decorator used to construct a pipeline.

importer(artifact_uri, artifact_class[, ...])

Imports an existing artifact for use in a downstream component.

notebook_component([func, base_image, ...])

Decorator to define a Notebook-based KFP component.

kfp.dsl.Input(*args, **kwargs)

Type generic used to represent an input artifact of type T, where T is an artifact class.

Use Input[Artifact] or Output[Artifact] to indicate whether the enclosed artifact is a component input or output.

Parameters:

T – The type of the input artifact.

Example

@dsl.component
def artifact_producer(model: Output[Artifact]):
    with open(model.path, 'w') as f:
        f.write('my model')

@dsl.component
def artifact_consumer(model: Input[Artifact]):
    print(model)

@dsl.pipeline
def my_pipeline():
    producer_task = artifact_producer()
    artifact_consumer(model=producer_task.output)

alias of Annotated[T, InputAnnotation]

class kfp.dsl.InputPath(type=None)[source]

Bases: object

Type annotation used in component definitions for indicating a parameter is a path to an input.

Example

@dsl.component
def create_dataset(dataset_path: OutputPath('Dataset'),):
    import json
    dataset = {'my_dataset': [[1, 2, 3], [4, 5, 6]]}
    with open(dataset_path, 'w') as f:
        json.dump(dataset, f)


@dsl.component
def consume_dataset(dataset: InputPath('Dataset')):
    print(dataset)


@dsl.pipeline(name='my-pipeline', pipeline_root='gs://my-bucket')
def my_pipeline():
    create_dataset_op = create_dataset()
    consume_dataset(dataset=create_dataset_op.outputs['dataset_path'])
class kfp.dsl.OutputPath(type=None)[source]

Bases: object

Type annotation used in component definitions for indicating a parameter is a path to an output. The path parameter typed with this annotation can be treated as a locally accessible filepath within the component body.

The argument typed with this annotation is provided at runtime by the executing backend and does not need to be passed as an input by the pipeline author (see example).

Parameters:

type – The type of the value written to the output path.

Example

@dsl.component
def create_parameter(
        message: str,
        output_parameter_path: OutputPath(str),
):
    with open(output_parameter_path, 'w') as f:
        f.write(message)


@dsl.component
def consume_parameter(message: str):
    print(message)


@dsl.pipeline(name='my-pipeline', pipeline_root='gs://my-bucket')
def my_pipeline(message: str = 'default message'):
    create_param_op = create_parameter(message=message)
    consume_parameter(message=create_param_op.outputs['output_parameter_path'])
class kfp.dsl.PipelineTaskFinalStatus(state: str, pipeline_job_resource_name: str, pipeline_task_name: str, error_code: int | None, error_message: str | None)[source]

Bases: object

A final status of a pipeline task. Annotate a component parameter with this class to obtain a handle to a task’s status (see example).

This is the Python representation of the proto message PipelineTaskFinalStatus.

Examples

@dsl.component
def task_status(user_input: str, status: PipelineTaskFinalStatus):
    print('Pipeline status: ', status.state)
    print('Job resource name: ', status.pipeline_job_resource_name)
    print('Pipeline task name: ', status.pipeline_task_name)
    print('Error code: ', status.error_code)
    print('Error message: ', status.error_message)

@dsl.pipeline(name='my_pipeline')
def my_pipeline():
    task = task_status(user_input='my_input')

Attributes:

state

Final state of the task.

pipeline_job_resource_name

Pipeline job resource name, in the format of projects/{project}/locations/{location}/pipelineJobs/{pipeline_job}.

pipeline_task_name

Name of the task that produced this status.

error_code

The google.rpc.Code in case of error.

error_message

In case of error, the detailed error message.

state: str

Final state of the task. The value could be one of 'SUCCEEDED', 'FAILED' or 'CANCELLED'.

pipeline_job_resource_name: str

Pipeline job resource name, in the format of projects/{project}/locations/{location}/pipelineJobs/{pipeline_job}.

pipeline_task_name: str

Name of the task that produced this status.

error_code: int | None

The google.rpc.Code in case of error. If state is 'SUCCEEDED', this is None.

error_message: str | None

In case of error, the detailed error message. If state is 'SUCCEEDED', this is None.

class kfp.dsl.TaskConfig(affinity: Dict[str, Any] | None = None, tolerations: List[Dict[str, Any]] | None = None, node_selector: Dict[str, str] | None = None, env: List[Dict[str, Any]] | None = None, volumes: List[Dict[str, Any]] | None = None, volume_mounts: List[Dict[str, Any]] | None = None, resources: Dict[str, Any] | None = None, resource_claims: List[Dict[str, Any]] | None = None)[source]

Bases: object

Configurations for a task.

Annotate a component parameter with this type when you want the task’s runtime configuration to be forwarded to an external workload and optionally applied to the task’s own pod. This is useful when the task launches another Kubernetes resource (for example, a Kubeflow Trainer job).

This is an empty value by default and is populated if the component is annotated with task_config_passthroughs.

All fields are optional and map 1:1 to fragments of the Kubernetes Pod spec. These fields have values as Python dictionaries/lists that conform to the Kubernetes JSON schema.

Example

@dsl.component(
    packages_to_install=["kubernetes"],
    task_config_passthroughs=[
        dsl.TaskConfigField.RESOURCES,
        dsl.TaskConfigField.KUBERNETES_TOLERATIONS,
        dsl.TaskConfigField.KUBERNETES_NODE_SELECTOR,
        dsl.TaskConfigField.KUBERNETES_AFFINITY,
        dsl.TaskConfigPassthrough(field=dsl.TaskConfigField.ENV, apply_to_task=True),
        dsl.TaskConfigPassthrough(field=dsl.TaskConfigField.KUBERNETES_VOLUMES, apply_to_task=True),
        dsl.TaskConfigField.KUBERNETES_RESOURCE_CLAIMS,
    ],
)
def train(num_nodes: int, workspace_path: str, output_model: dsl.Output[dsl.Model], task_config: dsl.TaskConfig):
    import os
    import shutil
    from kubernetes import client as k8s_client, config
    config.load_incluster_config()

    with open(
            "/var/run/secrets/kubernetes.io/serviceaccount/namespace", "r"
        ) as ns_file:
            namespace = ns_file.readline()

    train_job_script = "with open('/kfp-workspace/model', 'w') as f: f.write('hello')"

    container_resources = dict(task_config.resources or {})
    if task_config.resource_claims:
        container_resources["claims"] = [
            {"name": claim["name"]}
            for claim in task_config.resource_claims
        ]

    dataset_path = os.path.join(workspace_path, "dataset")
    with open(dataset_path, "w") as f:
        f.write("Prepare dataset here...")

    train_job = {
            "apiVersion": "trainer.kubeflow.org/v1alpha1",
            "kind": "TrainJob",
            "metadata": {"name": f"kfp-train-job", "namespace": namespace},
            "spec": {
                "runtimeRef": {"name": "torch-distributed"},
                "trainer": {
                    "numNodes": num_nodes,
                    "resourcesPerNode": task_config.resources,
                    "env": task_config.env,
                    "command": ["python", "-c", train_job_script],
                },
                "podSpecOverrides": [
                    {
                        "targetJobs": [{"name": "node"}],
                        "volumes": task_config.volumes,
                        "containers": [
                            {
                                "name": "node",
                                "volumeMounts": task_config.volume_mounts,
                                "resources": container_resources,
                            }
                        ],
                        "nodeSelector": task_config.node_selector,
                        "tolerations": task_config.tolerations,
                        "resourceClaims": task_config.resource_claims,
                    }
                ],
            },
        }
    print(train_job)
    api_client = k8s_client.ApiClient()
    custom_objects_api = k8s_client.CustomObjectsApi(api_client)
    response = custom_objects_api.create_namespaced_custom_object(
        group="trainer.kubeflow.org",
        version="v1alpha1",
        namespace=namespace,
        plural="trainjobs",
        body=train_job,
    )
    job_name = response["metadata"]["name"]
    print(f"TrainJob {job_name} created successfully")

    print("Polling train job code goes here...")

    print("Copying output model")
    shutil.copy(os.path.join(workspace_path, "model"), output_model.path)

@dsl.pipeline
def example_task_config():
    train_task = train(num_nodes=1, workspace_path=dsl.WORKSPACE_PATH_PLACEHOLDER)
    train_task.set_cpu_request("1")
    train_task.set_memory_request("20Gi")
    train_task.set_cpu_limit("2")
    train_task.set_memory_limit("50Gi")
    train_task.set_accelerator_type("nvidia.com/gpu")
    train_task.set_accelerator_limit("1")

Attributes:

affinity: Dict[str, Any] | None = None
tolerations: List[Dict[str, Any]] | None = None
node_selector: Dict[str, str] | None = None
env: List[Dict[str, Any]] | None = None
volumes: List[Dict[str, Any]] | None = None
volume_mounts: List[Dict[str, Any]] | None = None
resources: Dict[str, Any] | None = None
resource_claims: List[Dict[str, Any]] | None = None
class kfp.dsl.Artifact(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]

Bases: object

Represents a generic machine learning artifact.

This class and all artifact classes store the name, uri, and metadata for a machine learning artifact. Use this artifact type when an artifact does not fit into another more specific artifact type (e.g., Model, Dataset).

Parameters:
  • name – Name of the artifact.

  • uri – The artifact’s location on disk or cloud storage.

  • metadata – Arbitrary key-value pairs about the artifact.

Example

from kfp import dsl
from kfp.dsl import Output, Artifact, Input


@dsl.component
def create_artifact(
    data: str,
    output_artifact: Output[Artifact],
):
    with open(output_artifact.path, 'w') as f:
        f.write(data)


@dsl.component
def use_artifact(input_artifact: Input[Artifact]):
    with open(input_artifact.path) as input_file:
        artifact_contents = input_file.read()
        print(artifact_contents)


@dsl.pipeline(name='my-pipeline', pipeline_root='gs://my/storage')
def my_pipeline():
    create_task = create_artifact(data='my data')
    use_artifact(input_artifact=create_task.outputs['output_artifact'])

Note: Other artifacts are used similarly to the usage of Artifact in the example above (within Input[] and Output[]).

Attributes:

Methods:

set_path(path)

schema_title = 'system.Artifact'
schema_version = '0.0.1'
property path: str
property custom_path: str
set_path(path: str) → None[source]
class kfp.dsl.ClassificationMetrics(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]

Bases: Artifact

An artifact for storing classification metrics.

Parameters:
  • name – Name of the metrics artifact.

  • uri – The metrics artifact’s location on disk or cloud storage.

  • metadata – The key-value scalar metrics.

Attributes:

Methods:

log_roc_data_point(fpr, tpr, threshold)

Logs a single data point in the ROC curve to metadata.

log_roc_curve(fpr, tpr, threshold)

Logs an ROC curve to metadata.

set_confusion_matrix_categories(categories)

Stores confusion matrix categories to metadata.

log_confusion_matrix_row(row_category, row)

Logs a confusion matrix row to metadata.

log_confusion_matrix_cell(row_category, ...)

Logs a cell in the confusion matrix to metadata.

log_confusion_matrix(categories, matrix)

Logs a confusion matrix to metadata.

schema_title = 'system.ClassificationMetrics'
log_roc_data_point(fpr: float, tpr: float, threshold: float) → None[source]

Logs a single data point in the ROC curve to metadata.

Parameters:
  • fpr – False positive rate value of the data point.

  • tpr – True positive rate value of the data point.

  • threshold – Threshold value for the data point.

log_roc_curve(fpr: List[float], tpr: List[float], threshold: List[float]) → None[source]

Logs an ROC curve to metadata.

Parameters:
  • fpr – List of false positive rate values.

  • tpr – List of true positive rate values.

  • threshold – List of threshold values.

Raises:

ValueError – If the lists fpr, tpr and threshold are not the same length.

set_confusion_matrix_categories(categories: List[str]) → None[source]

Stores confusion matrix categories to metadata.

Parameters:

categories – List of strings specifying the categories.

log_confusion_matrix_row(row_category: str, row: List[float]) → None[source]

Logs a confusion matrix row to metadata.

Parameters:
  • row_category – Category to which the row belongs.

  • row – List of integers specifying the values for the row.

Raises:

ValueError – If row_category is not in the list of categories set in set_categories call.

log_confusion_matrix_cell(row_category: str, col_category: str, value: int) → None[source]

Logs a cell in the confusion matrix to metadata.

Parameters:
  • row_category – String representing the name of the row category.

  • col_category – String representing the name of the column category.

  • value – Value of the cell.

Raises:

ValueError – If row_category or col_category is not in the list of categories set in set_categories.

log_confusion_matrix(categories: List[str], matrix: List[List[int]]) → None[source]

Logs a confusion matrix to metadata.

Parameters:
  • categories – List of the category names.

  • matrix – Complete confusion matrix.

Raises:

ValueError – If the length of categories does not match number of rows or columns of matrix.

class kfp.dsl.Dataset(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]

Bases: Artifact

An artifact representing a machine learning dataset.

Parameters:
  • name – Name of the dataset.

  • uri – The dataset’s location on disk or cloud storage.

  • metadata – Arbitrary key-value pairs about the dataset.

Attributes:

schema_title = 'system.Dataset'
class kfp.dsl.HTML(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]

Bases: Artifact

An artifact representing an HTML file.

Parameters:
  • name – Name of the HTML file.

  • uri – The HTML file’s location on disk or cloud storage.

  • metadata – Arbitrary key-value pairs about the HTML file.

Attributes:

schema_title = 'system.HTML'
class kfp.dsl.Markdown(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]

Bases: Artifact

An artifact representing a markdown file.

Parameters:
  • name – Name of the markdown file.

  • uri – The markdown file’s location on disk or cloud storage.

  • metadata – Arbitrary key-value pairs about the markdown file.

Attributes:

schema_title = 'system.Markdown'
class kfp.dsl.Metrics(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]

Bases: Artifact

An artifact for storing key-value scalar metrics.

Parameters:
  • name – Name of the metrics artifact.

  • uri – The metrics artifact’s location on disk or cloud storage.

  • metadata – Key-value scalar metrics.

Attributes:

Methods:

log_metric(metric, value)

Sets a custom scalar metric in the artifact's metadata.

schema_title = 'system.Metrics'
log_metric(metric: str, value: float) → None[source]

Sets a custom scalar metric in the artifact’s metadata.

Parameters:
  • metric – The metric key.

  • value – The metric value.

class kfp.dsl.Model(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]

Bases: Artifact

An artifact representing a machine learning model.

Parameters:
  • name – Name of the model.

  • uri – The model’s location on disk or cloud storage.

  • metadata – Arbitrary key-value pairs about the model.

Attributes:

schema_title = 'system.Model'
property path: str
property framework: str
class kfp.dsl.SlicedClassificationMetrics(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]

Bases: Artifact

An artifact for storing sliced classification metrics.

Similar to ClassificationMetrics, tasks using this class are expected to use log methods of the class to log metrics with the difference being each log method takes a slice to associate the ClassificationMetrics.

Parameters:
  • name – Name of the metrics artifact.

  • uri – The metrics artifact’s location on disk or cloud storage.

  • metadata – Arbitrary key-value pairs about the metrics artifact.

Attributes:

Methods:

log_roc_reading(slice, threshold, tpr, fpr)

Logs a single data point in the ROC curve of a slice to metadata.

load_roc_readings(slice, readings)

Bulk loads ROC curve readings for a slice.

set_confusion_matrix_categories(slice, ...)

Logs confusion matrix categories for a slice to metadata.

log_confusion_matrix_row(slice, ...)

Logs a confusion matrix row for a slice to metadata.

log_confusion_matrix_cell(slice, ...)

Logs a confusion matrix cell for a slice to metadata.

load_confusion_matrix(slice, categories, matrix)

Bulk loads the whole confusion matrix for a slice.

schema_title = 'system.SlicedClassificationMetrics'
log_roc_reading(slice: str, threshold: float, tpr: float, fpr: float) → None[source]

Logs a single data point in the ROC curve of a slice to metadata.

Parameters:
  • slice – String representing slice label.

  • threshold – Thresold value for the data point.

  • tpr – True positive rate value of the data point.

  • fpr – False positive rate value of the data point.

load_roc_readings(slice: str, readings: List[List[float]]) → None[source]

Bulk loads ROC curve readings for a slice.

Parameters:
  • slice – String representing slice label.

  • readings – A 2-dimensional list providing ROC curve data points. The expected order of the data points is: threshold, true positive rate, false positive rate.

set_confusion_matrix_categories(slice: str, categories: List[str]) → None[source]

Logs confusion matrix categories for a slice to metadata.

Categories are stored in the internal metrics_utils.ConfusionMatrix instance of the slice.

Parameters:
  • slice – String representing slice label.

  • categories – List of strings specifying the categories.

log_confusion_matrix_row(slice: str, row_category: str, row: List[int]) → None[source]

Logs a confusion matrix row for a slice to metadata.

Row is updated on the internal metrics_utils.ConfusionMatrix instance of the slice.

Parameters:
  • slice – String representing slice label.

  • row_category – Category to which the row belongs.

  • row – List of integers specifying the values for the row.

log_confusion_matrix_cell(slice: str, row_category: str, col_category: str, value: int) → None[source]

Logs a confusion matrix cell for a slice to metadata.

Cell is updated on the internal metrics_utils.ConfusionMatrix instance of the slice.

Parameters:
  • slice – String representing slice label.

  • row_category – String representing the name of the row category.

  • col_category – String representing the name of the column category.

  • value – Value of the cell.

load_confusion_matrix(slice: str, categories: List[str], matrix: List[List[int]]) → None[source]

Bulk loads the whole confusion matrix for a slice.

Parameters:
  • slice – String representing slice label.

  • categories – List of the category names.

  • matrix – Complete confusion matrix.

kfp.dsl.get_uri(suffix: str = 'Output') → str[source]

Gets the task root URI, a unique object storage URI associated with the current task. This function may only be called at task runtime.

Returns an empty string if the task root cannot be inferred from the runtime environment.

Parameters:

suffix – A suffix to append to the URI. This is a helpful for creating unique subdirectories when the component has multiple outputs.

Returns:

The URI or empty string.

kfp.dsl.PIPELINE_JOB_NAME_PLACEHOLDER = '{{$.pipeline_job_name}}'

A placeholder used to obtain a pipeline job name within a task at pipeline runtime. In Kubeflow Pipelines, this maps to the pipeline run display name.

Example

@dsl.pipeline
def my_pipeline():
    print_op(
        msg='Job name:',
        value=dsl.PIPELINE_JOB_NAME_PLACEHOLDER,
    )
kfp.dsl.PIPELINE_JOB_RESOURCE_NAME_PLACEHOLDER = '{{$.pipeline_job_resource_name}}'

A placeholder used to obtain a pipeline job resource name within a task at pipeline runtime. In Kubeflow Pipelines, this maps to the pipeline run name in the underlying pipeline engine (e.g. an Argo Workflow object name).

Example

@dsl.pipeline
def my_pipeline():
    print_op(
        msg='Job resource name:',
        value=dsl.PIPELINE_JOB_RESOURCE_NAME_PLACEHOLDER,
    )
kfp.dsl.PIPELINE_JOB_ID_PLACEHOLDER = '{{$.pipeline_job_uuid}}'

A placeholder used to obtain a pipeline job ID within a task at pipeline runtime. In Kubeflow Pipelines, this maps to the pipeline run UUID.

Example

@dsl.pipeline
def my_pipeline():
    print_op(
        msg='Job ID:',
        value=dsl.PIPELINE_JOB_ID_PLACEHOLDER,
    )
kfp.dsl.PIPELINE_TASK_NAME_PLACEHOLDER = '{{$.pipeline_task_name}}'

A placeholder used to obtain a task name within a task at pipeline runtime. In Kubeflow Pipelines, this maps to the component name.

Example

@dsl.pipeline
def my_pipeline():
    print_op(
        msg='Task name:',
        value=dsl.PIPELINE_TASK_NAME_PLACEHOLDER,
    )
kfp.dsl.PIPELINE_TASK_ID_PLACEHOLDER = '{{$.pipeline_task_uuid}}'

A placeholder used to obtain a task ID within a task at pipeline runtime. In Kubeflow Pipelines, this maps to the component’s ML Metadata (MLMD) execution ID.

Example

@dsl.pipeline
def my_pipeline():
    print_op(
        msg='Task ID:',
        value=dsl.PIPELINE_TASK_ID_PLACEHOLDER,
    )
kfp.dsl.PIPELINE_TASK_EXECUTOR_OUTPUT_PATH_PLACEHOLDER = '{{$.outputs.output_file}}'

A placeholder used to obtain the path to the executor_output.json file within the task container.

Example

@dsl.pipeline
def my_pipeline():
    create_artifact_with_metadata(
        metadata={'foo': 'bar'},
        executor_output_destination=dsl.PIPELINE_TASK_EXECUTOR_OUTPUT_PATH_PLACEHOLDER,
    )
kfp.dsl.PIPELINE_TASK_EXECUTOR_INPUT_PLACEHOLDER = '{{$}}'

A placeholder used to obtain executor input message passed to the task.

Example

@dsl.pipeline
def my_pipeline():
    custom_container_op(
        executor_input=dsl.PIPELINE_TASK_EXECUTOR_INPUT_PLACEHOLDER,
    )
kfp.dsl.PIPELINE_ROOT_PLACEHOLDER = '{{$.pipeline_root}}'

A placeholder used to obtain the pipeline root.

Example

@dsl.pipeline
def my_pipeline():
    store_model(
        tmp_dir=dsl.PIPELINE_ROOT_PLACEHOLDER+'/tmp',
    )
kfp.dsl.PIPELINE_JOB_CREATE_TIME_UTC_PLACEHOLDER = '{{$.pipeline_job_create_time_utc}}'

A placeholder used to obtain the time that a pipeline job was created.

Example

@dsl.pipeline
def my_pipeline():
    print_op(
        msg='Job created at:',
        value=dsl.PIPELINE_JOB_CREATE_TIME_UTC,
    )
kfp.dsl.PIPELINE_JOB_SCHEDULE_TIME_UTC_PLACEHOLDER = '{{$.pipeline_job_schedule_time_utc}}'

A placeholder used to obtain the time for which a pipeline job is scheduled.

Example

@dsl.pipeline
def my_pipeline():
    print_op(
        msg='Job scheduled at:',
        value=dsl.PIPELINE_JOB_SCHEDULE_TIME_UTC,
    )
kfp.dsl.WORKSPACE_PATH_PLACEHOLDER = '{{$.workspace_path}}'

A placeholder used to obtain the path to the shared workspace within a component.

Example

@dsl.pipeline(
    pipeline_config=dsl.PipelineConfig(
        workspace=dsl.WorkspaceConfig(size='100Gi'),
    ),
)
def my_pipeline():
    clone_repo_task = clone_repo(
        workspacePath=dsl.WORKSPACE_PATH_PLACEHOLDER, repo='https://github.com/example/repo',
    )
kfp.dsl.run_notebook(**kwargs: Any) → None[source]

Execute the component’s embedded Jupyter notebook with injected parameters.

This is a stub placeholder. Inside a function decorated with @dsl.notebook_component, the SDK binds this symbol at runtime to a helper that materializes the embedded notebook, injects a parameters cell from the provided **kwargs, and executes the notebook via nbclient.

Calling this function outside of a notebook component context will raise NotImplementedError.

Parameters:

**kwargs – Parameter names and values to inject into the notebook’s parameters cell.

Raises:

NotImplementedError – Always, unless overridden at runtime inside a notebook component.

kfp.dsl.component(func: Callable | None = None, *, base_image: str | None = None, target_image: str | None = None, packages_to_install: List[str] = None, pip_index_urls: List[str] | None = None, output_component_file: str | None = None, install_kfp_package: bool = True, kfp_package_path: str | None = None, pip_trusted_hosts: List[str] | None = None, use_venv: bool = False, additional_funcs: List[Callable] | None = None, embedded_artifact_path: str | None = None, task_config_passthroughs: List[TaskConfigPassthrough | TaskConfigField] | None = None, artifact_previews: bool = True, artifact_preview_rows: bool = True)[source]

Decorator for Python-function based components.

A KFP component can either be a lightweight component or a containerized component.

If target_image is not specified, this function creates a lightweight component. A lightweight component is a self-contained Python function that includes all necessary imports and dependencies. In lightweight components, packages_to_install will be used to install dependencies at runtime. The parameters install_kfp_package and kfp_package_path can be used to control how and from where KFP should be installed when the lightweight component is executed.

If target_image is specified, this function creates a component definition based around the target_image. The assumption is that the function in func will be packaged by KFP into this target_image. You can use the KFP CLI’s build command to package the function into target_image.

Parameters:
  • func – Python function from which to create a component. The function should have type annotations for all its arguments, indicating how each argument is intended to be used (e.g. as an input/output artifact, a plain parameter, or a path to a file).

  • base_image – Image to use when executing the Python function. It should contain a default Python interpreter that is compatible with KFP.

  • target_image – Image to use when creating containerized components.

  • packages_to_install – List of packages to install before executing the Python function. These will always be installed at component runtime.

  • pip_index_urls – Python Package Index base URLs from which to install packages_to_install. Defaults to installing from only PyPI ('https://pypi.org/simple'). For more information, see pip install docs.

  • output_component_file –

    If specified, this function will write a shareable/loadable version of the component spec into this file.

    Warning: This compilation approach is deprecated.

  • embedded_artifact_path – Optional path to a local file or directory to embed into the component. At runtime the embedded content is extracted into a temporary directory and made available via parameters annotated as dsl.EmbeddedInput[T] (e.g., dsl.EmbeddedInput[dsl.Dataset]). For a directory, the injected artifact’s path points to the extraction root (the temporary directory containing the directory’s contents). For a single file, the injected artifact’s path points to the extracted file. The extraction root is also prepended to sys.path to enable importing embedded Python modules.

  • install_kfp_package – Specifies if the KFP SDK should add the kfp Python package to packages_to_install. Lightweight Python functions always require an installation of KFP in base_image to work. If you specify a base_image that already contains KFP, you can set this to False. This flag is ignored when target_image is specified, which implies a choice to build a containerized component. Containerized components will always install KFP as part of the build process.

  • kfp_package_path – Specifies the location from which to install KFP. By default, this will try to install from PyPI using the same version as that used when this component was created. Component authors can choose to override this to point to a GitHub pull request or other pip-compatible package server.

  • use_venv – Specifies if the component should be executed in a virtual environment. The environment will be created in a temporary directory and will inherit the system site packages. This is useful in restricted environments where most of the system is read-only.

  • additional_funcs – List of additional functions to include in the component. These functions will be available to the main function. This is useful for adding util functions that are shared across multiple components but are not packaged as an importable Python package.

  • task_config_passthroughs – List of task configurations (e.g. resources, env, volumes etc.) to pass through to the component. This is useful when the component launches another Kubernetes resource (for example, a Kubeflow Trainer job). Use this in conjunction with dsl.TaskConfig.

  • artifact_previews – Automatically generates bounded previews for directory Dataset and Model outputs by default. Set to False to disable previews for this component. A pipeline with PipelineConfig(artifact_previews=False) also disables generation; setting this component option to True does not override it. Supports Parquet, CSV, JSON, JSONL, Arrow/Feather, XGBoost, PyTorch, and Transformers outputs. Scientific libraries remain optional component dependencies. Model save hooks are scoped to execution and restored afterwards. Preview failures are logged without failing successful artifact production.

  • artifact_preview_rows – Includes up to ten dataset rows when previews are enabled. Set to False for schema-only dataset previews.

Returns:

A component task factory that can be used in pipeline definitions.

Example

from kfp import dsl

@dsl.component
def my_function_one(input: str, output: Output[Model]):
    ...

@dsl.component(
base_image='python:3.11',
output_component_file='my_function.yaml'
)
def my_function_two(input: Input[Mode])):
    ...

@dsl.pipeline(name='my-pipeline', pipeline_root='...')
def pipeline():
    my_function_one_task = my_function_one(input=...)
    my_function_two_task = my_function_two(input=my_function_one_task.outputs)
kfp.dsl.container_component(func: Callable) → ContainerComponent[source]

Decorator for container-based components in KFP v2.

Parameters:

func – The python function to create a component from. The function should have type annotations for all its arguments, indicating how it is intended to be used (e.g. as an input/output Artifact object, a plain parameter, or a path to a file).

Example

from kfp.dsl import container_component, ContainerSpec, InputPath, OutputPath, Output

@container_component
def my_component(
    dataset_path: InputPath(Dataset),
    model: Output[Model],
    num_epochs: int,
    output_parameter: OutputPath(str),
):
    return ContainerSpec(
        image='gcr.io/my-image',
        command=['sh', 'my_component.sh'],
        args=[
        '--dataset_path', dataset_path,
        '--model_path', model.path,
        '--output_parameter_path', output_parameter,
    ]
)
kfp.dsl.pipeline(func: Callable | None = None, *, name: str | None = None, description: str | None = None, pipeline_root: str | None = None, display_name: str | None = None, pipeline_config: PipelineConfig = None) → Callable[source]

Decorator used to construct a pipeline.

Example
@pipeline(
  name='my-pipeline',
  description='My ML Pipeline.'
  pipeline_root='gs://my-bucket/my-output-path'
)
def my_pipeline(a: str, b: int):
  ...
Parameters:
  • func – The Python function that defines a pipeline.

  • name – The pipeline name. Defaults to a sanitized version of the decorated function name.

  • description – A human-readable description of the pipeline.

  • pipeline_root – The root directory from which to read input and output parameters and artifacts.

  • display_name – A human-readable name for the pipeline.

  • pipeline_config – Pipeline-level config options.

kfp.dsl.importer(artifact_uri: PipelineParameterChannel | str, artifact_class: Type[Artifact], reimport: bool = False, metadata: Mapping[str, Any] | None = None, download_to_workspace: bool = False) → PipelineTask[source]

Imports an existing artifact for use in a downstream component.

Parameters:
  • artifact_uri – The URI of the artifact to import.

  • artifact_class – The artifact class being imported.

  • reimport – Whether to reimport the artifact.

  • metadata – Properties of the artifact.

  • download_to_workspace – If true, download artifact into pipeline workspace.

Returns:

A task with the artifact accessible via its .output attribute.

Examples:

@dsl.pipeline(name='pipeline-with-importer')
def pipeline_with_importer():

    importer1 = importer(
        artifact_uri='gs://ml-pipeline-playground/shakespeare1.txt',
        artifact_class=Dataset,
        reimport=False)
    train(dataset=importer1.output)
class kfp.dsl.ContainerSpec(image: str, command: List[str | Placeholder] | None = None, args: List[str | Placeholder] | None = None)[source]

Bases: object

Container definition.

This is only used for pipeline authors when constructing a containerized component using @container_component decorator.

Examples

@container_component
def container_with_artifact_output(
    num_epochs: int,  # built-in types are parsed as inputs
    model: Output[Model],
    model_config_path: OutputPath(str),
):
    return ContainerSpec(
        image='gcr.io/my-image',
        command=['sh', 'run.sh'],
        args=[
            '--epochs',
            num_epochs,
            '--model_path',
            model.uri,
            '--model_config_path',
            model_config_path,
        ])

Attributes:

image

Container image.

command

Container entrypoint.

args

Arguments to the container entrypoint.

image: str

Container image.

command: List[str | Placeholder] | None = None

Container entrypoint.

args: List[str | Placeholder] | None = None

Arguments to the container entrypoint.

class kfp.dsl.Condition(condition, name: str | None = None)[source]

Bases: If

Deprecated.

Use dsl.If instead.

class kfp.dsl.If(condition, name: str | None = None)[source]

Bases: _ConditionBase

A class for creating a conditional control flow “if” block within a pipeline.

Parameters:
  • condition – A comparative expression that evaluates to True or False. At least one of the operands must be an output from an upstream task or a pipeline parameter.

  • name – The name of the condition group.

Example

task1 = my_component1(...)
with dsl.If(task1.output=='pizza', 'pizza-condition'):
    task2 = my_component2(...)
class kfp.dsl.Elif(condition, name: str | None = None)[source]

Bases: _ConditionBase

A class for creating a conditional control flow “else if” block within a pipeline. Can be used following an upstream dsl.If or dsl.Elif.

Parameters:
  • condition – A comparative expression that evaluates to True or False. At least one of the operands must be an output from an upstream task or a pipeline parameter.

  • name – The name of the condition group.

Example

task1 = my_component1(...)
task2 = my_component2(...)
with dsl.If(task1.output=='pizza', 'pizza-condition'):
    task3 = my_component3(...)

with dsl.Elif(task2.output=='pasta', 'pasta-condition'):
    task4 = my_component4(...)
class kfp.dsl.Else(name: str | None = None)[source]

Bases: _ConditionBase

A class for creating a conditional control flow “else” block within a pipeline. Can be used following an upstream dsl.If or dsl.Elif.

Parameters:

name – The name of the condition group.

Example

task1 = my_component1(...)
task2 = my_component2(...)
with dsl.If(task1.output=='pizza', 'pizza-condition'):
    task3 = my_component3(...)

with dsl.Elif(task2.output=='pasta', 'pasta-condition'):
    task4 = my_component4(...)

with dsl.Else():
    my_component5(...)
class kfp.dsl.OneOf(*channels: PipelineParameterChannel | PipelineArtifactChannel)[source]

Bases: object

For collecting mutually exclusive outputs from conditional branches into a single pipeline channel.

Parameters:

channels – The channels to collect into a OneOf. Must be of the same type.

Example

@dsl.pipeline
def flip_coin_pipeline() -> str:
    flip_coin_task = flip_coin()
    with dsl.If(flip_coin_task.output == 'heads'):
        print_task_1 = print_and_return(text='Got heads!')
    with dsl.Else():
        print_task_2 = print_and_return(text='Got tails!')

    # use the output from the branch that gets executed
    oneof = dsl.OneOf(print_task_1.output, print_task_2.output)

    # consume it
    print_and_return(text=oneof)

    # return it
    return oneof
class kfp.dsl.ExitHandler(exit_task: PipelineTask, name: str | None = None)[source]

Bases: TasksGroup

A class for setting an exit handler task that is invoked upon exiting a group of other tasks.

Parameters:
  • exit_task – The task that is invoked after exiting a group of other tasks.

  • name – The name of the exit handler group.

Example

exit_task = ExitComponent(...)
with ExitHandler(exit_task):
    task1 = my_component1(...)
    task2 = my_component2(...)
class kfp.dsl.ParallelFor(items: List[int | float | str | Dict[str, Any]] | PipelineChannel, name: str | None = None, parallelism: int | None = None)[source]

Bases: TasksGroup

A class for creating parallelized for loop control flow over a static set of items within a pipeline definition.

Parameters:
  • items – The items to loop over. It can be either a constant Python list or a list output from an upstream task.

  • name – The name of the for loop group.

  • parallelism – The maximum number of concurrent iterations that can be scheduled for execution. A value of 0 represents unconstrained parallelism (default is unconstrained).

Example

with dsl.ParallelFor(
  items=[{'a': 1, 'b': 10}, {'a': 2, 'b': 20}],
  parallelism=1
) as item:
    task1 = my_component(..., number=item.a)
    task2 = my_component(..., number=item.b)

In the example, the group of tasks containing task1 and task2 would be executed twice, once with case args=[{'a': 1, 'b': 10}] and once with case args=[{'a': 2, 'b': 20}]. The parallelism=1 setting causes only 1 execution to be scheduled at a time.

class kfp.dsl.Collected(output: PipelineChannel)[source]

Bases: PipelineChannel

For collecting into a list the output from a task in dsl.ParallelFor loops.

Parameters:

output – The output of an upstream task within a dsl.ParallelFor loop.

Example

@dsl.pipeline
def math_pipeline() -> int:
    with dsl.ParallelFor([1, 2, 3]) as x:
        t = double(num=x)

return add(nums=dsl.Collected(t.output)).output
class kfp.dsl.IfPresentPlaceholder(input_name: str, then: str | Placeholder | List[str | Placeholder], else_: str | Placeholder | List[str | Placeholder] | None = None)[source]

Bases: Placeholder

Placeholder for handling cases where an input may or may not be passed. May contain other placeholders.

Parameters:
  • input_name – Name of the input/output.

  • then – If the input/output specified in name is present, the command-line argument will be replaced at run-time by the value of then.

  • else – If the input/output specified in name is not present, the command-line argument will be replaced at run-time by the value of else_.

Examples

@container_component
def container_with_if_placeholder(output_path: OutputPath(str),
                                  dataset: Output[Dataset],
                                  optional_input: str = 'default'):
    return ContainerSpec(
            image='python:3.11',
            command=[
                'my_program',
                IfPresentPlaceholder(
                    input_name='optional_input',
                    then=[optional_input],
                    else_=['no_input']), '--dataset',
                IfPresentPlaceholder(
                    input_name='optional_input', then=[dataset.uri], else_=['no_dataset'])
            ],
            args=['--output_path', output_path]
        )
class kfp.dsl.ConcatPlaceholder(items: List[str | Placeholder])[source]

Bases: Placeholder

Placeholder for concatenating multiple strings. May contain other placeholders.

Parameters:

items – Elements to concatenate.

Examples

@container_component
def container_with_concat_placeholder(text1: str, text2: Output[Dataset],
                                      output_path: OutputPath(str)):
    return ContainerSpec(
        image='python:3.11',
        command=[
            'my_program',
            ConcatPlaceholder(['prefix-', text1, text2.uri])
        ],
        args=['--output_path', output_path]
    )
class kfp.dsl.PipelineTask(component_spec: ComponentSpec, args: Dict[str, Any], execute_locally: bool = False, execution_caching_default: bool = True, execution_cache_key: str | None = None)[source]

Bases: object

Represents a pipeline task (instantiated component).

Note: PipelineTask should not be constructed by pipeline authors directly, but instead obtained via an instantiated component (see example).

Replaces ContainerOp from kfp v1. Holds operations available on a task object, such as .after(), .set_memory_limit(), .enable_caching(), etc.

Parameters:
  • component_spec – The component definition.

  • args – The dictionary of arguments on which the component was called to instantiate this task.

Example

@dsl.component
def identity(message: str) -> str:
    return message

@dsl.pipeline(name='my_pipeline')
def my_pipeline():
    # task is an instance of PipelineTask
    task = identity(message='my string')

Attributes:

platform_spec

PlatformSpec for all tasks in the pipeline as task.

name

The name of the task.

inputs

The inputs passed to the task.

output

The single output of the task.

outputs

The dictionary of outputs of the task.

dependent_tasks

A list of the dependent task names.

Methods:

register_pipeline_channels(pipeline_channels)

Registers additional pipeline channels consumed by the task.

set_caching_options(enable_caching[, cache_key])

Sets caching options for the task.

set_cpu_request(cpu)

Sets CPU request (minimum) for the task.

set_cpu_limit(cpu)

Sets CPU limit (maximum) for the task.

set_accelerator_limit(limit)

Sets accelerator limit (maximum) for the task.

set_memory_request(memory)

Sets memory request (minimum) for the task.

set_memory_limit(memory)

Sets memory limit (maximum) for the task.

set_retry(num_retries[, backoff_duration, ...])

Sets task retry parameters.

add_node_selector_constraint(accelerator)

Deprecated.

set_accelerator_type(accelerator)

Sets accelerator type to use when executing this task.

set_display_name(name)

Sets display name for the task.

set_env_variable(name, value)

Sets environment variable for the task.

set_debug_pause([before, after, on_error])

Enable interactive debug-pause for the pipeline task.

set_container_image(name)

Sets the container image to use when executing this task.

after(*tasks)

Specifies an explicit dependency on other tasks by requiring this task be executed after other tasks finish completion.

ignore_upstream_failure()

If called, the pipeline task will run when any specified upstream tasks complete, even if unsuccessful.

property platform_spec: PlatformSpec

PlatformSpec for all tasks in the pipeline as task.

Only for use on tasks created from GraphComponents.

property name: str

The name of the task.

Unique within its parent group.

property inputs: Dict[str, str | int | float | bool | dict | list | PipelineChannel]

The inputs passed to the task.

property output: PipelineChannel

The single output of the task.

Used when a task has exactly one output parameter.

property outputs: Mapping[str, PipelineChannel]

The dictionary of outputs of the task.

Used when a task has more the one output or uses an OutputPath or Output[Artifact] type annotation.

property dependent_tasks: List[str]

A list of the dependent task names.

register_pipeline_channels(pipeline_channels: List[PipelineChannel]) → None[source]

Registers additional pipeline channels consumed by the task.

Parameters:

pipeline_channels – Pipeline channels to add to the task’s tracked inputs.

Channels are deduplicated by their serialized pattern so the same runtime reference is only registered once.

set_caching_options(enable_caching: bool, cache_key: str | None = None) → PipelineTask[source]

Sets caching options for the task.

Parameters:
  • enable_caching – Whether to enable caching.

  • cache_key – Customized cache key for this task. The customized cache key is combined with component reference identity to prevent cross-component cache collisions. When a custom cache key is provided, all automatic inputs, base image, command, arguments, and PVC names are ignored during cache key generation. Default None.

Returns:

Self return to allow chained setting calls.

set_cpu_request(cpu: str | PipelineChannel) → PipelineTask[source]

Sets CPU request (minimum) for the task.

Parameters:

cpu – Minimum CPU requests required. This string should be a number or a number followed by an “m” to indicate millicores (1/1000). For more information, see Specify a CPU Request and a CPU Limit.

Returns:

Self return to allow chained setting calls.

set_cpu_limit(cpu: str | PipelineChannel) → PipelineTask[source]

Sets CPU limit (maximum) for the task.

Parameters:

cpu –

Maximum CPU requests allowed. This string should be a number or a number followed by an “m” to indicate millicores (1/1000). For more information, see Specify a CPU Request and a CPU Limit.

Returns:

Self return to allow chained setting calls.

set_accelerator_limit(limit: int | str | PipelineChannel) → PipelineTask[source]

Sets accelerator limit (maximum) for the task. Only applies if accelerator type is also set via .set_accelerator_type().

Parameters:

limit – Maximum number of accelerators allowed. Must be a non-negative integer.

Returns:

Self return to allow chained setting calls.

set_memory_request(memory: str | PipelineChannel) → PipelineTask[source]

Sets memory request (minimum) for the task.

Parameters:

memory – The minimum memory requests required. This string should be a number or a number followed by one of “E”, “Ei”, “P”, “Pi”, “T”, “Ti”, “G”, “Gi”, “M”, “Mi”, “K”, or “Ki”.

Returns:

Self return to allow chained setting calls.

set_memory_limit(memory: str | PipelineChannel) → PipelineTask[source]

Sets memory limit (maximum) for the task.

Parameters:

memory – The maximum memory requests allowed. This string should be a number or a number followed by one of “E”, “Ei”, “P”, “Pi”, “T”, “Ti”, “G”, “Gi”, “M”, “Mi”, “K”, or “Ki”.

Returns:

Self return to allow chained setting calls.

set_retry(num_retries: int, backoff_duration: str | None = None, backoff_factor: float | None = None, backoff_max_duration: str | None = None, policy: str | None = None) → PipelineTask[source]

Sets task retry parameters.

Parameters:
  • num_retries – Number of times to retry on failure.

  • backoff_duration – Number of seconds to wait before triggering a retry. Defaults to '0s' (immediate retry).

  • backoff_factor – Exponential backoff factor applied to backoff_duration. For example, if backoff_duration="60" (60 seconds) and backoff_factor=2, the first retry will happen after 60 seconds, then again after 120, 240, and so on. Defaults to 2.0.

  • backoff_max_duration – Maximum duration during which the task will be retried. Defaults to '3600s'.

  • policy – Controls which failure types trigger a retry. Only supported when using Argo Workflows as the pipeline execution engine. One of 'Always', 'OnFailure', 'OnError', or 'OnTransientError'. 'OnFailure' retries container exits, including SIGTERM (exit code 143); 'OnError' retries pods that report no exit code, such as deletion or node loss; 'OnTransientError' retries either when the failure message matches the controller’s transient-error pattern; 'Always' retries both. Defaults to None, which omits the policy so the deployment’s Argo configuration applies; KFP’s bundled manifests set 'OnError'.

Returns:

Self return to allow chained setting calls.

add_node_selector_constraint(accelerator: str) → PipelineTask[source]

Deprecated. Use set_accelerator_type() instead.

Sets accelerator type to use when executing this task.

Parameters:

accelerator – The name of the accelerator, such as 'NVIDIA_TESLA_K80', 'TPU_V3', 'nvidia.com/gpu' or 'cloud-tpus.google.com/v3'.

Returns:

Self return to allow chained setting calls.

set_accelerator_type(accelerator: str | PipelineChannel) → PipelineTask[source]

Sets accelerator type to use when executing this task.

Parameters:

accelerator – The name of the accelerator, such as 'NVIDIA_TESLA_K80', 'TPU_V3', 'nvidia.com/gpu' or 'cloud-tpus.google.com/v3'.

Returns:

Self return to allow chained setting calls.

set_display_name(name: str) → PipelineTask[source]

Sets display name for the task.

Parameters:

name – Display name.

Returns:

Self return to allow chained setting calls.

set_env_variable(name: str, value: str) → PipelineTask[source]

Sets environment variable for the task.

Parameters:
  • name – Environment variable name.

  • value – Environment variable value.

Returns:

Self return to allow chained setting calls.

set_debug_pause(before: bool = False, after: bool = True, on_error: bool = False) → PipelineTask[source]

Enable interactive debug-pause for the pipeline task.

Keeps the pod alive so you can kubectl exec into it interactively.

When enabled, Argo Workflows’ executor (the wait container) detects the corresponding ARGO_DEBUG_PAUSE_* environment variable and pauses the workflow node, preventing the pod from terminating.

This requires Argo Workflows 3.5.0 or later.

Parameters:
  • before – If True, pause before the main process starts. Useful for inspecting the environment, installing tools, or modifying inputs before execution.

  • after – If True (default), pause after the main process completes. Modified by on_error.

  • on_error – If True, only pause after execution when the component fails (sets ARGO_DEBUG_PAUSE_ON_ERROR instead of ARGO_DEBUG_PAUSE_AFTER). Requires after=True.

Returns:

Self return to allow chained setting calls.

Raises:
  • ValueError – If after=False and on_error=True (contradictory).

  • ValueError – If both before and after are False.

Example

@dsl.pipeline
def my_pipeline():
    task = my_component()
    task.set_debug_pause()

    task2 = my_component()
    task2.set_debug_pause(before=True, after=False)

    task3 = my_component()
    task3.set_debug_pause(on_error=True)
set_container_image(name: str | PipelineChannel) → PipelineTask[source]

Sets the container image to use when executing this task.

Takes precedence over the base_image specified in the @component decorator.

Unlike base_image, this method supports dynamic values such as Pipeline Parameters or outputs from previous tasks, which are resolved at runtime.

Parameters:

name – The container image name as a static string (e.g., “python:3.11-alpine”) or a dynamic reference (e.g., a PipelineParameter instance or a task output like task.outputs[‘image_name’]).

Returns:

Self return to allow chained setting calls.

after(*tasks: PipelineTask | tasks_group.ExitHandler) → PipelineTask[source]

Specifies an explicit dependency on other tasks by requiring this task be executed after other tasks finish completion.

Parameters:

*tasks – Upstream dependencies after which this task should be executed. Each dependency must be either a PipelineTask or a dsl.ExitHandler group.

Returns:

Self return to allow chained setting calls.

Example

@dsl.pipeline(name='my-pipeline')
def my_pipeline():
    task1 = my_component(text='1st task')
    task2 = my_component(text='2nd task').after(task1)
ignore_upstream_failure() → PipelineTask[source]

If called, the pipeline task will run when any specified upstream tasks complete, even if unsuccessful.

This method effectively turns the caller task into an exit task if the caller task has upstream dependencies.

If the task has no upstream tasks, either via data exchange or an explicit dependency via .after(), this method has no effect.

Returns:

Self return to allow chained setting calls.

Example

@dsl.pipeline()
def my_pipeline(text: str = 'message'):
    task = fail_op(message=text)
    clean_up_task = print_op(
        message=task.output).ignore_upstream_failure()
class kfp.dsl.PipelineConfig(workspace: WorkspaceConfig | None = None, ttl_seconds_after_finished: int | None = None, ttl_seconds_after_success: int | None = None, ttl_seconds_after_failure: int | None = None, active_deadline_seconds: int | None = None, artifact_previews: bool = True)[source]

Bases: object

PipelineConfig contains pipeline-level config options.

workspace

Optional configuration for a shared workspace PVC that persists for the duration of the pipeline run.

ttl_seconds_after_finished

Optional number of seconds to retain the Argo Workflow resource after the pipeline run completes, regardless of success or failure. Maps to Argo’s ttlStrategy.secondsAfterCompletion. When None (the default) no completion TTL is set.

ttl_seconds_after_success

Optional number of seconds to retain the Argo Workflow resource after a successful run. Maps to Argo’s ttlStrategy.secondsAfterSuccess. Takes precedence over ttl_seconds_after_finished for successful runs when both are set.

ttl_seconds_after_failure

Optional number of seconds to retain the Argo Workflow resource after a failed run. Maps to Argo’s ttlStrategy.secondsAfterFailure. Takes precedence over ttl_seconds_after_finished for failed runs when both are set.

active_deadline_seconds

Optional maximum number of seconds a workflow is allowed to run before it is forcibly terminated. Maps to Argo’s activeDeadlineSeconds. When None (the default) no deadline is applied.

artifact_previews

Whether supported Python components automatically preview their Dataset and Model outputs. Defaults to True. Set to False to disable generation throughout this pipeline, including nested pipelines and exit handlers. Component opt-outs remain effective when the pipeline enables previews.

class kfp.dsl.WorkspaceConfig(size: str, kubernetes: KubernetesWorkspaceConfig | None = None)[source]

Bases: object

Configuration for a shared workspace that persists during the pipeline run.

kubernetes

(Optional) Kubernetes-specific configuration for the underlying PVC.

Attributes:

size

The size of the workspace (e.g., '250Gi').

Methods:

get_workspace()

set_size(size)

set_kubernetes_config(kubernetes_config)

property size: str

The size of the workspace (e.g., '250Gi'). This is a required field.

See the Kubernetes quantity documentation for valid quantity formats.

get_workspace() → dict[source]
set_size(size: str)[source]
set_kubernetes_config(kubernetes_config: KubernetesWorkspaceConfig)[source]
class kfp.dsl.KubernetesWorkspaceConfig(pvcSpecPatch: Dict[str, Any] | None = None)[source]

Bases: object

Configuration for Kubernetes-specific workspace settings.

Use this to override the default PersistentVolumeClaim (PVC) configuration used when running pipelines on a Kubernetes cluster.

pvcSpecPatch

A dictionary of fields to patch onto the default PVC spec (e.g., ‘storageClassName’, ‘accessModes’).

Methods:

set_pvcSpecPatch(patch: Dict[str, Any])[source]
class kfp.dsl.TaskConfigField(*values)[source]

Bases: IntEnum

Attributes:

RESOURCES = 1
ENV = 2
KUBERNETES_AFFINITY = 3
KUBERNETES_TOLERATIONS = 4
KUBERNETES_NODE_SELECTOR = 5
KUBERNETES_VOLUMES = 6
KUBERNETES_RESOURCE_CLAIMS = 7
class kfp.dsl.TaskConfigPassthrough(field: kfp.dsl.component_task_config.TaskConfigField, apply_to_task: bool = False)[source]

Bases: object

Attributes:

Methods:

to_proto()

Converts this object to its proto representation.

field: TaskConfigField
apply_to_task: bool = False
to_proto() → TaskConfigPassthrough[source]

Converts this object to its proto representation.

kfp.dsl.notebook_component(func: Callable[[...], Any] | None = None, *, notebook_path: str, base_image: str | None = None, packages_to_install: List[str] | None = None, output_component_file: str | None = None, pip_index_urls: List[str] | None = None, pip_trusted_hosts: List[str] | None = None, use_venv: bool = False, kfp_package_path: str | None = None, install_kfp_package: bool = True, task_config_passthroughs: List[TaskConfigPassthrough] | None = None)[source]

Decorator to define a Notebook-based KFP component.

Parameters:
  • notebook_path – Path to the .ipynb file or a directory containing one to embed and execute.

  • base_image – Base container image for the component.

  • packages_to_install – Runtime-only packages to install inside the component container. When None, defaults to [“nbclient>=0.10,<1”, “ipykernel>=6,<7”, “jupyter_client>=7,<9”, “fastjsonschema<2.22; python_version < ‘3.10’”] to ensure the notebook can execute with a Python kernel and client. When [], installs nothing. When non-empty, installs the exact list.

  • output_component_file – Optional path to write the component YAML.

  • pip_index_urls – Optional pip index URLs for installation.

  • pip_trusted_hosts – Optional pip trusted hosts.

  • use_venv – Whether to create and use a venv inside the container.

  • kfp_package_path – Optional KFP package path to install.

  • install_kfp_package – Whether to auto-install KFP when appropriate.

  • task_config_passthroughs – Optional task config passthroughs.

Parameter injection and execution:
  • The notebook bytes are embedded in the component. At runtime, the helper binds dsl.run_notebook(**kwargs).

  • Inside the decorated function body, call dsl.run_notebook(…) with keyword arguments for parameters you want available in the notebook (e.g., dsl.run_notebook(text=my_text)).

  • Parameters are injected following Papermill semantics: if the notebook contains a code cell tagged parameters, an overriding injected-parameters cell is inserted immediately after it; otherwise the injected cell is placed at the top of the notebook before execution.

  • Notebook outputs you want to expose as KFP outputs should be written by the notebook to known paths, then copied or logged by the function after dsl.run_notebook(…) returns.