kfp.dsl¶
The kfp.dsl module contains domain-specific language objects used to compose pipelines.
Data:
Type generic used to represent an input artifact of type |
|
A placeholder used to obtain a pipeline job name within a task at pipeline runtime. |
|
A placeholder used to obtain a pipeline job resource name within a task at pipeline runtime. |
|
A placeholder used to obtain a pipeline job ID within a task at pipeline runtime. |
|
A placeholder used to obtain a task name within a task at pipeline runtime. |
|
A placeholder used to obtain a task ID within a task at pipeline runtime. |
|
A placeholder used to obtain the path to the executor_output.json file within the task container. |
|
A placeholder used to obtain executor input message passed to the task. |
|
A placeholder used to obtain the pipeline root. |
|
A placeholder used to obtain the time that a pipeline job was created. |
|
A placeholder used to obtain the time for which a pipeline job is scheduled. |
|
A placeholder used to obtain the path to the shared workspace within a component. |
Classes:
|
Type annotation used in component definitions for indicating a parameter is a path to an input. |
|
Type annotation used in component definitions for indicating a parameter is a path to an output. |
|
A final status of a pipeline task. |
|
Configurations for a task. |
|
Represents a generic machine learning artifact. |
|
An artifact for storing classification metrics. |
|
An artifact representing a machine learning dataset. |
|
An artifact representing an HTML file. |
|
An artifact representing a markdown file. |
|
An artifact for storing key-value scalar metrics. |
|
An artifact representing a machine learning model. |
|
An artifact for storing sliced classification metrics. |
|
Container definition. |
|
Deprecated. |
|
A class for creating a conditional control flow "if" block within a pipeline. |
|
A class for creating a conditional control flow "else if" block within a pipeline. |
|
A class for creating a conditional control flow "else" block within a pipeline. |
|
For collecting mutually exclusive outputs from conditional branches into a single pipeline channel. |
|
A class for setting an exit handler task that is invoked upon exiting a group of other tasks. |
|
A class for creating parallelized for loop control flow over a static set of items within a pipeline definition. |
|
For collecting into a list the output from a task in dsl.ParallelFor loops. |
|
Placeholder for handling cases where an input may or may not be passed. |
|
Placeholder for concatenating multiple strings. |
|
Represents a pipeline task (instantiated component). |
|
PipelineConfig contains pipeline-level config options. |
|
Configuration for a shared workspace that persists during the pipeline run. |
|
Configuration for Kubernetes-specific workspace settings. |
|
|
|
Functions:
|
Gets the task root URI, a unique object storage URI associated with the current task. |
|
Execute the component's embedded Jupyter notebook with injected parameters. |
|
Decorator for Python-function based components. |
|
Decorator for container-based components in KFP v2. |
|
Decorator used to construct a pipeline. |
|
Imports an existing artifact for use in a downstream component. |
|
Decorator to define a Notebook-based KFP component. |
- kfp.dsl.Input(*args, **kwargs)¶
Type generic used to represent an input artifact of type
T, whereTis an artifact class.Use
Input[Artifact]orOutput[Artifact]to indicate whether the enclosed artifact is a component input or output.- Parameters:
T – The type of the input artifact.
Example
@dsl.component def artifact_producer(model: Output[Artifact]): with open(model.path, 'w') as f: f.write('my model') @dsl.component def artifact_consumer(model: Input[Artifact]): print(model) @dsl.pipeline def my_pipeline(): producer_task = artifact_producer() artifact_consumer(model=producer_task.output)
alias of
Annotated[T,InputAnnotation]
- class kfp.dsl.InputPath(type=None)[source]¶
Bases:
objectType annotation used in component definitions for indicating a parameter is a path to an input.
Example
@dsl.component def create_dataset(dataset_path: OutputPath('Dataset'),): import json dataset = {'my_dataset': [[1, 2, 3], [4, 5, 6]]} with open(dataset_path, 'w') as f: json.dump(dataset, f) @dsl.component def consume_dataset(dataset: InputPath('Dataset')): print(dataset) @dsl.pipeline(name='my-pipeline', pipeline_root='gs://my-bucket') def my_pipeline(): create_dataset_op = create_dataset() consume_dataset(dataset=create_dataset_op.outputs['dataset_path'])
- class kfp.dsl.OutputPath(type=None)[source]¶
Bases:
objectType annotation used in component definitions for indicating a parameter is a path to an output. The path parameter typed with this annotation can be treated as a locally accessible filepath within the component body.
The argument typed with this annotation is provided at runtime by the executing backend and does not need to be passed as an input by the pipeline author (see example).
- Parameters:
type – The type of the value written to the output path.
Example
@dsl.component def create_parameter( message: str, output_parameter_path: OutputPath(str), ): with open(output_parameter_path, 'w') as f: f.write(message) @dsl.component def consume_parameter(message: str): print(message) @dsl.pipeline(name='my-pipeline', pipeline_root='gs://my-bucket') def my_pipeline(message: str = 'default message'): create_param_op = create_parameter(message=message) consume_parameter(message=create_param_op.outputs['output_parameter_path'])
- class kfp.dsl.PipelineTaskFinalStatus(state: str, pipeline_job_resource_name: str, pipeline_task_name: str, error_code: int | None, error_message: str | None)[source]¶
Bases:
objectA final status of a pipeline task. Annotate a component parameter with this class to obtain a handle to a task’s status (see example).
This is the Python representation of the proto message PipelineTaskFinalStatus.
Examples
@dsl.component def task_status(user_input: str, status: PipelineTaskFinalStatus): print('Pipeline status: ', status.state) print('Job resource name: ', status.pipeline_job_resource_name) print('Pipeline task name: ', status.pipeline_task_name) print('Error code: ', status.error_code) print('Error message: ', status.error_message) @dsl.pipeline(name='my_pipeline') def my_pipeline(): task = task_status(user_input='my_input')
Attributes:
Final state of the task.
Pipeline job resource name, in the format of
projects/{project}/locations/{location}/pipelineJobs/{pipeline_job}.Name of the task that produced this status.
The google.rpc.Code in case of error.
In case of error, the detailed error message.
- state: str¶
Final state of the task. The value could be one of
'SUCCEEDED','FAILED'or'CANCELLED'.
- pipeline_job_resource_name: str¶
Pipeline job resource name, in the format of
projects/{project}/locations/{location}/pipelineJobs/{pipeline_job}.
- pipeline_task_name: str¶
Name of the task that produced this status.
- error_code: int | None¶
The google.rpc.Code in case of error. If state is
'SUCCEEDED', this isNone.
- error_message: str | None¶
In case of error, the detailed error message. If state is
'SUCCEEDED', this isNone.
- class kfp.dsl.TaskConfig(affinity: Dict[str, Any] | None = None, tolerations: List[Dict[str, Any]] | None = None, node_selector: Dict[str, str] | None = None, env: List[Dict[str, Any]] | None = None, volumes: List[Dict[str, Any]] | None = None, volume_mounts: List[Dict[str, Any]] | None = None, resources: Dict[str, Any] | None = None, resource_claims: List[Dict[str, Any]] | None = None)[source]¶
Bases:
objectConfigurations for a task.
Annotate a component parameter with this type when you want the task’s runtime configuration to be forwarded to an external workload and optionally applied to the task’s own pod. This is useful when the task launches another Kubernetes resource (for example, a Kubeflow Trainer job).
This is an empty value by default and is populated if the component is annotated with task_config_passthroughs.
All fields are optional and map 1:1 to fragments of the Kubernetes Pod spec. These fields have values as Python dictionaries/lists that conform to the Kubernetes JSON schema.
Example
@dsl.component( packages_to_install=["kubernetes"], task_config_passthroughs=[ dsl.TaskConfigField.RESOURCES, dsl.TaskConfigField.KUBERNETES_TOLERATIONS, dsl.TaskConfigField.KUBERNETES_NODE_SELECTOR, dsl.TaskConfigField.KUBERNETES_AFFINITY, dsl.TaskConfigPassthrough(field=dsl.TaskConfigField.ENV, apply_to_task=True), dsl.TaskConfigPassthrough(field=dsl.TaskConfigField.KUBERNETES_VOLUMES, apply_to_task=True), dsl.TaskConfigField.KUBERNETES_RESOURCE_CLAIMS, ], ) def train(num_nodes: int, workspace_path: str, output_model: dsl.Output[dsl.Model], task_config: dsl.TaskConfig): import os import shutil from kubernetes import client as k8s_client, config config.load_incluster_config() with open( "/var/run/secrets/kubernetes.io/serviceaccount/namespace", "r" ) as ns_file: namespace = ns_file.readline() train_job_script = "with open('/kfp-workspace/model', 'w') as f: f.write('hello')" container_resources = dict(task_config.resources or {}) if task_config.resource_claims: container_resources["claims"] = [ {"name": claim["name"]} for claim in task_config.resource_claims ] dataset_path = os.path.join(workspace_path, "dataset") with open(dataset_path, "w") as f: f.write("Prepare dataset here...") train_job = { "apiVersion": "trainer.kubeflow.org/v1alpha1", "kind": "TrainJob", "metadata": {"name": f"kfp-train-job", "namespace": namespace}, "spec": { "runtimeRef": {"name": "torch-distributed"}, "trainer": { "numNodes": num_nodes, "resourcesPerNode": task_config.resources, "env": task_config.env, "command": ["python", "-c", train_job_script], }, "podSpecOverrides": [ { "targetJobs": [{"name": "node"}], "volumes": task_config.volumes, "containers": [ { "name": "node", "volumeMounts": task_config.volume_mounts, "resources": container_resources, } ], "nodeSelector": task_config.node_selector, "tolerations": task_config.tolerations, "resourceClaims": task_config.resource_claims, } ], }, } print(train_job) api_client = k8s_client.ApiClient() custom_objects_api = k8s_client.CustomObjectsApi(api_client) response = custom_objects_api.create_namespaced_custom_object( group="trainer.kubeflow.org", version="v1alpha1", namespace=namespace, plural="trainjobs", body=train_job, ) job_name = response["metadata"]["name"] print(f"TrainJob {job_name} created successfully") print("Polling train job code goes here...") print("Copying output model") shutil.copy(os.path.join(workspace_path, "model"), output_model.path) @dsl.pipeline def example_task_config(): train_task = train(num_nodes=1, workspace_path=dsl.WORKSPACE_PATH_PLACEHOLDER) train_task.set_cpu_request("1") train_task.set_memory_request("20Gi") train_task.set_cpu_limit("2") train_task.set_memory_limit("50Gi") train_task.set_accelerator_type("nvidia.com/gpu") train_task.set_accelerator_limit("1")
Attributes:
- affinity: Dict[str, Any] | None = None¶
- tolerations: List[Dict[str, Any]] | None = None¶
- node_selector: Dict[str, str] | None = None¶
- env: List[Dict[str, Any]] | None = None¶
- volumes: List[Dict[str, Any]] | None = None¶
- volume_mounts: List[Dict[str, Any]] | None = None¶
- resources: Dict[str, Any] | None = None¶
- resource_claims: List[Dict[str, Any]] | None = None¶
- class kfp.dsl.Artifact(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]¶
Bases:
objectRepresents a generic machine learning artifact.
This class and all artifact classes store the name, uri, and metadata for a machine learning artifact. Use this artifact type when an artifact does not fit into another more specific artifact type (e.g.,
Model,Dataset).- Parameters:
name – Name of the artifact.
uri – The artifact’s location on disk or cloud storage.
metadata – Arbitrary key-value pairs about the artifact.
Example
from kfp import dsl from kfp.dsl import Output, Artifact, Input @dsl.component def create_artifact( data: str, output_artifact: Output[Artifact], ): with open(output_artifact.path, 'w') as f: f.write(data) @dsl.component def use_artifact(input_artifact: Input[Artifact]): with open(input_artifact.path) as input_file: artifact_contents = input_file.read() print(artifact_contents) @dsl.pipeline(name='my-pipeline', pipeline_root='gs://my/storage') def my_pipeline(): create_task = create_artifact(data='my data') use_artifact(input_artifact=create_task.outputs['output_artifact'])
Note: Other artifacts are used similarly to the usage of
Artifactin the example above (withinInput[]andOutput[]).Attributes:
Methods:
set_path(path)- schema_title = 'system.Artifact'¶
- schema_version = '0.0.1'¶
- property path: str¶
- property custom_path: str¶
- class kfp.dsl.ClassificationMetrics(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]¶
Bases:
ArtifactAn artifact for storing classification metrics.
- Parameters:
name – Name of the metrics artifact.
uri – The metrics artifact’s location on disk or cloud storage.
metadata – The key-value scalar metrics.
Attributes:
Methods:
log_roc_data_point(fpr, tpr, threshold)Logs a single data point in the ROC curve to metadata.
log_roc_curve(fpr, tpr, threshold)Logs an ROC curve to metadata.
set_confusion_matrix_categories(categories)Stores confusion matrix categories to metadata.
log_confusion_matrix_row(row_category, row)Logs a confusion matrix row to metadata.
log_confusion_matrix_cell(row_category, ...)Logs a cell in the confusion matrix to metadata.
log_confusion_matrix(categories, matrix)Logs a confusion matrix to metadata.
- schema_title = 'system.ClassificationMetrics'¶
- log_roc_data_point(fpr: float, tpr: float, threshold: float) None[source]¶
Logs a single data point in the ROC curve to metadata.
- Parameters:
fpr – False positive rate value of the data point.
tpr – True positive rate value of the data point.
threshold – Threshold value for the data point.
- log_roc_curve(fpr: List[float], tpr: List[float], threshold: List[float]) None[source]¶
Logs an ROC curve to metadata.
- Parameters:
fpr – List of false positive rate values.
tpr – List of true positive rate values.
threshold – List of threshold values.
- Raises:
ValueError – If the lists
fpr,tprandthresholdare not the same length.
- set_confusion_matrix_categories(categories: List[str]) None[source]¶
Stores confusion matrix categories to metadata.
- Parameters:
categories – List of strings specifying the categories.
- log_confusion_matrix_row(row_category: str, row: List[float]) None[source]¶
Logs a confusion matrix row to metadata.
- Parameters:
row_category – Category to which the row belongs.
row – List of integers specifying the values for the row.
- Raises:
ValueError – If
row_categoryis not in the list of categories set inset_categoriescall.
- log_confusion_matrix_cell(row_category: str, col_category: str, value: int) None[source]¶
Logs a cell in the confusion matrix to metadata.
- Parameters:
row_category – String representing the name of the row category.
col_category – String representing the name of the column category.
value – Value of the cell.
- Raises:
ValueError – If
row_categoryorcol_categoryis not in the list of categories set inset_categories.
- log_confusion_matrix(categories: List[str], matrix: List[List[int]]) None[source]¶
Logs a confusion matrix to metadata.
- Parameters:
categories – List of the category names.
matrix – Complete confusion matrix.
- Raises:
ValueError – If the length of
categoriesdoes not match number of rows or columns ofmatrix.
- class kfp.dsl.Dataset(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]¶
Bases:
ArtifactAn artifact representing a machine learning dataset.
- Parameters:
name – Name of the dataset.
uri – The dataset’s location on disk or cloud storage.
metadata – Arbitrary key-value pairs about the dataset.
Attributes:
- schema_title = 'system.Dataset'¶
- class kfp.dsl.HTML(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]¶
Bases:
ArtifactAn artifact representing an HTML file.
- Parameters:
name – Name of the HTML file.
uri – The HTML file’s location on disk or cloud storage.
metadata – Arbitrary key-value pairs about the HTML file.
Attributes:
- schema_title = 'system.HTML'¶
- class kfp.dsl.Markdown(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]¶
Bases:
ArtifactAn artifact representing a markdown file.
- Parameters:
name – Name of the markdown file.
uri – The markdown file’s location on disk or cloud storage.
metadata – Arbitrary key-value pairs about the markdown file.
Attributes:
- schema_title = 'system.Markdown'¶
- class kfp.dsl.Metrics(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]¶
Bases:
ArtifactAn artifact for storing key-value scalar metrics.
- Parameters:
name – Name of the metrics artifact.
uri – The metrics artifact’s location on disk or cloud storage.
metadata – Key-value scalar metrics.
Attributes:
Methods:
log_metric(metric, value)Sets a custom scalar metric in the artifact's metadata.
- schema_title = 'system.Metrics'¶
- class kfp.dsl.Model(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]¶
Bases:
ArtifactAn artifact representing a machine learning model.
- Parameters:
name – Name of the model.
uri – The model’s location on disk or cloud storage.
metadata – Arbitrary key-value pairs about the model.
Attributes:
- schema_title = 'system.Model'¶
- property path: str¶
- property framework: str¶
- class kfp.dsl.SlicedClassificationMetrics(name: str | None = None, uri: str | None = None, metadata: Dict | None = None)[source]¶
Bases:
ArtifactAn artifact for storing sliced classification metrics.
Similar to
ClassificationMetrics, tasks using this class are expected to use log methods of the class to log metrics with the difference being each log method takes a slice to associate theClassificationMetrics.- Parameters:
name – Name of the metrics artifact.
uri – The metrics artifact’s location on disk or cloud storage.
metadata – Arbitrary key-value pairs about the metrics artifact.
Attributes:
Methods:
log_roc_reading(slice, threshold, tpr, fpr)Logs a single data point in the ROC curve of a slice to metadata.
load_roc_readings(slice, readings)Bulk loads ROC curve readings for a slice.
set_confusion_matrix_categories(slice, ...)Logs confusion matrix categories for a slice to metadata.
log_confusion_matrix_row(slice, ...)Logs a confusion matrix row for a slice to metadata.
log_confusion_matrix_cell(slice, ...)Logs a confusion matrix cell for a slice to metadata.
load_confusion_matrix(slice, categories, matrix)Bulk loads the whole confusion matrix for a slice.
- schema_title = 'system.SlicedClassificationMetrics'¶
- log_roc_reading(slice: str, threshold: float, tpr: float, fpr: float) None[source]¶
Logs a single data point in the ROC curve of a slice to metadata.
- Parameters:
slice – String representing slice label.
threshold – Thresold value for the data point.
tpr – True positive rate value of the data point.
fpr – False positive rate value of the data point.
- load_roc_readings(slice: str, readings: List[List[float]]) None[source]¶
Bulk loads ROC curve readings for a slice.
- Parameters:
slice – String representing slice label.
readings – A 2-dimensional list providing ROC curve data points. The expected order of the data points is: threshold, true positive rate, false positive rate.
- set_confusion_matrix_categories(slice: str, categories: List[str]) None[source]¶
Logs confusion matrix categories for a slice to metadata.
Categories are stored in the internal
metrics_utils.ConfusionMatrixinstance of the slice.- Parameters:
slice – String representing slice label.
categories – List of strings specifying the categories.
- log_confusion_matrix_row(slice: str, row_category: str, row: List[int]) None[source]¶
Logs a confusion matrix row for a slice to metadata.
Row is updated on the internal
metrics_utils.ConfusionMatrixinstance of the slice.- Parameters:
slice – String representing slice label.
row_category – Category to which the row belongs.
row – List of integers specifying the values for the row.
- log_confusion_matrix_cell(slice: str, row_category: str, col_category: str, value: int) None[source]¶
Logs a confusion matrix cell for a slice to metadata.
Cell is updated on the internal
metrics_utils.ConfusionMatrixinstance of the slice.- Parameters:
slice – String representing slice label.
row_category – String representing the name of the row category.
col_category – String representing the name of the column category.
value – Value of the cell.
- kfp.dsl.get_uri(suffix: str = 'Output') str[source]¶
Gets the task root URI, a unique object storage URI associated with the current task. This function may only be called at task runtime.
Returns an empty string if the task root cannot be inferred from the runtime environment.
- Parameters:
suffix – A suffix to append to the URI. This is a helpful for creating unique subdirectories when the component has multiple outputs.
- Returns:
The URI or empty string.
- kfp.dsl.PIPELINE_JOB_NAME_PLACEHOLDER = '{{$.pipeline_job_name}}'¶
A placeholder used to obtain a pipeline job name within a task at pipeline runtime. In Kubeflow Pipelines, this maps to the pipeline run display name.
Example
@dsl.pipeline def my_pipeline(): print_op( msg='Job name:', value=dsl.PIPELINE_JOB_NAME_PLACEHOLDER, )
- kfp.dsl.PIPELINE_JOB_RESOURCE_NAME_PLACEHOLDER = '{{$.pipeline_job_resource_name}}'¶
A placeholder used to obtain a pipeline job resource name within a task at pipeline runtime. In Kubeflow Pipelines, this maps to the pipeline run name in the underlying pipeline engine (e.g. an Argo Workflow object name).
Example
@dsl.pipeline def my_pipeline(): print_op( msg='Job resource name:', value=dsl.PIPELINE_JOB_RESOURCE_NAME_PLACEHOLDER, )
- kfp.dsl.PIPELINE_JOB_ID_PLACEHOLDER = '{{$.pipeline_job_uuid}}'¶
A placeholder used to obtain a pipeline job ID within a task at pipeline runtime. In Kubeflow Pipelines, this maps to the pipeline run UUID.
Example
@dsl.pipeline def my_pipeline(): print_op( msg='Job ID:', value=dsl.PIPELINE_JOB_ID_PLACEHOLDER, )
- kfp.dsl.PIPELINE_TASK_NAME_PLACEHOLDER = '{{$.pipeline_task_name}}'¶
A placeholder used to obtain a task name within a task at pipeline runtime. In Kubeflow Pipelines, this maps to the component name.
Example
@dsl.pipeline def my_pipeline(): print_op( msg='Task name:', value=dsl.PIPELINE_TASK_NAME_PLACEHOLDER, )
- kfp.dsl.PIPELINE_TASK_ID_PLACEHOLDER = '{{$.pipeline_task_uuid}}'¶
A placeholder used to obtain a task ID within a task at pipeline runtime. In Kubeflow Pipelines, this maps to the component’s ML Metadata (MLMD) execution ID.
Example
@dsl.pipeline def my_pipeline(): print_op( msg='Task ID:', value=dsl.PIPELINE_TASK_ID_PLACEHOLDER, )
- kfp.dsl.PIPELINE_TASK_EXECUTOR_OUTPUT_PATH_PLACEHOLDER = '{{$.outputs.output_file}}'¶
A placeholder used to obtain the path to the executor_output.json file within the task container.
Example
@dsl.pipeline def my_pipeline(): create_artifact_with_metadata( metadata={'foo': 'bar'}, executor_output_destination=dsl.PIPELINE_TASK_EXECUTOR_OUTPUT_PATH_PLACEHOLDER, )
- kfp.dsl.PIPELINE_TASK_EXECUTOR_INPUT_PLACEHOLDER = '{{$}}'¶
A placeholder used to obtain executor input message passed to the task.
Example
@dsl.pipeline def my_pipeline(): custom_container_op( executor_input=dsl.PIPELINE_TASK_EXECUTOR_INPUT_PLACEHOLDER, )
- kfp.dsl.PIPELINE_ROOT_PLACEHOLDER = '{{$.pipeline_root}}'¶
A placeholder used to obtain the pipeline root.
Example
@dsl.pipeline def my_pipeline(): store_model( tmp_dir=dsl.PIPELINE_ROOT_PLACEHOLDER+'/tmp', )
- kfp.dsl.PIPELINE_JOB_CREATE_TIME_UTC_PLACEHOLDER = '{{$.pipeline_job_create_time_utc}}'¶
A placeholder used to obtain the time that a pipeline job was created.
Example
@dsl.pipeline def my_pipeline(): print_op( msg='Job created at:', value=dsl.PIPELINE_JOB_CREATE_TIME_UTC, )
- kfp.dsl.PIPELINE_JOB_SCHEDULE_TIME_UTC_PLACEHOLDER = '{{$.pipeline_job_schedule_time_utc}}'¶
A placeholder used to obtain the time for which a pipeline job is scheduled.
Example
@dsl.pipeline def my_pipeline(): print_op( msg='Job scheduled at:', value=dsl.PIPELINE_JOB_SCHEDULE_TIME_UTC, )
- kfp.dsl.WORKSPACE_PATH_PLACEHOLDER = '{{$.workspace_path}}'¶
A placeholder used to obtain the path to the shared workspace within a component.
Example
@dsl.pipeline( pipeline_config=dsl.PipelineConfig( workspace=dsl.WorkspaceConfig(size='100Gi'), ), ) def my_pipeline(): clone_repo_task = clone_repo( workspacePath=dsl.WORKSPACE_PATH_PLACEHOLDER, repo='https://github.com/example/repo', )
- kfp.dsl.run_notebook(**kwargs: Any) None[source]¶
Execute the component’s embedded Jupyter notebook with injected parameters.
This is a stub placeholder. Inside a function decorated with @dsl.notebook_component, the SDK binds this symbol at runtime to a helper that materializes the embedded notebook, injects a parameters cell from the provided **kwargs, and executes the notebook via nbclient.
Calling this function outside of a notebook component context will raise NotImplementedError.
- Parameters:
**kwargs – Parameter names and values to inject into the notebook’s parameters cell.
- Raises:
NotImplementedError – Always, unless overridden at runtime inside a notebook component.
- kfp.dsl.component(func: Callable | None = None, *, base_image: str | None = None, target_image: str | None = None, packages_to_install: List[str] = None, pip_index_urls: List[str] | None = None, output_component_file: str | None = None, install_kfp_package: bool = True, kfp_package_path: str | None = None, pip_trusted_hosts: List[str] | None = None, use_venv: bool = False, additional_funcs: List[Callable] | None = None, embedded_artifact_path: str | None = None, task_config_passthroughs: List[TaskConfigPassthrough | TaskConfigField] | None = None, artifact_previews: bool = True, artifact_preview_rows: bool = True)[source]¶
Decorator for Python-function based components.
A KFP component can either be a lightweight component or a containerized component.
If
target_imageis not specified, this function creates a lightweight component. A lightweight component is a self-contained Python function that includes all necessary imports and dependencies. In lightweight components,packages_to_installwill be used to install dependencies at runtime. The parametersinstall_kfp_packageandkfp_package_pathcan be used to control how and from where KFP should be installed when the lightweight component is executed.If
target_imageis specified, this function creates a component definition based around thetarget_image. The assumption is that the function infuncwill be packaged by KFP into thistarget_image. You can use the KFP CLI’sbuildcommand to package the function intotarget_image.- Parameters:
func – Python function from which to create a component. The function should have type annotations for all its arguments, indicating how each argument is intended to be used (e.g. as an input/output artifact, a plain parameter, or a path to a file).
base_image – Image to use when executing the Python function. It should contain a default Python interpreter that is compatible with KFP.
target_image – Image to use when creating containerized components.
packages_to_install – List of packages to install before executing the Python function. These will always be installed at component runtime.
pip_index_urls – Python Package Index base URLs from which to install
packages_to_install. Defaults to installing from only PyPI ('https://pypi.org/simple'). For more information, see pip install docs.output_component_file –
If specified, this function will write a shareable/loadable version of the component spec into this file.
Warning: This compilation approach is deprecated.
embedded_artifact_path – Optional path to a local file or directory to embed into the component. At runtime the embedded content is extracted into a temporary directory and made available via parameters annotated as
dsl.EmbeddedInput[T](e.g.,dsl.EmbeddedInput[dsl.Dataset]). For a directory, the injected artifact’spathpoints to the extraction root (the temporary directory containing the directory’s contents). For a single file, the injected artifact’spathpoints to the extracted file. The extraction root is also prepended tosys.pathto enable importing embedded Python modules.install_kfp_package – Specifies if the KFP SDK should add the
kfpPython package topackages_to_install. Lightweight Python functions always require an installation of KFP inbase_imageto work. If you specify abase_imagethat already contains KFP, you can set this toFalse. This flag is ignored whentarget_imageis specified, which implies a choice to build a containerized component. Containerized components will always install KFP as part of the build process.kfp_package_path – Specifies the location from which to install KFP. By default, this will try to install from PyPI using the same version as that used when this component was created. Component authors can choose to override this to point to a GitHub pull request or other pip-compatible package server.
use_venv – Specifies if the component should be executed in a virtual environment. The environment will be created in a temporary directory and will inherit the system site packages. This is useful in restricted environments where most of the system is read-only.
additional_funcs – List of additional functions to include in the component. These functions will be available to the main function. This is useful for adding util functions that are shared across multiple components but are not packaged as an importable Python package.
task_config_passthroughs – List of task configurations (e.g. resources, env, volumes etc.) to pass through to the component. This is useful when the component launches another Kubernetes resource (for example, a Kubeflow Trainer job). Use this in conjunction with dsl.TaskConfig.
artifact_previews – Automatically generates bounded previews for directory Dataset and Model outputs by default. Set to
Falseto disable previews for this component. A pipeline withPipelineConfig(artifact_previews=False)also disables generation; setting this component option toTruedoes not override it. Supports Parquet, CSV, JSON, JSONL, Arrow/Feather, XGBoost, PyTorch, and Transformers outputs. Scientific libraries remain optional component dependencies. Model save hooks are scoped to execution and restored afterwards. Preview failures are logged without failing successful artifact production.artifact_preview_rows – Includes up to ten dataset rows when previews are enabled. Set to
Falsefor schema-only dataset previews.
- Returns:
A component task factory that can be used in pipeline definitions.
Example
from kfp import dsl @dsl.component def my_function_one(input: str, output: Output[Model]): ... @dsl.component( base_image='python:3.11', output_component_file='my_function.yaml' ) def my_function_two(input: Input[Mode])): ... @dsl.pipeline(name='my-pipeline', pipeline_root='...') def pipeline(): my_function_one_task = my_function_one(input=...) my_function_two_task = my_function_two(input=my_function_one_task.outputs)
- kfp.dsl.container_component(func: Callable) ContainerComponent[source]¶
Decorator for container-based components in KFP v2.
- Parameters:
func – The python function to create a component from. The function should have type annotations for all its arguments, indicating how it is intended to be used (e.g. as an input/output Artifact object, a plain parameter, or a path to a file).
Example
from kfp.dsl import container_component, ContainerSpec, InputPath, OutputPath, Output @container_component def my_component( dataset_path: InputPath(Dataset), model: Output[Model], num_epochs: int, output_parameter: OutputPath(str), ): return ContainerSpec( image='gcr.io/my-image', command=['sh', 'my_component.sh'], args=[ '--dataset_path', dataset_path, '--model_path', model.path, '--output_parameter_path', output_parameter, ] )
- kfp.dsl.pipeline(func: Callable | None = None, *, name: str | None = None, description: str | None = None, pipeline_root: str | None = None, display_name: str | None = None, pipeline_config: PipelineConfig = None) Callable[source]¶
Decorator used to construct a pipeline.
- Example
@pipeline( name='my-pipeline', description='My ML Pipeline.' pipeline_root='gs://my-bucket/my-output-path' ) def my_pipeline(a: str, b: int): ...
- Parameters:
func – The Python function that defines a pipeline.
name – The pipeline name. Defaults to a sanitized version of the decorated function name.
description – A human-readable description of the pipeline.
pipeline_root – The root directory from which to read input and output parameters and artifacts.
display_name – A human-readable name for the pipeline.
pipeline_config – Pipeline-level config options.
- kfp.dsl.importer(artifact_uri: PipelineParameterChannel | str, artifact_class: Type[Artifact], reimport: bool = False, metadata: Mapping[str, Any] | None = None, download_to_workspace: bool = False) PipelineTask[source]¶
Imports an existing artifact for use in a downstream component.
- Parameters:
artifact_uri – The URI of the artifact to import.
artifact_class – The artifact class being imported.
reimport – Whether to reimport the artifact.
metadata – Properties of the artifact.
download_to_workspace – If true, download artifact into pipeline workspace.
- Returns:
A task with the artifact accessible via its
.outputattribute.
Examples:
@dsl.pipeline(name='pipeline-with-importer') def pipeline_with_importer(): importer1 = importer( artifact_uri='gs://ml-pipeline-playground/shakespeare1.txt', artifact_class=Dataset, reimport=False) train(dataset=importer1.output)
- class kfp.dsl.ContainerSpec(image: str, command: List[str | Placeholder] | None = None, args: List[str | Placeholder] | None = None)[source]¶
Bases:
objectContainer definition.
This is only used for pipeline authors when constructing a containerized component using @container_component decorator.
Examples
@container_component def container_with_artifact_output( num_epochs: int, # built-in types are parsed as inputs model: Output[Model], model_config_path: OutputPath(str), ): return ContainerSpec( image='gcr.io/my-image', command=['sh', 'run.sh'], args=[ '--epochs', num_epochs, '--model_path', model.uri, '--model_config_path', model_config_path, ])
Attributes:
- image: str¶
Container image.
- command: List[str | Placeholder] | None = None¶
Container entrypoint.
- args: List[str | Placeholder] | None = None¶
Arguments to the container entrypoint.
- class kfp.dsl.Condition(condition, name: str | None = None)[source]¶
Bases:
IfDeprecated.
Use dsl.If instead.
- class kfp.dsl.If(condition, name: str | None = None)[source]¶
Bases:
_ConditionBaseA class for creating a conditional control flow “if” block within a pipeline.
- Parameters:
condition – A comparative expression that evaluates to True or False. At least one of the operands must be an output from an upstream task or a pipeline parameter.
name – The name of the condition group.
Example
task1 = my_component1(...) with dsl.If(task1.output=='pizza', 'pizza-condition'): task2 = my_component2(...)
- class kfp.dsl.Elif(condition, name: str | None = None)[source]¶
Bases:
_ConditionBaseA class for creating a conditional control flow “else if” block within a pipeline. Can be used following an upstream dsl.If or dsl.Elif.
- Parameters:
condition – A comparative expression that evaluates to True or False. At least one of the operands must be an output from an upstream task or a pipeline parameter.
name – The name of the condition group.
Example
task1 = my_component1(...) task2 = my_component2(...) with dsl.If(task1.output=='pizza', 'pizza-condition'): task3 = my_component3(...) with dsl.Elif(task2.output=='pasta', 'pasta-condition'): task4 = my_component4(...)
- class kfp.dsl.Else(name: str | None = None)[source]¶
Bases:
_ConditionBaseA class for creating a conditional control flow “else” block within a pipeline. Can be used following an upstream dsl.If or dsl.Elif.
- Parameters:
name – The name of the condition group.
Example
task1 = my_component1(...) task2 = my_component2(...) with dsl.If(task1.output=='pizza', 'pizza-condition'): task3 = my_component3(...) with dsl.Elif(task2.output=='pasta', 'pasta-condition'): task4 = my_component4(...) with dsl.Else(): my_component5(...)
- class kfp.dsl.OneOf(*channels: PipelineParameterChannel | PipelineArtifactChannel)[source]¶
Bases:
objectFor collecting mutually exclusive outputs from conditional branches into a single pipeline channel.
- Parameters:
channels – The channels to collect into a OneOf. Must be of the same type.
Example
@dsl.pipeline def flip_coin_pipeline() -> str: flip_coin_task = flip_coin() with dsl.If(flip_coin_task.output == 'heads'): print_task_1 = print_and_return(text='Got heads!') with dsl.Else(): print_task_2 = print_and_return(text='Got tails!') # use the output from the branch that gets executed oneof = dsl.OneOf(print_task_1.output, print_task_2.output) # consume it print_and_return(text=oneof) # return it return oneof
- class kfp.dsl.ExitHandler(exit_task: PipelineTask, name: str | None = None)[source]¶
Bases:
TasksGroupA class for setting an exit handler task that is invoked upon exiting a group of other tasks.
- Parameters:
exit_task – The task that is invoked after exiting a group of other tasks.
name – The name of the exit handler group.
Example
exit_task = ExitComponent(...) with ExitHandler(exit_task): task1 = my_component1(...) task2 = my_component2(...)
- class kfp.dsl.ParallelFor(items: List[int | float | str | Dict[str, Any]] | PipelineChannel, name: str | None = None, parallelism: int | None = None)[source]¶
Bases:
TasksGroupA class for creating parallelized for loop control flow over a static set of items within a pipeline definition.
- Parameters:
items – The items to loop over. It can be either a constant Python list or a list output from an upstream task.
name – The name of the for loop group.
parallelism – The maximum number of concurrent iterations that can be scheduled for execution. A value of 0 represents unconstrained parallelism (default is unconstrained).
Example
with dsl.ParallelFor( items=[{'a': 1, 'b': 10}, {'a': 2, 'b': 20}], parallelism=1 ) as item: task1 = my_component(..., number=item.a) task2 = my_component(..., number=item.b)
In the example, the group of tasks containing
task1andtask2would be executed twice, once with caseargs=[{'a': 1, 'b': 10}]and once with caseargs=[{'a': 2, 'b': 20}]. Theparallelism=1setting causes only 1 execution to be scheduled at a time.
- class kfp.dsl.Collected(output: PipelineChannel)[source]¶
Bases:
PipelineChannelFor collecting into a list the output from a task in dsl.ParallelFor loops.
- Parameters:
output – The output of an upstream task within a dsl.ParallelFor loop.
Example
@dsl.pipeline def math_pipeline() -> int: with dsl.ParallelFor([1, 2, 3]) as x: t = double(num=x) return add(nums=dsl.Collected(t.output)).output
- class kfp.dsl.IfPresentPlaceholder(input_name: str, then: str | Placeholder | List[str | Placeholder], else_: str | Placeholder | List[str | Placeholder] | None = None)[source]¶
Bases:
PlaceholderPlaceholder for handling cases where an input may or may not be passed. May contain other placeholders.
- Parameters:
input_name – Name of the input/output.
then – If the input/output specified in name is present, the command-line argument will be replaced at run-time by the value of then.
else – If the input/output specified in name is not present, the command-line argument will be replaced at run-time by the value of
else_.
Examples
@container_component def container_with_if_placeholder(output_path: OutputPath(str), dataset: Output[Dataset], optional_input: str = 'default'): return ContainerSpec( image='python:3.11', command=[ 'my_program', IfPresentPlaceholder( input_name='optional_input', then=[optional_input], else_=['no_input']), '--dataset', IfPresentPlaceholder( input_name='optional_input', then=[dataset.uri], else_=['no_dataset']) ], args=['--output_path', output_path] )
- class kfp.dsl.ConcatPlaceholder(items: List[str | Placeholder])[source]¶
Bases:
PlaceholderPlaceholder for concatenating multiple strings. May contain other placeholders.
- Parameters:
items – Elements to concatenate.
Examples
@container_component def container_with_concat_placeholder(text1: str, text2: Output[Dataset], output_path: OutputPath(str)): return ContainerSpec( image='python:3.11', command=[ 'my_program', ConcatPlaceholder(['prefix-', text1, text2.uri]) ], args=['--output_path', output_path] )
- class kfp.dsl.PipelineTask(component_spec: ComponentSpec, args: Dict[str, Any], execute_locally: bool = False, execution_caching_default: bool = True, execution_cache_key: str | None = None)[source]¶
Bases:
objectRepresents a pipeline task (instantiated component).
Note:
PipelineTaskshould not be constructed by pipeline authors directly, but instead obtained via an instantiated component (see example).Replaces
ContainerOpfromkfpv1. Holds operations available on a task object, such as.after(),.set_memory_limit(),.enable_caching(), etc.- Parameters:
component_spec – The component definition.
args – The dictionary of arguments on which the component was called to instantiate this task.
Example
@dsl.component def identity(message: str) -> str: return message @dsl.pipeline(name='my_pipeline') def my_pipeline(): # task is an instance of PipelineTask task = identity(message='my string')
Attributes:
PlatformSpec for all tasks in the pipeline as task.
The name of the task.
The inputs passed to the task.
The single output of the task.
The dictionary of outputs of the task.
A list of the dependent task names.
Methods:
register_pipeline_channels(pipeline_channels)Registers additional pipeline channels consumed by the task.
set_caching_options(enable_caching[, cache_key])Sets caching options for the task.
set_cpu_request(cpu)Sets CPU request (minimum) for the task.
set_cpu_limit(cpu)Sets CPU limit (maximum) for the task.
set_accelerator_limit(limit)Sets accelerator limit (maximum) for the task.
set_memory_request(memory)Sets memory request (minimum) for the task.
set_memory_limit(memory)Sets memory limit (maximum) for the task.
set_retry(num_retries[, backoff_duration, ...])Sets task retry parameters.
add_node_selector_constraint(accelerator)Deprecated.
set_accelerator_type(accelerator)Sets accelerator type to use when executing this task.
set_display_name(name)Sets display name for the task.
set_env_variable(name, value)Sets environment variable for the task.
set_debug_pause([before, after, on_error])Enable interactive debug-pause for the pipeline task.
set_container_image(name)Sets the container image to use when executing this task.
after(*tasks)Specifies an explicit dependency on other tasks by requiring this task be executed after other tasks finish completion.
If called, the pipeline task will run when any specified upstream tasks complete, even if unsuccessful.
- property platform_spec: PlatformSpec¶
PlatformSpec for all tasks in the pipeline as task.
Only for use on tasks created from GraphComponents.
- property name: str¶
The name of the task.
Unique within its parent group.
- property inputs: Dict[str, str | int | float | bool | dict | list | PipelineChannel]¶
The inputs passed to the task.
- property output: PipelineChannel¶
The single output of the task.
Used when a task has exactly one output parameter.
- property outputs: Mapping[str, PipelineChannel]¶
The dictionary of outputs of the task.
Used when a task has more the one output or uses an
OutputPathorOutput[Artifact]type annotation.
- property dependent_tasks: List[str]¶
A list of the dependent task names.
- register_pipeline_channels(pipeline_channels: List[PipelineChannel]) None[source]¶
Registers additional pipeline channels consumed by the task.
- Parameters:
pipeline_channels – Pipeline channels to add to the task’s tracked inputs.
Channels are deduplicated by their serialized
patternso the same runtime reference is only registered once.
- set_caching_options(enable_caching: bool, cache_key: str | None = None) PipelineTask[source]¶
Sets caching options for the task.
- Parameters:
enable_caching – Whether to enable caching.
cache_key – Customized cache key for this task. The customized cache key is combined with component reference identity to prevent cross-component cache collisions. When a custom cache key is provided, all automatic inputs, base image, command, arguments, and PVC names are ignored during cache key generation. Default None.
- Returns:
Self return to allow chained setting calls.
- set_cpu_request(cpu: str | PipelineChannel) PipelineTask[source]¶
Sets CPU request (minimum) for the task.
- Parameters:
cpu – Minimum CPU requests required. This string should be a number or a number followed by an “m” to indicate millicores (1/1000). For more information, see Specify a CPU Request and a CPU Limit.
- Returns:
Self return to allow chained setting calls.
- set_cpu_limit(cpu: str | PipelineChannel) PipelineTask[source]¶
Sets CPU limit (maximum) for the task.
- Parameters:
cpu –
Maximum CPU requests allowed. This string should be a number or a number followed by an “m” to indicate millicores (1/1000). For more information, see Specify a CPU Request and a CPU Limit.
- Returns:
Self return to allow chained setting calls.
- set_accelerator_limit(limit: int | str | PipelineChannel) PipelineTask[source]¶
Sets accelerator limit (maximum) for the task. Only applies if accelerator type is also set via .set_accelerator_type().
- Parameters:
limit – Maximum number of accelerators allowed. Must be a non-negative integer.
- Returns:
Self return to allow chained setting calls.
- set_memory_request(memory: str | PipelineChannel) PipelineTask[source]¶
Sets memory request (minimum) for the task.
- Parameters:
memory – The minimum memory requests required. This string should be a number or a number followed by one of “E”, “Ei”, “P”, “Pi”, “T”, “Ti”, “G”, “Gi”, “M”, “Mi”, “K”, or “Ki”.
- Returns:
Self return to allow chained setting calls.
- set_memory_limit(memory: str | PipelineChannel) PipelineTask[source]¶
Sets memory limit (maximum) for the task.
- Parameters:
memory – The maximum memory requests allowed. This string should be a number or a number followed by one of “E”, “Ei”, “P”, “Pi”, “T”, “Ti”, “G”, “Gi”, “M”, “Mi”, “K”, or “Ki”.
- Returns:
Self return to allow chained setting calls.
- set_retry(num_retries: int, backoff_duration: str | None = None, backoff_factor: float | None = None, backoff_max_duration: str | None = None, policy: str | None = None) PipelineTask[source]¶
Sets task retry parameters.
- Parameters:
num_retries – Number of times to retry on failure.
backoff_duration – Number of seconds to wait before triggering a retry. Defaults to
'0s'(immediate retry).backoff_factor – Exponential backoff factor applied to
backoff_duration. For example, ifbackoff_duration="60"(60 seconds) andbackoff_factor=2, the first retry will happen after 60 seconds, then again after 120, 240, and so on. Defaults to2.0.backoff_max_duration – Maximum duration during which the task will be retried. Defaults to
'3600s'.policy – Controls which failure types trigger a retry. Only supported when using Argo Workflows as the pipeline execution engine. One of
'Always','OnFailure','OnError', or'OnTransientError'.'OnFailure'retries container exits, including SIGTERM (exit code 143);'OnError'retries pods that report no exit code, such as deletion or node loss;'OnTransientError'retries either when the failure message matches the controller’s transient-error pattern;'Always'retries both. Defaults toNone, which omits the policy so the deployment’s Argo configuration applies; KFP’s bundled manifests set'OnError'.
- Returns:
Self return to allow chained setting calls.
- add_node_selector_constraint(accelerator: str) PipelineTask[source]¶
Deprecated. Use
set_accelerator_type()instead.Sets accelerator type to use when executing this task.
- Parameters:
accelerator – The name of the accelerator, such as
'NVIDIA_TESLA_K80','TPU_V3','nvidia.com/gpu'or'cloud-tpus.google.com/v3'.- Returns:
Self return to allow chained setting calls.
- set_accelerator_type(accelerator: str | PipelineChannel) PipelineTask[source]¶
Sets accelerator type to use when executing this task.
- Parameters:
accelerator – The name of the accelerator, such as
'NVIDIA_TESLA_K80','TPU_V3','nvidia.com/gpu'or'cloud-tpus.google.com/v3'.- Returns:
Self return to allow chained setting calls.
- set_display_name(name: str) PipelineTask[source]¶
Sets display name for the task.
- Parameters:
name – Display name.
- Returns:
Self return to allow chained setting calls.
- set_env_variable(name: str, value: str) PipelineTask[source]¶
Sets environment variable for the task.
- Parameters:
name – Environment variable name.
value – Environment variable value.
- Returns:
Self return to allow chained setting calls.
- set_debug_pause(before: bool = False, after: bool = True, on_error: bool = False) PipelineTask[source]¶
Enable interactive debug-pause for the pipeline task.
Keeps the pod alive so you can
kubectl execinto it interactively.When enabled, Argo Workflows’ executor (the
waitcontainer) detects the correspondingARGO_DEBUG_PAUSE_*environment variable and pauses the workflow node, preventing the pod from terminating.This requires Argo Workflows 3.5.0 or later.
- Parameters:
before – If
True, pause before the main process starts. Useful for inspecting the environment, installing tools, or modifying inputs before execution.after – If
True(default), pause after the main process completes. Modified byon_error.on_error – If
True, only pause after execution when the component fails (setsARGO_DEBUG_PAUSE_ON_ERRORinstead ofARGO_DEBUG_PAUSE_AFTER). Requiresafter=True.
- Returns:
Self return to allow chained setting calls.
- Raises:
ValueError – If
after=Falseandon_error=True(contradictory).ValueError – If both
beforeandafterareFalse.
Example
@dsl.pipeline def my_pipeline(): task = my_component() task.set_debug_pause() task2 = my_component() task2.set_debug_pause(before=True, after=False) task3 = my_component() task3.set_debug_pause(on_error=True)
- set_container_image(name: str | PipelineChannel) PipelineTask[source]¶
Sets the container image to use when executing this task.
Takes precedence over the base_image specified in the @component decorator.
Unlike base_image, this method supports dynamic values such as Pipeline Parameters or outputs from previous tasks, which are resolved at runtime.
- Parameters:
name – The container image name as a static string (e.g., “python:3.11-alpine”) or a dynamic reference (e.g., a PipelineParameter instance or a task output like task.outputs[‘image_name’]).
- Returns:
Self return to allow chained setting calls.
- after(*tasks: PipelineTask | tasks_group.ExitHandler) PipelineTask[source]¶
Specifies an explicit dependency on other tasks by requiring this task be executed after other tasks finish completion.
- Parameters:
*tasks – Upstream dependencies after which this task should be executed. Each dependency must be either a
PipelineTaskor adsl.ExitHandlergroup.- Returns:
Self return to allow chained setting calls.
Example
@dsl.pipeline(name='my-pipeline') def my_pipeline(): task1 = my_component(text='1st task') task2 = my_component(text='2nd task').after(task1)
- ignore_upstream_failure() PipelineTask[source]¶
If called, the pipeline task will run when any specified upstream tasks complete, even if unsuccessful.
This method effectively turns the caller task into an exit task if the caller task has upstream dependencies.
If the task has no upstream tasks, either via data exchange or an explicit dependency via .after(), this method has no effect.
- Returns:
Self return to allow chained setting calls.
Example
@dsl.pipeline() def my_pipeline(text: str = 'message'): task = fail_op(message=text) clean_up_task = print_op( message=task.output).ignore_upstream_failure()
- class kfp.dsl.PipelineConfig(workspace: WorkspaceConfig | None = None, ttl_seconds_after_finished: int | None = None, ttl_seconds_after_success: int | None = None, ttl_seconds_after_failure: int | None = None, active_deadline_seconds: int | None = None, artifact_previews: bool = True)[source]¶
Bases:
objectPipelineConfig contains pipeline-level config options.
- workspace¶
Optional configuration for a shared workspace PVC that persists for the duration of the pipeline run.
- ttl_seconds_after_finished¶
Optional number of seconds to retain the Argo Workflow resource after the pipeline run completes, regardless of success or failure. Maps to Argo’s
ttlStrategy.secondsAfterCompletion. WhenNone(the default) no completion TTL is set.
- ttl_seconds_after_success¶
Optional number of seconds to retain the Argo Workflow resource after a successful run. Maps to Argo’s
ttlStrategy.secondsAfterSuccess. Takes precedence overttl_seconds_after_finishedfor successful runs when both are set.
- ttl_seconds_after_failure¶
Optional number of seconds to retain the Argo Workflow resource after a failed run. Maps to Argo’s
ttlStrategy.secondsAfterFailure. Takes precedence overttl_seconds_after_finishedfor failed runs when both are set.
- active_deadline_seconds¶
Optional maximum number of seconds a workflow is allowed to run before it is forcibly terminated. Maps to Argo’s
activeDeadlineSeconds. WhenNone(the default) no deadline is applied.
- artifact_previews¶
Whether supported Python components automatically preview their Dataset and Model outputs. Defaults to
True. Set toFalseto disable generation throughout this pipeline, including nested pipelines and exit handlers. Component opt-outs remain effective when the pipeline enables previews.
- class kfp.dsl.WorkspaceConfig(size: str, kubernetes: KubernetesWorkspaceConfig | None = None)[source]¶
Bases:
objectConfiguration for a shared workspace that persists during the pipeline run.
- kubernetes¶
(Optional) Kubernetes-specific configuration for the underlying PVC.
Attributes:
The size of the workspace (e.g.,
'250Gi').Methods:
set_size(size)set_kubernetes_config(kubernetes_config)- property size: str¶
The size of the workspace (e.g.,
'250Gi'). This is a required field.See the Kubernetes quantity documentation for valid quantity formats.
- set_kubernetes_config(kubernetes_config: KubernetesWorkspaceConfig)[source]¶
- class kfp.dsl.KubernetesWorkspaceConfig(pvcSpecPatch: Dict[str, Any] | None = None)[source]¶
Bases:
objectConfiguration for Kubernetes-specific workspace settings.
Use this to override the default PersistentVolumeClaim (PVC) configuration used when running pipelines on a Kubernetes cluster.
- pvcSpecPatch¶
A dictionary of fields to patch onto the default PVC spec (e.g., ‘storageClassName’, ‘accessModes’).
Methods:
set_pvcSpecPatch(patch)
- class kfp.dsl.TaskConfigField(*values)[source]¶
Bases:
IntEnumAttributes:
- RESOURCES = 1¶
- ENV = 2¶
- KUBERNETES_AFFINITY = 3¶
- KUBERNETES_TOLERATIONS = 4¶
- KUBERNETES_NODE_SELECTOR = 5¶
- KUBERNETES_VOLUMES = 6¶
- KUBERNETES_RESOURCE_CLAIMS = 7¶
- class kfp.dsl.TaskConfigPassthrough(field: kfp.dsl.component_task_config.TaskConfigField, apply_to_task: bool = False)[source]¶
Bases:
objectAttributes:
Methods:
to_proto()Converts this object to its proto representation.
- field: TaskConfigField¶
- apply_to_task: bool = False¶
- kfp.dsl.notebook_component(func: Callable[[...], Any] | None = None, *, notebook_path: str, base_image: str | None = None, packages_to_install: List[str] | None = None, output_component_file: str | None = None, pip_index_urls: List[str] | None = None, pip_trusted_hosts: List[str] | None = None, use_venv: bool = False, kfp_package_path: str | None = None, install_kfp_package: bool = True, task_config_passthroughs: List[TaskConfigPassthrough] | None = None)[source]¶
Decorator to define a Notebook-based KFP component.
- Parameters:
notebook_path – Path to the .ipynb file or a directory containing one to embed and execute.
base_image – Base container image for the component.
packages_to_install – Runtime-only packages to install inside the component container. When None, defaults to [“nbclient>=0.10,<1”, “ipykernel>=6,<7”, “jupyter_client>=7,<9”, “fastjsonschema<2.22; python_version < ‘3.10’”] to ensure the notebook can execute with a Python kernel and client. When [], installs nothing. When non-empty, installs the exact list.
output_component_file – Optional path to write the component YAML.
pip_index_urls – Optional pip index URLs for installation.
pip_trusted_hosts – Optional pip trusted hosts.
use_venv – Whether to create and use a venv inside the container.
kfp_package_path – Optional KFP package path to install.
install_kfp_package – Whether to auto-install KFP when appropriate.
task_config_passthroughs – Optional task config passthroughs.
- Parameter injection and execution:
The notebook bytes are embedded in the component. At runtime, the helper binds dsl.run_notebook(**kwargs).
Inside the decorated function body, call dsl.run_notebook(…) with keyword arguments for parameters you want available in the notebook (e.g., dsl.run_notebook(text=my_text)).
Parameters are injected following Papermill semantics: if the notebook contains a code cell tagged parameters, an overriding injected-parameters cell is inserted immediately after it; otherwise the injected cell is placed at the top of the notebook before execution.
Notebook outputs you want to expose as KFP outputs should be written by the notebook to known paths, then copied or logged by the function after dsl.run_notebook(…) returns.