kfp.components.structures package

Definitions for component spec.

class kfp.components.structures.ComponentSpec(name: str, implementation: kfp.components.structures.Implementation, description: Optional[str] = None, inputs: Optional[Dict[str, kfp.components.structures.InputSpec]] = None, outputs: Optional[Dict[str, kfp.components.structures.OutputSpec]] = None)[source]

Bases: kfp.components.base_model.BaseModel

The definition of a component.

name

The name of the component.

description

the description of the component.

Type:optional
inputs

the input definitions of the component.

Type:optional
outputs

the output definitions of the component.

Type:optional
implementation

The implementation of the component. Either an executor (container, importer) or a DAG consists of other components.

description = None
classmethod from_pipeline_spec_dict(pipeline_spec_dict: Dict[str, Any]) → kfp.components.structures.ComponentSpec[source]
classmethod from_pipeline_spec_yaml(pipeline_spec_yaml: str) → kfp.components.structures.ComponentSpec[source]

Creates a ComponentSpec from a pipeline spec in YAML format.

Parameters:component_yaml (str) – Component spec in YAML format.
Returns:The component spec object.
Return type:ComponentSpec
classmethod from_v1_component_spec(v1_component_spec: kfp.components.v1_structures.ComponentSpec) → kfp.components.structures.ComponentSpec[source]

Converts V1 ComponentSpec to V2 ComponentSpec.

Parameters:

v1_component_spec – The V1 ComponentSpec.

Returns:

Component spec in the form of V2 ComponentSpec.

Raises:
  • ValueError – If implementation is not found.
  • TypeError – if any argument is neither a str nor Dict.
inputs = None
classmethod load_from_component_yaml(component_yaml: str) → kfp.components.structures.ComponentSpec[source]

Loads V1 or V2 component yaml into ComponentSpec.

Parameters:component_yaml – the component yaml in string format.
Returns:Component spec in the form of V2 ComponentSpec.
outputs = None
save_to_component_yaml(output_file: str) → None[source]

Saves ComponentSpec into IR YAML file.

Parameters:output_file – File path to store the component yaml.
to_pipeline_spec() → pipeline_spec_pb2.PipelineSpec[source]

Creates a pipeline instance and constructs the pipeline spec for a single component.

Parameters:component_spec – The ComponentSpec to convert to PipelineSpec.
Returns:A PipelineSpec proto representing the compiled component.
transform_command_input_placeholders() → None[source]

Converts command and args elements to a placholder type where applicable.

transform_inputs() → None[source]

Use None instead of empty list for inputs.

transform_name() → None[source]

Converts the name to a valid name.

transform_outputs() → None[source]

Use None instead of empty list for outputs.

validate_placeholders()[source]

Validates that input/output placeholders refer to an existing input/output.

class kfp.components.structures.ConcatPlaceholder(items: List[Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]])[source]

Bases: kfp.components.base_model.BaseModel

Class that extends basePlaceholders for concatenation.

items

string or ValidCommandArgs for concatenation.

classmethod from_concat_string(concat_string: str) → kfp.components.structures.ConcatPlaceholder[source]

Creates a concat placeholder from an IR string indicating concatenation.

Parameters:concat_string (str) – The IR string (e.g. {{$.inputs.parameters[‘input1’]}}+{{$.inputs.parameters[‘input2’]}})
Returns:The ConcatPlaceholder instance.
Return type:ConcatPlaceholder
class kfp.components.structures.ContainerSpec(image: str, command: Optional[List[Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]]] = None, args: Optional[List[Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]]] = None, env: Optional[Mapping[str, Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]]] = None, resources: Optional[kfp.components.structures.ResourceSpec] = None)[source]

Bases: kfp.components.base_model.BaseModel

Container implementation definition.

image

The container image.

command

the container entrypoint.

Type:optional
args

the arguments to the container entrypoint.

Type:optional
env

the environment variables to be passed to the container.

Type:optional
resources

the specification on the resource requirements.

Type:optional
args = None
command = None
env = None
classmethod from_container_dict(container_dict: Dict[str, Any]) → kfp.components.structures.ContainerSpec[source]

Creates a ContainerSpec from a PipelineContainerSpec message in dict format (pipeline_spec.deploymentSpec.executors.<executor- key>.container).

Parameters:container_dict (Dict[str, Any]) – PipelineContainerSpec message in dict format.
Returns:The ContainerSpec instance.
Return type:ContainerSpec
resources = None
transform_args() → None[source]

Use None instead of empty list for args.

transform_command() → None[source]

Use None instead of empty list for command.

transform_env() → None[source]

Use None instead of empty dict for env.

class kfp.components.structures.DagSpec(tasks: Mapping[str, kfp.components.structures.TaskSpec], outputs: Mapping[str, Any])[source]

Bases: kfp.components.base_model.BaseModel

DAG(graph) implementation definition.

tasks

The tasks inside the DAG.

outputs

Defines how the outputs of the dag are linked to the sub tasks.

class kfp.components.structures.IfPresentPlaceholder(if_structure: kfp.components.structures.IfPresentPlaceholderStructure)[source]

Bases: kfp.components.base_model.BaseModel

Class that extends basePlaceholders for conditional cases.

if_present

holds structure for conditional cases.

Type:ifPresent
class kfp.components.structures.IfPresentPlaceholderStructure(input_name: str, then: List[Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]], otherwise: Optional[List[Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]]] = None)[source]

Bases: kfp.components.base_model.BaseModel

Class that holds structure for conditional cases.

input_name

name of the input/output.

then

If the input/output specified in name is present, the command-line argument will be replaced at run-time by the expanded value of then.

otherwise

If the input/output specified in name is not present, the command-line argument will be replaced at run-time by the expanded value of otherwise.

otherwise = None
transform_otherwise() → None[source]

Use None instead of empty list for optional.

class kfp.components.structures.Implementation(container: Optional[kfp.components.structures.ContainerSpec] = None, graph: Optional[kfp.components.structures.DagSpec] = None, importer: Optional[kfp.components.structures.ImporterSpec] = None)[source]

Bases: kfp.components.base_model.BaseModel

Implementation definition.

container

container implementation details.

graph

graph implementation details.

importer

importer implementation details.

container = None
classmethod from_deployment_spec_dict(deployment_spec_dict: Dict[str, Any], component_name: str) → kfp.components.structures.Implementation[source]

Creates an Implmentation object from a deployment spec message in dict format (pipeline_spec.deploymentSpec).

Parameters:
  • deployment_spec_dict (Dict[str, Any]) – PipelineDeploymentConfig message in dict format.
  • component_name (str) – The name of the component.
Returns:

An implementation object.

Return type:

Implementation

graph = None
importer = None
class kfp.components.structures.ImporterSpec(artifact_uri: str, type_schema: str, reimport: bool, metadata: Optional[Mapping[str, Any]] = None)[source]

Bases: kfp.components.base_model.BaseModel

ImporterSpec definition.

artifact_uri

The URI of the artifact.

type_schema

The type of the artifact.

reimport

Whether or not import an artifact regardless it has been imported before.

metadata

the properties of the artifact.

Type:optional
metadata = None
class kfp.components.structures.InputPathPlaceholder(input_name: str)[source]

Bases: kfp.components.base_model.BaseModel, kfp.components.structures.PlaceholderSerializationMixin

Class that holds input path for conditional cases.

input_name

name of the input.

class kfp.components.structures.InputSpec(type: Union[str, dict], default: Optional[Any] = None)[source]

Bases: kfp.components.structures.InputSpec_, kfp.components.base_model.BaseModel

Component input definitions.

type

The type of the input.

default

the default value for the input.

Type:optional
_optional

Wether the input is optional. An input is optional when it has an explicit default value.

classmethod from_ir_parameter_dict(ir_parameter_dict: Dict[str, Any]) → kfp.components.structures.InputSpec[source]

Creates an InputSpec from a ComponentInputsSpec message in dict format (pipeline_spec.components.<component- key>.inputDefinitions.parameters.<input-key>).

Parameters:ir_parameter_dict (Dict[str, Any]) – The ComponentInputsSpec message in dict format.
Returns:The InputSpec object.
Return type:InputSpec
class kfp.components.structures.InputSpec_(type: Union[str, dict], default: Optional[Any] = None)[source]

Bases: kfp.components.base_model.BaseModel

Component input definitions. (Inner class).

type

The type of the input.

default

the default value for the input.

Type:optional
description

Optional: the user description of the input.

default = None
class kfp.components.structures.InputUriPlaceholder(input_name: str)[source]

Bases: kfp.components.base_model.BaseModel, kfp.components.structures.PlaceholderSerializationMixin

Class that holds input uri for conditional cases.

input_name

name of the input.

class kfp.components.structures.InputValuePlaceholder(input_name: str)[source]

Bases: kfp.components.base_model.BaseModel, kfp.components.structures.PlaceholderSerializationMixin

Class that holds input value for conditional cases.

input_name

name of the input.

class kfp.components.structures.OutputParameterPlaceholder(output_name: str)[source]

Bases: kfp.components.base_model.BaseModel, kfp.components.structures.PlaceholderSerializationMixin

Class that holds output path for conditional cases.

output_name

name of the output.

class kfp.components.structures.OutputPathPlaceholder(output_name: str)[source]

Bases: kfp.components.base_model.BaseModel, kfp.components.structures.PlaceholderSerializationMixin

Class that holds output path for conditional cases.

output_name

name of the output.

class kfp.components.structures.OutputSpec(type: Union[str, dict])[source]

Bases: kfp.components.base_model.BaseModel

Component output definitions.

type

The type of the output.

classmethod from_ir_parameter_dict(ir_parameter_dict: Dict[str, Any]) → kfp.components.structures.OutputSpec[source]

Creates an OutputSpec from a ComponentOutputsSpec message in dict format (pipeline_spec.components.<component- key>.outputDefinitions.parameters|artifacts.<output-key>).

Parameters:ir_parameter_dict (Dict[str, Any]) – The ComponentOutputsSpec in dict format.
Returns:The OutputSpec object.
Return type:OutputSpec
class kfp.components.structures.OutputUriPlaceholder(output_name: str)[source]

Bases: kfp.components.base_model.BaseModel, kfp.components.structures.PlaceholderSerializationMixin

Class that holds output uri for conditional cases.

output_name

name of the output.

class kfp.components.structures.PlaceholderSerializationMixin[source]

Bases: object

Mixin for *Placeholder objects that handles the serialization/deserialization of the placeholder.

classmethod from_placeholder(placeholder_string: str) → P[source]

Converts a placeholder string into a placeholder object.

Parameters:placeholder_string (str) – The placeholder.
Returns:The placeholder object.
Return type:PlaceholderSerializationMixin subclass
classmethod is_match(placeholder_string: str) → bool[source]

Determines if the placeholder_string matches the placeholder pattern.

Parameters:placeholder_string (str) – The string (often “{{$.inputs/outputs…}}”) to check.
Returns:Determines if the placeholder_string matches the placeholder pattern.
Return type:bool
to_placeholder() → str[source]

Converts a placeholder object into a placeholder string.

Returns:The placeholder string.
Return type:str
class kfp.components.structures.ResourceSpec(cpu_limit: Optional[float] = None, memory_limit: Optional[float] = None, accelerator_type: Optional[str] = None, accelerator_count: Optional[int] = None)[source]

Bases: kfp.components.base_model.BaseModel

The resource requirements of a container execution.

cpu_limit

the limit of the number of vCPU cores.

Type:optional
memory_limit

the memory limit in GB.

Type:optional
accelerator_type

the type of accelerators attached to the container.

Type:optional
accelerator_count

the number of accelerators attached.

Type:optional
accelerator_count = None
accelerator_type = None
cpu_limit = None
memory_limit = None
class kfp.components.structures.TaskSpec(name: str, inputs: Mapping[str, Any], dependent_tasks: List[str], component_ref: str, trigger_condition: Optional[str] = None, trigger_strategy: Optional[str] = None, iterator_items: Optional[Any] = None, iterator_item_input: Optional[str] = None, enable_caching: bool = True, display_name: Optional[str] = None)[source]

Bases: kfp.components.base_model.BaseModel

The spec of a pipeline task.

name

The name of the task.

inputs

The sources of task inputs. Constant values or PipelineParams.

dependent_tasks

The list of upstream tasks.

component_ref

The name of a component spec this task is based on.

trigger_condition

an expression which will be evaluated into a boolean value. True to trigger the task to run.

Type:optional
trigger_strategy

when the task will be ready to be triggered. Valid values include: “TRIGGER_STRATEGY_UNSPECIFIED”, “ALL_UPSTREAM_TASKS_SUCCEEDED”, and “ALL_UPSTREAM_TASKS_COMPLETED”.

Type:optional
iterator_items

the items to iterate on. A constant value or a PipelineParam.

Type:optional
iterator_item_input

the name of the input which has the item from the [items][] collection.

Type:optional
enable_caching

whether or not to enable caching for the task. Default is True.

Type:optional
display_name

the display name of the task. If not specified, the task name will be used as the display name.

Type:optional
display_name = None
enable_caching = True
iterator_item_input = None
iterator_items = None
trigger_condition = None
trigger_strategy = None
class kfp.components.structures.TypeVar(name, *constraints, bound=None, covariant=False, contravariant=False)[source]

Bases: typing._Final, typing._Immutable

Type variable.

Usage:

T = TypeVar('T')  # Can be anything
A = TypeVar('A', str, bytes)  # Must be str or bytes

Type variables exist primarily for the benefit of static type checkers. They serve as the parameters for generic types as well as for generic function definitions. See class Generic for more information on generic types. Generic functions work as follows:

def repeat(x: T, n: int) -> List[T]:
‘’’Return a list containing n references to x.’’’ return [x]*n
def longest(x: A, y: A) -> A:
‘’’Return the longest of two strings.’’’ return x if len(x) >= len(y) else y

The latter example’s signature is essentially the overloading of (str, str) -> str and (bytes, bytes) -> bytes. Also note that if the arguments are instances of some subclass of str, the return type is still plain str.

At runtime, isinstance(x, T) and issubclass(C, T) will raise TypeError.

Type variables defined with covariant=True or contravariant=True can be used to declare covariant or contravariant generic types. See PEP 484 for more details. By default generic types are invariant in all type variables.

Type variables can be introspected. e.g.:

T.__name__ == ‘T’ T.__constraints__ == () T.__covariant__ == False T.__contravariant__ = False A.__constraints__ == (str, bytes)

Note that only type variables defined in global scope can be pickled.

kfp.components.structures.convert_str_or_dict_to_placeholder(element: Union[str, dict, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]) → Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, kfp.components.structures.IfPresentPlaceholder, kfp.components.structures.ConcatPlaceholder][source]

Converts command and args elements to a placholder type based on value of the key of the placeholder string, else returns the input.

Parameters:element (Union[str, dict, ValidCommandArgs]) – A ContainerSpec.command or ContainerSpec.args element.
Raises:TypeError – If element is invalid.
Returns:Possibly converted placeholder or original input.
Return type:Union[str, ValidCommandArgs]
kfp.components.structures.maybe_convert_command_arg_to_placeholder(arg: str) → Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, kfp.components.structures.IfPresentPlaceholder, kfp.components.structures.ConcatPlaceholder][source]

Infers if a command is a placeholder and converts it to the correct Placeholder object.

Parameters:arg (str) – The arg or command to possibly convert.
Returns:The converted command or original string.
Return type:ValidCommandArgs
kfp.components.structures.try_to_get_dict_from_string(string: str) → Union[dict, str][source]

Tries to parse a dictionary from a string if possible, else returns the string.

Parameters:string (str) – The string to parse.
Returns:The dictionary if one was successfully parsed, else the original string.
Return type:Union[dict, str]