kfp.components.structures package¶
Definitions for component spec.
-
class
kfp.components.structures.
ComponentSpec
(name: str, implementation: kfp.components.structures.Implementation, description: Optional[str] = None, inputs: Optional[Dict[str, kfp.components.structures.InputSpec]] = None, outputs: Optional[Dict[str, kfp.components.structures.OutputSpec]] = None)[source]¶ Bases:
kfp.components.base_model.BaseModel
The definition of a component.
-
name
¶ The name of the component.
-
description
¶ the description of the component.
Type: optional
-
inputs
¶ the input definitions of the component.
Type: optional
-
outputs
¶ the output definitions of the component.
Type: optional
-
implementation
¶ The implementation of the component. Either an executor (container, importer) or a DAG consists of other components.
-
description
= None
-
classmethod
from_pipeline_spec_dict
(pipeline_spec_dict: Dict[str, Any]) → kfp.components.structures.ComponentSpec[source]¶
-
classmethod
from_pipeline_spec_yaml
(pipeline_spec_yaml: str) → kfp.components.structures.ComponentSpec[source]¶ Creates a ComponentSpec from a pipeline spec in YAML format.
Parameters: component_yaml (str) – Component spec in YAML format. Returns: The component spec object. Return type: ComponentSpec
-
classmethod
from_v1_component_spec
(v1_component_spec: kfp.components.v1_structures.ComponentSpec) → kfp.components.structures.ComponentSpec[source]¶ Converts V1 ComponentSpec to V2 ComponentSpec.
Parameters: v1_component_spec – The V1 ComponentSpec.
Returns: Component spec in the form of V2 ComponentSpec.
Raises: ValueError
– If implementation is not found.TypeError
– if any argument is neither a str nor Dict.
-
inputs
= None
-
classmethod
load_from_component_yaml
(component_yaml: str) → kfp.components.structures.ComponentSpec[source]¶ Loads V1 or V2 component yaml into ComponentSpec.
Parameters: component_yaml – the component yaml in string format. Returns: Component spec in the form of V2 ComponentSpec.
-
outputs
= None
-
save_to_component_yaml
(output_file: str) → None[source]¶ Saves ComponentSpec into IR YAML file.
Parameters: output_file – File path to store the component yaml.
-
to_pipeline_spec
() → pipeline_spec_pb2.PipelineSpec[source]¶ Creates a pipeline instance and constructs the pipeline spec for a single component.
Parameters: component_spec – The ComponentSpec to convert to PipelineSpec. Returns: A PipelineSpec proto representing the compiled component.
-
-
class
kfp.components.structures.
ConcatPlaceholder
(items: List[Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]])[source]¶ Bases:
kfp.components.base_model.BaseModel
Class that extends basePlaceholders for concatenation.
-
items
¶ string or ValidCommandArgs for concatenation.
-
classmethod
from_concat_string
(concat_string: str) → kfp.components.structures.ConcatPlaceholder[source]¶ Creates a concat placeholder from an IR string indicating concatenation.
Parameters: concat_string (str) – The IR string (e.g. {{$.inputs.parameters[‘input1’]}}+{{$.inputs.parameters[‘input2’]}}) Returns: The ConcatPlaceholder instance. Return type: ConcatPlaceholder
-
-
class
kfp.components.structures.
ContainerSpec
(image: str, command: Optional[List[Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]]] = None, args: Optional[List[Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]]] = None, env: Optional[Mapping[str, Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]]] = None, resources: Optional[kfp.components.structures.ResourceSpec] = None)[source]¶ Bases:
kfp.components.base_model.BaseModel
Container implementation definition.
-
image
¶ The container image.
-
command
¶ the container entrypoint.
Type: optional
-
args
¶ the arguments to the container entrypoint.
Type: optional
-
env
¶ the environment variables to be passed to the container.
Type: optional
-
resources
¶ the specification on the resource requirements.
Type: optional
-
args
= None
-
command
= None
-
env
= None
-
classmethod
from_container_dict
(container_dict: Dict[str, Any]) → kfp.components.structures.ContainerSpec[source]¶ Creates a ContainerSpec from a PipelineContainerSpec message in dict format (pipeline_spec.deploymentSpec.executors.<executor- key>.container).
Parameters: container_dict (Dict[str, Any]) – PipelineContainerSpec message in dict format. Returns: The ContainerSpec instance. Return type: ContainerSpec
-
resources
= None
-
-
class
kfp.components.structures.
DagSpec
(tasks: Mapping[str, kfp.components.structures.TaskSpec], outputs: Mapping[str, Any])[source]¶ Bases:
kfp.components.base_model.BaseModel
DAG(graph) implementation definition.
-
tasks
¶ The tasks inside the DAG.
-
outputs
¶ Defines how the outputs of the dag are linked to the sub tasks.
-
-
class
kfp.components.structures.
IfPresentPlaceholder
(if_structure: kfp.components.structures.IfPresentPlaceholderStructure)[source]¶ Bases:
kfp.components.base_model.BaseModel
Class that extends basePlaceholders for conditional cases.
-
if_present
¶ holds structure for conditional cases.
Type: ifPresent
-
-
class
kfp.components.structures.
IfPresentPlaceholderStructure
(input_name: str, then: List[Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]], otherwise: Optional[List[Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]]] = None)[source]¶ Bases:
kfp.components.base_model.BaseModel
Class that holds structure for conditional cases.
-
input_name
¶ name of the input/output.
-
then
¶ If the input/output specified in name is present, the command-line argument will be replaced at run-time by the expanded value of then.
-
otherwise
¶ If the input/output specified in name is not present, the command-line argument will be replaced at run-time by the expanded value of otherwise.
-
otherwise
= None
-
-
class
kfp.components.structures.
Implementation
(container: Optional[kfp.components.structures.ContainerSpec] = None, graph: Optional[kfp.components.structures.DagSpec] = None, importer: Optional[kfp.components.structures.ImporterSpec] = None)[source]¶ Bases:
kfp.components.base_model.BaseModel
Implementation definition.
-
container
¶ container implementation details.
-
graph
¶ graph implementation details.
-
importer
¶ importer implementation details.
-
container
= None
-
classmethod
from_deployment_spec_dict
(deployment_spec_dict: Dict[str, Any], component_name: str) → kfp.components.structures.Implementation[source]¶ Creates an Implmentation object from a deployment spec message in dict format (pipeline_spec.deploymentSpec).
Parameters: - deployment_spec_dict (Dict[str, Any]) – PipelineDeploymentConfig message in dict format.
- component_name (str) – The name of the component.
Returns: An implementation object.
Return type:
-
graph
= None
-
importer
= None
-
-
class
kfp.components.structures.
ImporterSpec
(artifact_uri: str, type_schema: str, reimport: bool, metadata: Optional[Mapping[str, Any]] = None)[source]¶ Bases:
kfp.components.base_model.BaseModel
ImporterSpec definition.
-
artifact_uri
¶ The URI of the artifact.
-
type_schema
¶ The type of the artifact.
-
reimport
¶ Whether or not import an artifact regardless it has been imported before.
-
metadata
¶ the properties of the artifact.
Type: optional
-
metadata
= None
-
-
class
kfp.components.structures.
InputPathPlaceholder
(input_name: str)[source]¶ Bases:
kfp.components.base_model.BaseModel
,kfp.components.structures.PlaceholderSerializationMixin
Class that holds input path for conditional cases.
-
input_name
¶ name of the input.
-
-
class
kfp.components.structures.
InputSpec
(type: Union[str, dict], default: Optional[Any] = None)[source]¶ Bases:
kfp.components.structures.InputSpec_
,kfp.components.base_model.BaseModel
Component input definitions.
-
type
¶ The type of the input.
-
default
¶ the default value for the input.
Type: optional
-
_optional
¶ Wether the input is optional. An input is optional when it has an explicit default value.
-
classmethod
from_ir_parameter_dict
(ir_parameter_dict: Dict[str, Any]) → kfp.components.structures.InputSpec[source]¶ Creates an InputSpec from a ComponentInputsSpec message in dict format (pipeline_spec.components.<component- key>.inputDefinitions.parameters.<input-key>).
Parameters: ir_parameter_dict (Dict[str, Any]) – The ComponentInputsSpec message in dict format. Returns: The InputSpec object. Return type: InputSpec
-
-
class
kfp.components.structures.
InputSpec_
(type: Union[str, dict], default: Optional[Any] = None)[source]¶ Bases:
kfp.components.base_model.BaseModel
Component input definitions. (Inner class).
-
type
¶ The type of the input.
-
default
¶ the default value for the input.
Type: optional
-
description
¶ Optional: the user description of the input.
-
default
= None
-
-
class
kfp.components.structures.
InputUriPlaceholder
(input_name: str)[source]¶ Bases:
kfp.components.base_model.BaseModel
,kfp.components.structures.PlaceholderSerializationMixin
Class that holds input uri for conditional cases.
-
input_name
¶ name of the input.
-
-
class
kfp.components.structures.
InputValuePlaceholder
(input_name: str)[source]¶ Bases:
kfp.components.base_model.BaseModel
,kfp.components.structures.PlaceholderSerializationMixin
Class that holds input value for conditional cases.
-
input_name
¶ name of the input.
-
-
class
kfp.components.structures.
OutputParameterPlaceholder
(output_name: str)[source]¶ Bases:
kfp.components.base_model.BaseModel
,kfp.components.structures.PlaceholderSerializationMixin
Class that holds output path for conditional cases.
-
output_name
¶ name of the output.
-
-
class
kfp.components.structures.
OutputPathPlaceholder
(output_name: str)[source]¶ Bases:
kfp.components.base_model.BaseModel
,kfp.components.structures.PlaceholderSerializationMixin
Class that holds output path for conditional cases.
-
output_name
¶ name of the output.
-
-
class
kfp.components.structures.
OutputSpec
(type: Union[str, dict])[source]¶ Bases:
kfp.components.base_model.BaseModel
Component output definitions.
-
type
¶ The type of the output.
-
classmethod
from_ir_parameter_dict
(ir_parameter_dict: Dict[str, Any]) → kfp.components.structures.OutputSpec[source]¶ Creates an OutputSpec from a ComponentOutputsSpec message in dict format (pipeline_spec.components.<component- key>.outputDefinitions.parameters|artifacts.<output-key>).
Parameters: ir_parameter_dict (Dict[str, Any]) – The ComponentOutputsSpec in dict format. Returns: The OutputSpec object. Return type: OutputSpec
-
-
class
kfp.components.structures.
OutputUriPlaceholder
(output_name: str)[source]¶ Bases:
kfp.components.base_model.BaseModel
,kfp.components.structures.PlaceholderSerializationMixin
Class that holds output uri for conditional cases.
-
output_name
¶ name of the output.
-
-
class
kfp.components.structures.
PlaceholderSerializationMixin
[source]¶ Bases:
object
Mixin for *Placeholder objects that handles the serialization/deserialization of the placeholder.
-
classmethod
from_placeholder
(placeholder_string: str) → P[source]¶ Converts a placeholder string into a placeholder object.
Parameters: placeholder_string (str) – The placeholder. Returns: The placeholder object. Return type: PlaceholderSerializationMixin subclass
-
classmethod
is_match
(placeholder_string: str) → bool[source]¶ Determines if the placeholder_string matches the placeholder pattern.
Parameters: placeholder_string (str) – The string (often “{{$.inputs/outputs…}}”) to check. Returns: Determines if the placeholder_string matches the placeholder pattern. Return type: bool
-
classmethod
-
class
kfp.components.structures.
ResourceSpec
(cpu_limit: Optional[float] = None, memory_limit: Optional[float] = None, accelerator_type: Optional[str] = None, accelerator_count: Optional[int] = None)[source]¶ Bases:
kfp.components.base_model.BaseModel
The resource requirements of a container execution.
-
cpu_limit
¶ the limit of the number of vCPU cores.
Type: optional
-
memory_limit
¶ the memory limit in GB.
Type: optional
-
accelerator_type
¶ the type of accelerators attached to the container.
Type: optional
-
accelerator_count
¶ the number of accelerators attached.
Type: optional
-
accelerator_count
= None
-
accelerator_type
= None
-
cpu_limit
= None
-
memory_limit
= None
-
-
class
kfp.components.structures.
TaskSpec
(name: str, inputs: Mapping[str, Any], dependent_tasks: List[str], component_ref: str, trigger_condition: Optional[str] = None, trigger_strategy: Optional[str] = None, iterator_items: Optional[Any] = None, iterator_item_input: Optional[str] = None, enable_caching: bool = True, display_name: Optional[str] = None)[source]¶ Bases:
kfp.components.base_model.BaseModel
The spec of a pipeline task.
-
name
¶ The name of the task.
-
inputs
¶ The sources of task inputs. Constant values or PipelineParams.
-
dependent_tasks
¶ The list of upstream tasks.
-
component_ref
¶ The name of a component spec this task is based on.
-
trigger_condition
¶ an expression which will be evaluated into a boolean value. True to trigger the task to run.
Type: optional
-
trigger_strategy
¶ when the task will be ready to be triggered. Valid values include: “TRIGGER_STRATEGY_UNSPECIFIED”, “ALL_UPSTREAM_TASKS_SUCCEEDED”, and “ALL_UPSTREAM_TASKS_COMPLETED”.
Type: optional
-
iterator_items
¶ the items to iterate on. A constant value or a PipelineParam.
Type: optional
-
iterator_item_input
¶ the name of the input which has the item from the [items][] collection.
Type: optional
-
enable_caching
¶ whether or not to enable caching for the task. Default is True.
Type: optional
-
display_name
¶ the display name of the task. If not specified, the task name will be used as the display name.
Type: optional
-
display_name
= None
-
enable_caching
= True
-
iterator_item_input
= None
-
iterator_items
= None
-
trigger_condition
= None
-
trigger_strategy
= None
-
-
class
kfp.components.structures.
TypeVar
(name, *constraints, bound=None, covariant=False, contravariant=False)[source]¶ Bases:
typing._Final
,typing._Immutable
Type variable.
Usage:
T = TypeVar('T') # Can be anything A = TypeVar('A', str, bytes) # Must be str or bytes
Type variables exist primarily for the benefit of static type checkers. They serve as the parameters for generic types as well as for generic function definitions. See class Generic for more information on generic types. Generic functions work as follows:
- def repeat(x: T, n: int) -> List[T]:
- ‘’’Return a list containing n references to x.’’’ return [x]*n
- def longest(x: A, y: A) -> A:
- ‘’’Return the longest of two strings.’’’ return x if len(x) >= len(y) else y
The latter example’s signature is essentially the overloading of (str, str) -> str and (bytes, bytes) -> bytes. Also note that if the arguments are instances of some subclass of str, the return type is still plain str.
At runtime, isinstance(x, T) and issubclass(C, T) will raise TypeError.
Type variables defined with covariant=True or contravariant=True can be used to declare covariant or contravariant generic types. See PEP 484 for more details. By default generic types are invariant in all type variables.
Type variables can be introspected. e.g.:
T.__name__ == ‘T’ T.__constraints__ == () T.__covariant__ == False T.__contravariant__ = False A.__constraints__ == (str, bytes)Note that only type variables defined in global scope can be pickled.
-
kfp.components.structures.
convert_str_or_dict_to_placeholder
(element: Union[str, dict, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, IfPresentPlaceholder, ConcatPlaceholder]) → Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, kfp.components.structures.IfPresentPlaceholder, kfp.components.structures.ConcatPlaceholder][source]¶ Converts command and args elements to a placholder type based on value of the key of the placeholder string, else returns the input.
Parameters: element (Union[str, dict, ValidCommandArgs]) – A ContainerSpec.command or ContainerSpec.args element. Raises: TypeError
– If element is invalid.Returns: Possibly converted placeholder or original input. Return type: Union[str, ValidCommandArgs]
-
kfp.components.structures.
maybe_convert_command_arg_to_placeholder
(arg: str) → Union[str, kfp.components.structures.InputValuePlaceholder, kfp.components.structures.InputPathPlaceholder, kfp.components.structures.InputUriPlaceholder, kfp.components.structures.OutputPathPlaceholder, kfp.components.structures.OutputUriPlaceholder, kfp.components.structures.OutputParameterPlaceholder, kfp.components.structures.IfPresentPlaceholder, kfp.components.structures.ConcatPlaceholder][source]¶ Infers if a command is a placeholder and converts it to the correct Placeholder object.
Parameters: arg (str) – The arg or command to possibly convert. Returns: The converted command or original string. Return type: ValidCommandArgs
-
kfp.components.structures.
try_to_get_dict_from_string
(string: str) → Union[dict, str][source]¶ Tries to parse a dictionary from a string if possible, else returns the string.
Parameters: string (str) – The string to parse. Returns: The dictionary if one was successfully parsed, else the original string. Return type: Union[dict, str]