kfp.compiler package

class kfp.compiler.Compiler

Bases: object

DSL Compiler that compiles pipeline functions into workflow yaml.

Example

How to use the compiler to construct workflow yaml:

@dsl.pipeline(
  name='name',
  description='description'
)
def my_pipeline(a: int = 1, b: str = "default value"):
  ...

Compiler().compile(my_pipeline, 'path/to/workflow.yaml')

compile(pipeline_func, package_path, type_check=True, pipeline_conf: kfp.dsl._pipeline.PipelineConf = None)

Compile the given pipeline function into workflow yaml.

Parameters:

• pipeline_func – Pipeline functions with @dsl.pipeline decorator.
• package_path – The output workflow tar.gz file path. for example, “~/a.tar.gz”
• type_check – Whether to enable the type check or not, default: False.
• pipeline_conf – PipelineConf instance. Can specify op transforms, image pull secrets and other pipeline-level configuration options. Overrides any configuration that may be set by the pipeline.

create_workflow(pipeline_func: Callable, pipeline_name: str = None, pipeline_description: str = None, params_list: List[kfp.dsl._pipeline_param.PipelineParam] = None, pipeline_conf: kfp.dsl._pipeline.PipelineConf = None) → Dict[str, Any]

Create workflow spec from pipeline function and specified pipeline params/metadata. Currently, the pipeline params are either specified in the signature of the pipeline function or by passing a list of dsl.PipelineParam. Conflict will cause ValueError.

Parameters:

• pipeline_func – Pipeline function where ContainerOps are invoked.
• pipeline_name – The name of the pipeline to compile.
• pipeline_description – The description of the pipeline.
• params_list – List of pipeline params to append to the pipeline.
• pipeline_conf – PipelineConf instance. Can specify op transforms, image pull secrets and other pipeline-level configuration options. Overrides any configuration that may be set by the pipeline.

Returns:

The created workflow dictionary.

get_arguments_for_sub_group(sub_group: Union[kfp.dsl._ops_group.OpsGroup, kfp.dsl._container_op.BaseOp], is_recursive_subgroup: Optional[bool], inputs: Dict[str, Tuple[str, str]])

class kfp.compiler.VersionedDependency(name, version=None, min_version=None, max_version=None)

Bases: object

DependencyVersion specifies the versions

has_max_version()

has_min_version()

has_versions()

max_version

min_version

name

kfp.compiler.build_docker_image(staging_gcs_path, target_image, dockerfile_path, timeout=600, namespace=None)

build_docker_image automatically builds a container image based on the specification in the dockerfile and

pushes to the target_image.

Args:

staging_gcs_path (str): GCS blob that can store temporary build files target_image (str): gcr path to push the final image dockerfile_path (str): local path to the dockerfile timeout (int): the timeout for the image build(in secs), default is 600 seconds namespace (str): the namespace within which to run the kubernetes kaniko job. Default is None. If the job is running on GKE and value is None the underlying functions will use the default namespace from GKE.

Deprecated since version 0.1.32: build_docker_image is deprecated. Use kfp.containers.build_image_from_working_dir instead.

kfp.compiler.build_python_component(component_func, target_image, base_image=None, dependency=[], staging_gcs_path=None, timeout=600, namespace=None, target_component_file=None, python_version='python3')

build_component automatically builds a container image for the component_func based on the base_image and pushes to the target_image.

Args:

component_func (python function): The python function to build components upon base_image (str): Docker image to use as a base image target_image (str): Full URI to push the target image staging_gcs_path (str): GCS blob that can store temporary build files target_image (str): target image path timeout (int): the timeout for the image build(in secs), default is 600 seconds namespace (str): the namespace within which to run the kubernetes kaniko job. If the job is running on GKE and value is None the underlying functions will use the default namespace from GKE. . dependency (list): a list of VersionedDependency, which includes the package name and versions, default is empty python_version (str): choose python2 or python3, default is python3

Raises:

ValueError: The function is not decorated with python_component decorator or the python_version is neither python2 nor python3

Deprecated since version 0.1.32: build_python_component is deprecated. Use kfp.containers.build_image_from_working_dir + kfp.components.func_to_container_op instead.