kfp.compiler package

class kfp.compiler.Compiler

Bases: object

DSL Compiler that compiles pipeline functions into workflow yaml.

Example

How to use the compiler to construct workflow yaml:

@dsl.pipeline(
  name='name',
  description='description'
)
def my_pipeline(a: int = 1, b: str = "default value"):
  ...

Compiler().compile(my_pipeline, 'path/to/workflow.yaml')

compile(pipeline_func, package_path, type_check=True, pipeline_conf: kfp.dsl._pipeline.PipelineConf = None)

Compile the given pipeline function into workflow yaml.

Parameters:

• pipeline_func – Pipeline functions with @dsl.pipeline decorator.
• package_path – The output workflow tar.gz file path. for example, “~/a.tar.gz”
• type_check – Whether to enable the type check or not, default: True.
• pipeline_conf – PipelineConf instance. Can specify op transforms, image pull secrets and other pipeline-level configuration options. Overrides any configuration that may be set by the pipeline.

create_workflow(pipeline_func: Callable, pipeline_name: str = None, pipeline_description: str = None, params_list: List[kfp.dsl._pipeline_param.PipelineParam] = None, pipeline_conf: kfp.dsl._pipeline.PipelineConf = None) → Dict[str, Any]

Create workflow spec from pipeline function and specified pipeline params/metadata. Currently, the pipeline params are either specified in the signature of the pipeline function or by passing a list of dsl.PipelineParam. Conflict will cause ValueError.

Parameters:

• pipeline_func – Pipeline function where ContainerOps are invoked.
• pipeline_name – The name of the pipeline to compile.
• pipeline_description – The description of the pipeline.
• params_list – List of pipeline params to append to the pipeline.
• pipeline_conf – PipelineConf instance. Can specify op transforms, image pull secrets and other pipeline-level configuration options. Overrides any configuration that may be set by the pipeline.

Returns:

The created workflow dictionary.

get_arguments_for_sub_group(sub_group: Union[kfp.dsl._ops_group.OpsGroup, kfp.dsl._container_op.BaseOp], is_recursive_subgroup: Optional[bool], inputs: Dict[str, Tuple[str, str]])

class kfp.compiler.VersionedDependency(name, version=None, min_version=None, max_version=None)

Bases: object

DependencyVersion specifies the versions

has_max_version()

has_min_version()

has_versions()

max_version

min_version

name

kfp.compiler.build_docker_image(staging_gcs_path, target_image, dockerfile_path, timeout=600, namespace=None)

build_docker_image automatically builds a container image based on the specification in the dockerfile and

pushes to the target_image.

Args:

staging_gcs_path (str): GCS blob that can store temporary build files target_image (str): gcr path to push the final image dockerfile_path (str): local path to the dockerfile timeout (int): the timeout for the image build(in secs), default is 600 seconds namespace (str): the namespace within which to run the kubernetes kaniko job. Default is None. If the job is running on GKE and value is None the underlying functions will use the default namespace from GKE.

Deprecated since version 0.1.32: build_docker_image is deprecated. Use kfp.containers.build_image_from_working_dir instead.

kfp.compiler.build_python_component(component_func: Callable, target_image: str, base_image: Optional[str] = None, dependency: Optional[List[kfp.containers._component_builder.VersionedDependency]] = None, staging_gcs_path: Optional[str] = None, timeout: int = 600, namespace: Optional[str] = None, target_component_file: Optional[str] = None, is_v2: bool = False)

build_component automatically builds a container image for the component_func based on the base_image and pushes to the target_image.

Parameters:

• component_func (python function) – The python function to build components upon.
• base_image (str) – Docker image to use as a base image.
• target_image (str) – Full URI to push the target image.
• staging_gcs_path (str) – GCS blob that can store temporary build files.
• target_image – The target image path.
• timeout (int) – The timeout for the image build(in secs), default is 600 seconds.
• namespace (str) – The namespace within which to run the kubernetes Kaniko job. If the job is running on GKE and value is None the underlying functions will use the default namespace from GKE.
• dependency (list) – The list of VersionedDependency, which includes the package name and versions, default is empty.
• target_component_file (str) – The path to save the generated component YAML spec.
• is_v2 – Whether or not generating a v2 KFP component, default is false.

Raises:

ValueError – The function is not decorated with python_component decorator or the python_version is neither python2 nor python3