KFP extension modules

kfp.onprem module

kfp.onprem.add_default_resource_spec(memory_limit: Optional[str] = None, cpu_limit: Optional[str] = None, memory_request: Optional[str] = None, cpu_request: Optional[str] = None)

Add default resource requests and limits.

For resource units, refer to https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes.

Parameters:

• memory_limit – optional, memory limit. Format can be 512Mi, 2Gi etc.
• cpu_limit – optional, cpu limit. Format can be 0.5, 500m etc.
• memory_request – optional, defaults to memory limit.
• cpu_request – optional, defaults to cpu limit.

kfp.onprem.mount_pvc(pvc_name='pipeline-claim', volume_name='pipeline', volume_mount_path='/mnt/pipeline')

Modifier function to apply to a Container Op to simplify volume, volume mount addition and enable better reuse of volumes, volume claims across container ops.

Example

train = train_op(...)
train.apply(mount_pvc('claim-name', 'pipeline', '/mnt/pipeline'))

kfp.onprem.use_k8s_secret(secret_name: str = 'k8s-secret', k8s_secret_key_to_env: Optional[Dict[KT, VT]] = None)

An operator that configures the container to use k8s credentials.

k8s_secret_key_to_env specifies a mapping from the name of the keys in the k8s secret to the name of the environment variables where the values will be added.

The secret needs to be deployed manually a priori.

Example

train = train_op(...)
train.apply(use_k8s_secret(secret_name='s3-secret',
k8s_secret_key_to_env={'secret_key': 'AWS_SECRET_ACCESS_KEY'}))

This will load the value in secret ‘s3-secret’ at key ‘secret_key’ and source it as the environment variable ‘AWS_SECRET_ACCESS_KEY’. I.e. it will produce the following section on the pod: env: - name: AWS_SECRET_ACCESS_KEY

valueFrom:

secretKeyRef:

name: s3-secret key: secret_key

kfp.gcp module

Extension module for KFP on GCP deployment.

kfp.gcp.add_gpu_toleration(toleration: kubernetes.client.models.v1_toleration.V1Toleration = {'effect': 'NoSchedule', 'key': 'nvidia.com/gpu', 'operator': 'Equal', 'toleration_seconds': None, 'value': 'true'})

An operator that configures the GKE GPU nodes in a container op.

Parameters:toleration – toleration to pods, default is the nvidia.com/gpu label.

kfp.gcp.use_gcp_secret(secret_name='user-gcp-sa', secret_file_path_in_volume=None, volume_name=None, secret_volume_mount_path='/secret/gcp-credentials')

An operator that configures the container to use GCP service account by service account key stored in a Kubernetes secret.

For cluster setup and alternatives to using service account key, check https://www.kubeflow.org/docs/gke/authentication-pipelines/.

kfp.gcp.use_preemptible_nodepool(toleration: kubernetes.client.models.v1_toleration.V1Toleration = {'effect': 'NoSchedule', 'key': 'preemptible', 'operator': 'Equal', 'toleration_seconds': None, 'value': 'true'}, hard_constraint: bool = False)

An operator that configures the GKE preemptible in a container op.

Parameters:

• toleration – toleration to pods, default is the preemptible label.
• hard_constraint – the constraint of scheduling the pods on preemptible nodepools is hard. (Default: False)

kfp.gcp.use_tpu(tpu_cores: int, tpu_resource: str, tf_version: str)

An operator that configures GCP TPU spec in a container op.

Parameters:

• tpu_cores – Required. The number of cores of TPU resource. For example, the value can be ‘8’, ‘32’, ‘128’, etc. Check more details at: https://cloud.google.com/tpu/docs/kubernetes-engine-setup#pod-spec.
• tpu_resource – Required. The resource name of the TPU resource. For example, the value can be ‘v2’, ‘preemptible-v1’, ‘v3’ or ‘preemptible-v3’. Check more details at: https://cloud.google.com/tpu/docs/kubernetes-engine-setup#pod-spec.
• tf_version – Required. The TensorFlow version that the TPU nodes use. For example, the value can be ‘1.12’, ‘1.11’, ‘1.9’ or ‘1.8’. Check more details at: https://cloud.google.com/tpu/docs/supported-versions.

kfp.aws module

kfp.aws.use_aws_secret(secret_name='aws-secret', aws_access_key_id_name='AWS_ACCESS_KEY_ID', aws_secret_access_key_name='AWS_SECRET_ACCESS_KEY', aws_region=None)

An operator that configures the container to use AWS credentials.

AWS doesn’t create secret along with kubeflow deployment and it requires users to manually create credential secret with proper permissions.

apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
type: Opaque
data:
  AWS_ACCESS_KEY_ID: BASE64_YOUR_AWS_ACCESS_KEY_ID
  AWS_SECRET_ACCESS_KEY: BASE64_YOUR_AWS_SECRET_ACCESS_KEY

kfp.azure module

kfp.azure.use_azure_secret(secret_name='azcreds')

An operator that configures the container to use Azure user credentials.

The azcreds secret is created as part of the kubeflow deployment that stores the client ID and secrets for the kubeflow azure service principal.

With this service principal, the container has a range of Azure APIs to access to.