KFP extension modules¶
kfp.onprem module¶
-
kfp.onprem.
mount_pvc
(pvc_name='pipeline-claim', volume_name='pipeline', volume_mount_path='/mnt/pipeline')[source]¶ Modifier function to apply to a Container Op to simplify volume, volume mount addition and enable better reuse of volumes, volume claims across container ops.
Example
train = train_op(...) train.apply(mount_pvc('claim-name', 'pipeline', '/mnt/pipeline'))
kfp.gcp module¶
Extension module for KFP on GCP deployment.
-
kfp.gcp.
add_gpu_toleration
(toleration: kubernetes.client.models.v1_toleration.V1Toleration = {'effect': 'NoSchedule', 'key': 'nvidia.com/gpu', 'operator': 'Equal', 'toleration_seconds': None, 'value': 'true'})[source]¶ An operator that configures the GKE GPU nodes in a container op.
Parameters: toleration – toleration to pods, default is the nvidia.com/gpu label.
-
kfp.gcp.
use_gcp_secret
(secret_name='user-gcp-sa', secret_file_path_in_volume=None, volume_name=None, secret_volume_mount_path='/secret/gcp-credentials')[source]¶ An operator that configures the container to use GCP service account by service account key stored in a Kubernetes secret.
For cluster setup and alternatives to using service account key, check https://www.kubeflow.org/docs/gke/authentication-pipelines/.
-
kfp.gcp.
use_preemptible_nodepool
(toleration: kubernetes.client.models.v1_toleration.V1Toleration = {'effect': 'NoSchedule', 'key': 'preemptible', 'operator': 'Equal', 'toleration_seconds': None, 'value': 'true'}, hard_constraint: bool = False)[source]¶ An operator that configures the GKE preemptible in a container op.
Parameters: - toleration – toleration to pods, default is the preemptible label.
- hard_constraint – the constraint of scheduling the pods on preemptible nodepools is hard. (Default: False)
-
kfp.gcp.
use_tpu
(tpu_cores: int, tpu_resource: str, tf_version: str)[source]¶ An operator that configures GCP TPU spec in a container op.
Parameters: - tpu_cores – Required. The number of cores of TPU resource. For example, the value can be ‘8’, ‘32’, ‘128’, etc. Check more details at: https://cloud.google.com/tpu/docs/kubernetes-engine-setup#pod-spec.
- tpu_resource – Required. The resource name of the TPU resource. For example, the value can be ‘v2’, ‘preemptible-v1’, ‘v3’ or ‘preemptible-v3’. Check more details at: https://cloud.google.com/tpu/docs/kubernetes-engine-setup#pod-spec.
- tf_version – Required. The TensorFlow version that the TPU nodes use. For example, the value can be ‘1.12’, ‘1.11’, ‘1.9’ or ‘1.8’. Check more details at: https://cloud.google.com/tpu/docs/supported-versions.
kfp.aws module¶
-
kfp.aws.
use_aws_secret
(secret_name='aws-secret', aws_access_key_id_name='AWS_ACCESS_KEY_ID', aws_secret_access_key_name='AWS_SECRET_ACCESS_KEY', aws_region=None)[source]¶ An operator that configures the container to use AWS credentials.
AWS doesn’t create secret along with kubeflow deployment and it requires users to manually create credential secret with proper permissions.
apiVersion: v1 kind: Secret metadata: name: aws-secret type: Opaque data: AWS_ACCESS_KEY_ID: BASE64_YOUR_AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY: BASE64_YOUR_AWS_SECRET_ACCESS_KEY
kfp.azure module¶
-
kfp.azure.
use_azure_secret
(secret_name='azcreds')[source]¶ An operator that configures the container to use Azure user credentials.
The azcreds secret is created as part of the kubeflow deployment that stores the client ID and secrets for the kubeflow azure service principal.
With this service principal, the container has a range of Azure APIs to access to.