KFP extension modules

kfp.onprem module

kfp.onprem.mount_pvc(pvc_name='pipeline-claim', volume_name='pipeline', volume_mount_path='/mnt/pipeline')

Modifier function to apply to a Container Op to simplify volume, volume mount addition and enable better reuse of volumes, volume claims across container ops. Usage:

train = train_op(…) train.apply(mount_pvc(‘claim-name’, ‘pipeline’, ‘/mnt/pipeline’))

kfp.gcp module

kfp.gcp.use_gcp_secret(secret_name='user-gcp-sa', secret_file_path_in_volume='/user-gcp-sa.json', volume_name='gcp-credentials', secret_volume_mount_path='/secret/gcp-credentials')

An operator that configures the container to use GCP service account.

The user-gcp-sa secret is created as part of the kubeflow deployment that stores the access token for kubeflow user service account.

With this service account, the container has a range of GCP APIs to access to. This service account is automatically created as part of the kubeflow deployment.

For the list of the GCP APIs this service account can access to, check https://github.com/kubeflow/kubeflow/blob/7b0db0d92d65c0746ac52b000cbc290dac7c62b1/deployment/gke/deployment_manager_configs/iam_bindings_template.yaml#L18

If you want to call the GCP APIs in a different project, grant the kf-user service account access permission.

kfp.gcp.use_tpu(tpu_cores: int, tpu_resource: str, tf_version: str)

An operator that configures GCP TPU spec in a container op.

Parameters:

• tpu_cores – Required. The number of cores of TPU resource. For example, the value can be ‘8’, ‘32’, ‘128’, etc. Check more details at: https://cloud.google.com/tpu/docs/kubernetes-engine-setup#pod-spec.
• tpu_resource – Required. The resource name of the TPU resource. For example, the value can be ‘v2’, ‘preemptible-v1’, ‘v3’ or ‘preemptible-v3’. Check more details at: https://cloud.google.com/tpu/docs/kubernetes-engine-setup#pod-spec.
• tf_version – Required. The TensorFlow version that the TPU nodes use. For example, the value can be ‘1.12’, ‘1.11’, ‘1.9’ or ‘1.8’. Check more details at: https://cloud.google.com/tpu/docs/supported-versions.

kfp.aws module

kfp.aws.use_aws_secret(secret_name='aws-secret', aws_access_key_id_name='AWS_ACCESS_KEY_ID', aws_secret_access_key_name='AWS_SECRET_ACCESS_KEY')

An operator that configures the container to use AWS credentials.

AWS doesn’t create secret along with kubeflow deployment and it requires users to manually create credential secret with proper permissions. — apiVersion: v1 kind: Secret metadata:

name: aws-secret

type: Opaque data:

AWS_ACCESS_KEY_ID: BASE64_YOUR_AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY: BASE64_YOUR_AWS_SECRET_ACCESS_KEY

kfp.azure module

kfp.azure.use_azure_secret(secret_name='azcreds')

An operator that configures the container to use Azure user credentials.

The azcreds secret is created as part of the kubeflow deployment that stores the client ID and secrets for the kubeflow azure service principal.

With this service principal, the container has a range of Azure APIs to access to.