kfp.dsl package

class kfp.dsl.Condition(condition)

Bases: kfp.dsl._ops_group.OpsGroup

Represents an condition group with a condition.

Example usage: ```python with Condition(param1==’pizza’):

op1 = ContainerOp(…) op2 = ContainerOp(…)

```

class kfp.dsl.ContainerOp(name: str, image: str, command: Union[str, List[str]] = None, arguments: Union[str, List[str]] = None, sidecars: List[kfp.dsl._container_op.Sidecar] = None, container_kwargs: Dict[KT, VT] = None, file_outputs: Dict[str, str] = None, output_artifact_paths: Dict[str, str] = None, is_exit_handler=False, pvolumes: Dict[str, kubernetes.client.models.v1_volume.V1Volume] = None)

Bases: kfp.dsl._container_op.BaseOp

Represents an op implemented by a container image.

Example

from kfp import dsl from kubernetes.client.models import V1EnvVar

@dsl.pipeline(

name=’foo’, description=’hello world’)

def foo_pipeline(tag: str, pull_image_policy: str):

# any attributes can be parameterized (both serialized string or actual PipelineParam) op = dsl.ContainerOp(name=’foo’,

image=’busybox:%s’ % tag, # pass in sidecars list sidecars=[dsl.Sidecar(‘print’, ‘busybox:latest’, command=’echo “hello”’)], # pass in k8s container kwargs container_kwargs={‘env’: [V1EnvVar(‘foo’, ‘bar’)]})

# set imagePullPolicy property for container with PipelineParam op.container.set_pull_image_policy(pull_image_policy)

# add sidecar with parameterized image tag # sidecar follows the argo sidecar swagger spec op.add_sidecar(dsl.Sidecar(‘redis’, ‘redis:%s’ % tag).set_image_pull_policy(‘Always’))

arguments

command

container

Container object that represents the container property in io.argoproj.workflow.v1alpha1.Template. Can be used to update the container configurations.

Example

import kfp.dsl as dsl from kubernetes.client.models import V1EnvVar

@dsl.pipeline(name=’example_pipeline’) def immediate_value_pipeline():

op1 = (dsl.ContainerOp(name=’example’, image=’nginx:alpine’)

.container

.add_env_variable(V1EnvVar(name=’HOST’, value=’foo.bar’)) .add_env_variable(V1EnvVar(name=’PORT’, value=‘80’)) .parent # return the parent ContainerOp

)

env_variables

image

class kfp.dsl.ExitHandler(exit_op: kfp.dsl._container_op.ContainerOp)

Bases: kfp.dsl._ops_group.OpsGroup

Represents an exit handler that is invoked upon exiting a group of ops.

Example usage: ```python exit_op = ContainerOp(…) with ExitHandler(exit_op):

op1 = ContainerOp(…) op2 = ContainerOp(…)

```

class kfp.dsl.PipelineParam(name: str, op_name: str = None, value: str = None, param_type: kfp.dsl._metadata.TypeMeta = <kfp.dsl._metadata.TypeMeta object>, pattern: str = None)

Bases: object

Representing a future value that is passed between pipeline components.

A PipelineParam object can be used as a pipeline function argument so that it will be a pipeline parameter that shows up in ML Pipelines system UI. It can also represent an intermediate value passed between components.

full_name

Unique name in the argo yaml for the PipelineParam

ignore_type()

ignore_type ignores the type information such that type checking would also pass

class kfp.dsl.PipelineVolume(pvc: str = None, volume: kubernetes.client.models.v1_volume.V1Volume = None, **kwargs)

Bases: kubernetes.client.models.v1_volume.V1Volume

Representing a volume that is passed between pipeline operators and is to be mounted by a ContainerOp or its inherited type.

A PipelineVolume object can be used as an extention of the pipeline function’s filesystem. It may then be passed between ContainerOps, exposing dependencies.

after(*ops)

Creates a duplicate of self with the required dependecies excluding the redundant dependenices. :param *ops: Pipeline operators to add as dependencies

class kfp.dsl.ResourceOp(k8s_resource=None, action: str = 'create', merge_strategy: str = None, success_condition: str = None, failure_condition: str = None, attribute_outputs: Dict[str, str] = None, **kwargs)

Bases: kfp.dsl._container_op.BaseOp

Represents an op which will be translated into a resource template

resource

Resource object that represents the resource property in io.argoproj.workflow.v1alpha1.Template.

class kfp.dsl.Sidecar(name: str, image: str, command: Union[str, List[str]] = None, args: Union[str, List[str]] = None, mirror_volume_mounts: bool = None, **kwargs)

Bases: kfp.dsl._container_op.Container

Represents an argo workflow sidecar (io.argoproj.workflow.v1alpha1.Sidecar) to be used in sidecars property in argo’s workflow template (io.argoproj.workflow.v1alpha1.Template).

Sidecar inherits from Container class with an addition of mirror_volume_mounts attribute (mirrorVolumeMounts property).

See https://github.com/argoproj/argo/blob/master/api/openapi-spec/swagger.json

Example

from kfp.dsl import ContainerOp, Sidecar

# creates a ContainerOp and adds a redis Sidecar op = (ContainerOp(name=’foo-op’, image=’busybox:latest’)

.add_sidecar(

Sidecar(name=’redis’, image=’redis:alpine’)))

attribute_map = {'args': 'args', 'command': 'command', 'env': 'env', 'env_from': 'envFrom', 'image': 'image', 'image_pull_policy': 'imagePullPolicy', 'lifecycle': 'lifecycle', 'liveness_probe': 'livenessProbe', 'mirror_volume_mounts': 'mirrorVolumeMounts', 'name': 'name', 'ports': 'ports', 'readiness_probe': 'readinessProbe', 'resources': 'resources', 'security_context': 'securityContext', 'stdin': 'stdin', 'stdin_once': 'stdinOnce', 'termination_message_path': 'terminationMessagePath', 'termination_message_policy': 'terminationMessagePolicy', 'tty': 'tty', 'volume_devices': 'volumeDevices', 'volume_mounts': 'volumeMounts', 'working_dir': 'workingDir'}

inputs

A list of PipelineParam found in the Sidecar object.

set_mirror_volume_mounts(mirror_volume_mounts=True)

Setting mirrorVolumeMounts to true will mount the same volumes specified in the main container to the sidecar (including artifacts), at the same mountPaths. This enables dind daemon to partially see the same filesystem as the main container in order to use features such as docker volume binding.

Parameters:mirror_volume_mounts – boolean flag

swagger_types = {'args': 'list[str]', 'command': 'list[str]', 'env': 'list[V1EnvVar]', 'env_from': 'list[V1EnvFromSource]', 'image': 'str', 'image_pull_policy': 'str', 'lifecycle': 'V1Lifecycle', 'liveness_probe': 'V1Probe', 'mirror_volume_mounts': 'bool', 'name': 'str', 'ports': 'list[V1ContainerPort]', 'readiness_probe': 'V1Probe', 'resources': 'V1ResourceRequirements', 'security_context': 'V1SecurityContext', 'stdin': 'bool', 'stdin_once': 'bool', 'termination_message_path': 'str', 'termination_message_policy': 'str', 'tty': 'bool', 'volume_devices': 'list[V1VolumeDevice]', 'volume_mounts': 'list[V1VolumeMount]', 'working_dir': 'str'}

class kfp.dsl.VolumeOp(resource_name: str = None, size: str = None, storage_class: str = None, modes: List[str] = ['ReadWriteMany'], annotations: Dict[str, str] = None, data_source=None, **kwargs)

Bases: kfp.dsl._resource_op.ResourceOp

Represents an op which will be translated into a resource template which will be creating a PVC.

class kfp.dsl.VolumeSnapshotOp(resource_name: str = None, pvc: str = None, snapshot_class: str = None, annotations: Dict[str, str] = None, volume: kubernetes.client.models.v1_volume.V1Volume = None, **kwargs)

Bases: kfp.dsl._resource_op.ResourceOp

Represents an op which will be translated into a resource template which will be creating a VolumeSnapshot.

At the time that this feature is written, VolumeSnapshots are an Alpha feature in Kubernetes. You should check with your Kubernetes Cluster admin if they have it enabled.

kfp.dsl.component(func)

Decorator for component functions that returns a ContainerOp. This is useful to enable type checking in the DSL compiler

Usage: ```python @dsl.component def foobar(model: TFModel(), step: MLStep()):

return dsl.ContainerOp()

kfp.dsl.get_pipeline_conf()

Configure the pipeline level setting to the current pipeline Note: call the function inside the user defined pipeline function.

kfp.dsl.graph_component(func)

Decorator for graph component functions. This decorator returns an ops_group.

Usage: ```python import kfp.dsl as dsl @dsl.graph_component def flip_component(flip_result):

print_flip = PrintOp(flip_result) flipA = FlipCoinOp().after(print_flip) with dsl.Condition(flipA.output == ‘heads’):

flip_component(flipA.output)

return {‘flip_result’: flipA.output}

kfp.dsl.pipeline(name, description)

Decorator of pipeline functions.

Usage: ```python @pipeline(

name=’my awesome pipeline’, description=’Is it really awesome?’

) def my_pipeline(a: PipelineParam, b: PipelineParam):

…

```

kfp.dsl.python_component(name, description=None, base_image=None, target_component_file: str = None)

Decorator for Python component functions. This decorator adds the metadata to the function object itself.

Parameters:

• name – Human-readable name of the component
• description – Optional. Description of the component
• base_image – Optional. Docker container image to use as the base of the component. Needs to have Python 3.5+ installed.
• target_component_file – Optional. Local file to store the component definition. The file can then be used for sharing.

Returns:

The same function (with some metadata fields set).

Usage: ```python @dsl.python_component(

name=’my awesome component’, description=’Come, Let’s play’, base_image=’tensorflow/tensorflow:1.11.0-py3’,

) def my_component(a: str, b: int) -> str:

…

```

• kfp.dsl.types module