Pipeline Anatomy

This document clarifies the anatomy of a single-component pipeline. Complexity sometimes discourages scrutiny. My hope is that improved clarity will pave the way for consolidation and simplification.

Flowchart

        graph TD

a(["entrypoint (DAG)"]) --> b[root-driver]
b --> c(["root (DAG)"])
c --> d[component-name-driver]
d --> e[component-name]
    

Table

Argo Workflows (AWF) Task Name

AWF Task Type

AWF Template Name

Driver Type

Pod

Responsibilities

entrypoint

dag

entrypoint

N/A

In Argo Workflows, the entrypoint DAG defines the primary template to be executed when a workflow is initiated. It acts as the starting point or “main” function of your workflow.

The entrypoint has two sequential tasks:

  1. root-driver
  2. driver

root-driver

container

system-dag-driver

ROOT_DAG

✅

  1. Validates provided options
  2. Determines where artifacts will be stored
  3. Creates the execution record in MLMD

root

dag

root

N/A

The root DAG has two sequential tasks:

  1. <component-name>-driver
  2. <component-name>

<component-name>-driver

container

system-container-driver

CONTAINER

✅

  1. Resolves inputs from upstream tasks
  2. Evaluates conditions
  3. Prepares output destinations
  4. Makes caching decisions
  5. Creates execution record in MLMD
  6. Dynamically configures subsequent execution pod (including K8s-specific config) at runtime using podSpecPatch

<component-name>

container

system-container-executor

N/A

✅

  1. Downloads input artifacts
  2. Executes end-user code
  3. Uploads output artifacts
  4. Records output metadata and execution status to MLMD
  5. Publishes logs

Reference Pipeline

from kfp import dsl
from kfp import compiler


@dsl.component()
def hello_world():
    print("hello world")


@dsl.pipeline()
def pipeline_hello_world():
    task = hello_world()

Pipeline Execution Flow on Argo Workflows Backend

flow

Table

Pipeline Execution Component Name

Responsibilities

Workflow Controller

Workflow Controller is the central component in the Argo Workflows backend. It monitors and executes Argo Workflow Custom Resources (CRs)

MLMD Metadata*

MLMD Metadata is a storage system for tracking experiments

Argo Workflow CR

Argo Workflow CR is the representation of a pipeline at the Argo Workflows level. It contains information about all tasks and their dependencies within the pipeline.

API-server

API Server — from the pipeline execution perspective, it provides access to the previously executed pipeline cache (if it exists).

The MLMD implementation is planned to be replaced in the future.