Pipeline Anatomy¶
This document clarifies the anatomy of a single-component pipeline. Complexity sometimes discourages scrutiny. My hope is that improved clarity will pave the way for consolidation and simplification.
Flowchart¶
graph TD
a(["entrypoint (DAG)"]) --> b[root-driver]
b --> c(["root (DAG)"])
c --> d[component-name-driver]
d --> e[component-name]
Table¶
Argo Workflows (AWF) Task Name |
AWF Task Type |
AWF Template Name |
Driver Type |
Pod |
Responsibilities |
|---|---|---|---|---|---|
entrypoint |
dag |
entrypoint |
N/A |
In Argo Workflows, the entrypoint DAG defines the primary template to be executed when a workflow is initiated. It acts as the starting point or “main” function of your workflow.
|
|
root-driver |
container |
system-dag-driver |
ROOT_DAG |
✅ |
|
root |
dag |
root |
N/A |
The root DAG has two sequential tasks:
|
|
<component-name>-driver |
container |
system-container-driver |
CONTAINER |
✅ |
|
<component-name> |
container |
system-container-executor |
N/A |
✅ |
|
Reference Pipeline¶
from kfp import dsl
from kfp import compiler
@dsl.component()
def hello_world():
print("hello world")
@dsl.pipeline()
def pipeline_hello_world():
task = hello_world()
Pipeline Execution Flow on Argo Workflows Backend¶

Table¶
Pipeline Execution Component Name |
Responsibilities |
|---|---|
Workflow Controller |
Workflow Controller is the central component in the Argo Workflows backend. It monitors and executes Argo Workflow Custom Resources (CRs) |
MLMD Metadata* |
MLMD Metadata is a storage system for tracking experiments |
Argo Workflow CR |
Argo Workflow CR is the representation of a pipeline at the Argo Workflows level. It contains information about all tasks and their dependencies within the pipeline. |
API-server |
API Server — from the pipeline execution perspective, it provides access to the previously executed pipeline cache (if it exists). |
The MLMD implementation is planned to be replaced in the future.