Overview

Kubeflow Pipelines (KFP) is a platform for building and deploying portable and scalable machine learning (ML) workflows using Docker containers.

With KFP you can author components and pipelines using the KFP Python SDK, compile pipelines to an intermediate representation YAML, and submit the pipeline to run on a KFP-conformant backend such as the open source KFP backend or Google Cloud Vertex AI Pipelines.

The open source KFP backend is available as a core component of Kubeflow or as a standalone installation. Follow the installation instructions and Hello World Pipeline example to quickly get started with KFP.

Why Kubeflow Pipelines?

KFP enables data scientists and machine learning engineers to:

• Author end-to-end ML workflows natively in Python
• Create fully custom ML components or leverage an ecosystem of existing components
• Easily manage, track, and visualize pipeline definitions, runs, experiments, and ML artifacts
• Efficiently use compute resources through parallel task execution and through caching to eliminating redundant executions
• Maintain cross-platform pipeline portability through a platform-neutral IR YAML pipeline definition

What is a pipeline?

A pipeline is a definition of a workflow that composes one or more components together to form a computational directed acyclic graph (DAG). At runtime, each component execution corresponds to a single container execution, which may create ML artifacts. Pipelines may also feature control flow.

Next steps

• Hello World Pipeline
• Learn more about authoring components
• Learn more about authoring pipelines