Integration with Volcano for Batch Scheduling

Volcano is a batch system built on Kubernetes. It provides a suite of mechanisms currently missing from Kubernetes that are commonly required by many classes of batch & elastic workloads. With the integration with Volcano, Spark application pods can be scheduled for better scheduling efficiency.

Volcano components

Before using Kubernetes Operator for Apache Spark, with Volcano enabled, user need to ensure Volcano has been successfully installed in the same environment, please refer Quick Start Guide for Volcano installation.

Install Kubernetes Operator for Apache Spark with Volcano enabled

Within the help of Helm chart, Kubernetes Operator for Apache Spark with Volcano can be easily installed with the command below:

helm repo add spark-operator https://kubeflow.github.io/spark-operator

helm install my-release spark-operator/spark-operator \
    --namespace spark-operator \
    --set webhook.enable=true \
    --set batchScheduler.enable=true

Run Spark Application with Volcano scheduler

Now, we can run an updated version of spark application (with batchScheduler configured), for instance:

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: spark:3.5.1
  imagePullPolicy: Always
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-v3.5.1.jar
  sparkVersion: 3.5.1
  batchScheduler: volcano # Note: the batch scheduler name must be specified with `volcano`
  restartPolicy:
    type: Never
  volumes:
    - name: test-volume
      hostPath:
        path: /tmp
        type: Directory
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    labels:
      version: 3.5.1
    serviceAccount: spark
    volumeMounts:
      - name: test-volume
        mountPath: /tmp
  executor:
    cores: 1
    instances: 1
    memory: 512m
    labels:
      version: 3.5.1
    volumeMounts:
      - name: test-volume
        mountPath: "/tmp"

When running, the Pods Events can be used to verify that whether the pods have been scheduled via Volcano.

Type    Reason     Age   From                          Message
----    ------     ----  ----                          -------
Normal  Scheduled  23s   volcano                       Successfully assigned default/spark-pi-driver to integration-worker2

Technological detail

If SparkApplication is configured to run with Volcano, there are some details underground that make the two systems integrated:

1. Kubernetes Operator for Apache Spark’s webhook will patch pods’ schedulerName according to the batchScheduler in SparkApplication Spec.
2. Before submitting spark application, Kubernetes Operator for Apache Spark will create a Volcano native resource PodGrouphere for the whole application. and as a brief introduction, most of the Volcano’s advanced scheduling features, such as pod delay creation, resource fairness and gang scheduling are all depend on this resource. Also, a new pod annotation named scheduling.k8s.io/group-name will be added.
3. Volcano scheduler will take over all of the pods that both have schedulerName and annotation correctly configured for scheduling.

Kubernetes Operator for Apache Spark enables end user to have fine-grained controlled on batch scheduling via attribute BatchSchedulerOptions. BatchSchedulerOptions is a string dictionary that different batch scheduler can utilize it to expose different attributes. For now, volcano support these attributes below: