Kubernetes Dynamic Volume Provisioning: A Comprehensive Guide

Background: Understanding Storage in Kubernetes

Before exploring Dynamic Volume Provisioning, let’s revisit the foundational storage concepts in Kubernetes. These are critical for understanding how dynamic provisioning simplifies and automates storage management.

Key Concepts

  1. Pods and Volumes:
  2. Pods are the smallest deployable units in Kubernetes, running one or more containers. Containers within a Pod share resources, including storage, provided via volumes.

  3. Volumes can be ephemeral (tied to the Pod’s lifecycle) or persistent (surviving Pod restarts), enabling data storage and sharing.

  4. Persistent Volumes (PVs) and Persistent Volume Claims (PVCs):

  5. PVs are cluster-wide resources representing physical storage (e.g., cloud disks, NFS shares, iSCSI volumes). They can be created manually (static provisioning) or automatically (dynamic provisioning).
  6. PVCs are user requests for storage, specifying requirements like size, access modes, and optionally a Storage Class. PVCs bind to PVs, allowing Pods to use persistent storage.

  7. PVs and PVCs decouple storage provisioning from consumption, enabling users to request storage without managing backend details.

  8. Storage Classes:

  9. Storage Classes define storage profiles (e.g., performance, reclaim policies) and specify a provisioner to create PVs dynamically.

  10. They allow administrators to offer different types of storage (e.g., SSD vs. HDD) and abstract provisioning complexity from users.

  11. Static vs. Dynamic Provisioning:

  12. Static Provisioning: Administrators manually create PVs by interacting with the storage provider (e.g., creating cloud disks) and defining PV objects in Kubernetes. This is labor-intensive and error-prone in large clusters.
  13. Dynamic Provisioning: Automatically creates PVs on-demand when a PVC is created, using a Storage Class’s provisioner. This eliminates manual intervention, improving scalability.

Why Dynamic Volume Provisioning?

Dynamic volume provisioning addresses the limitations of static provisioning by automating storage creation. It enables: - Scalability: Supports large clusters with frequent storage requests without manual overhead. - User Simplicity: Allows users to request storage via PVCs without understanding backend storage systems. - Flexibility: Supports multiple storage types (e.g., fast SSDs, standard disks) through Storage Classes. - Topology Awareness: Ensures PVs are provisioned in locations compatible with Pod scheduling, especially in multi-zone clusters.

Analogy: Static provisioning is like a librarian manually fetching and assigning books (PVs) for each patron’s request (PVC). Dynamic provisioning is like an automated library system that delivers the right book based on a catalog entry (Storage Class) when a patron submits a request.

With this background, students should understand that dynamic volume provisioning is a key feature for automating and scaling storage management in Kubernetes. Now, let’s dive into the details.

Introduction to Dynamic Volume Provisioning

Dynamic Volume Provisioning is a Kubernetes feature that allows storage volumes (PVs) to be created automatically when a user submits a Persistent Volume Claim (PVC). Instead of requiring administrators to pre-provision storage and create PV objects manually, dynamic provisioning leverages Storage Classes to define how storage should be created, including the provisioner and configuration parameters. This on-demand approach simplifies storage management and supports scalable, user-driven workflows.

Key Characteristics

  • On-Demand Creation: PVs are provisioned only when a PVC requests storage, reducing wasted resources.
  • Storage Class-Driven: Relies on Storage Classes to specify the provisioner (e.g., cloud provider, CSI driver) and storage attributes (e.g., performance, filesystem type).
  • User-Friendly: Users specify a Storage Class in the PVC, abstracting backend complexity.
  • Topology Awareness: Supports provisioning in specific topological domains (e.g., zones) to align with Pod scheduling.
  • Automation: Eliminates manual interaction with storage providers, streamlining cluster operations.

Explanation: Dynamic provisioning shifts the burden of storage provisioning from administrators to Kubernetes, enabling seamless integration with cloud and on-premises storage systems. It’s particularly valuable in dynamic environments where storage needs change frequently.

Enabling Dynamic Volume Provisioning

To enable dynamic volume provisioning, cluster administrators must create one or more Storage Class objects. These define the available storage options and specify how PVs should be provisioned when a PVC requests them.

Steps to Enable

  1. Define Storage Class Objects:
  2. Create StorageClass objects in the storage.k8s.io/v1 API group.
  3. Specify the provisioner, parameters, and other fields like reclaimPolicy or volumeBindingMode.

  4. Ensure the name is a valid DNS subdomain (e.g., slow, fast).

  5. Configure Provisioners:

  6. Choose an internal provisioner (e.g., kubernetes.io/gce-pd) or an external provisioner (e.g., CSI driver, NFS provisioner).

  7. Ensure the provisioner is installed and configured in the cluster.

  8. Enable Default Storage Class (Optional):

  9. Mark a Storage Class as default using the storageclass.kubernetes.io/is-default-class: "true" annotation.
  10. Enable the DefaultStorageClass admission controller on the API server (configured via --enable-admission-plugins=DefaultStorageClass).
  11. This ensures PVCs without a storageClassName use the default Storage Class.

Example Storage Classes

Slow Storage (Standard Disks)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: slow
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Explanation: This Storage Class provisions standard Google Cloud Persistent Disks (HDD-like) with a Delete reclaim policy. It’s marked as the default and delays provisioning until a Pod is scheduled (WaitForFirstConsumer).

Fast Storage (SSDs)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

Explanation: This Storage Class provisions SSD-based Google Cloud Persistent Disks with a Retain reclaim policy, supporting volume expansion. It also uses WaitForFirstConsumer for topology-aware provisioning.

Note: The parameters field is specific to the provisioner. For example, type: pd-ssd is a Google Cloud-specific parameter.

Using Dynamic Volume Provisioning

Users request dynamically provisioned storage by creating a PVC that references a Storage Class via the storageClassName field. Kubernetes then provisions a PV using the specified Storage Class’s provisioner and parameters.

PVC Configuration

  • Set the storageClassName field to the name of the desired Storage Class (e.g., fast, slow).
  • Specify accessModes and resources.requests.storage as needed.
  • If storageClassName is omitted and a default Storage Class exists, the default is used.
  • To disable dynamic provisioning, set storageClassName: "", which restricts the PVC to binding with existing PVs without a Storage Class.

Example PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: claim1
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast
  resources:
    requests:
      storage: 30Gi

Explanation: This PVC requests 30Gi of storage from the fast Storage Class, resulting in an SSD-based PV being provisioned. When the PVC is deleted, the PV’s fate depends on the reclaimPolicy (Retain in this case, requiring manual cleanup).

Deprecated Annotation

Before Kubernetes v1.6, dynamic provisioning was specified using the volume.beta.kubernetes.io/storage-class annotation in the PVC:

metadata:
  annotations:
    volume.beta.kubernetes.io/storage-class: fast

Note: This annotation is deprecated since v1.9. Always use the storageClassName field for modern Kubernetes clusters.

Defaulting Behavior

Dynamic provisioning can be configured to apply automatically to PVCs that don’t specify a storageClassName, simplifying user workflows.

Configuration

  1. Mark a Default Storage Class:
  2. Add the annotation: 
    metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"

  3. Only one Storage Class should be marked as default. If multiple are marked, Kubernetes uses the most recently created one.

  4. Enable DefaultStorageClass Admission Controller:

  5. Ensure the API server includes DefaultStorageClass in its --enable-admission-plugins flag.
  6. This controller automatically sets the storageClassName field of PVCs without it to the default Storage Class.

Behavior [->]

  • With a Default Storage Class:
  • PVCs without storageClassName use the default Storage Class, triggering dynamic provisioning.
  • PVCs with storageClassName: "" opt out of dynamic provisioning, binding only to PVs with no Storage Class.
  • Without a Default Storage Class:
  • PVCs without storageClassName remain unset and do not trigger dynamic provisioning until a default is created.
  • Retroactive assignment (Stable, v1.28) updates existing PVCs without storageClassName to the new default, unless storageClassName: "" is set.
  • Multiple Defaults:
  • If multiple Storage Classes are marked as default, Kubernetes selects the most recently created one for new PVCs. This supports seamless migration but should be avoided to prevent confusion.

Example PVC with Default

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: claim2
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  # No storageClassName; uses default ("slow" in this case)

Explanation: This PVC triggers dynamic provisioning using the slow Storage Class (default), provisioning a standard disk.

Topology Awareness

In multi-zone or multi-region clusters, Pods may be scheduled across different topological domains (e.g., availability zones). Dynamic provisioning must ensure PVs are created in locations accessible to the Pods using them, especially for topology-constrained storage (e.g., local volumes, zoned cloud disks).

Volume Binding Mode

The volumeBindingMode field in a Storage Class controls when PV provisioning occurs, enabling topology-aware provisioning:

  1. Immediate (default):
  2. Provisions the PV as soon as the PVC is created.
  3. May provision in a zone incompatible with the Pod’s scheduling constraints, causing scheduling failures.
  4. WaitForFirstConsumer:
  5. Delays provisioning until a Pod using the PVC is scheduled.
  6. Ensures the PV is created in a topological domain (e.g., zone) accessible to the Pod, respecting scheduling constraints like node selectors or affinity rules.

Example with Topology Awareness

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: zoned-storage
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: topology.ebs.csi.aws.com/zone
    values:
    - us-east-1a
    - us-east-1b
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zoned-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: zoned-storage
  resources:
    requests:
      storage: 20Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: zoned-pod
spec:
  nodeSelector:
    kubernetes.io/hostname: kube-01
  containers:
  - name: app
    image: nginx
    volumeMounts:
    - mountPath: /data
      name: storage
  volumes:
  - name: storage
    persistentVolumeClaim:
      claimName: zoned-claim

Explanation: The zoned-storage Storage Class uses WaitForFirstConsumer to delay PV provisioning until the zoned-pod is scheduled. The PV is provisioned in us-east-1a or us-east-1b, ensuring compatibility with the Pod’s node selector. This prevents scheduling issues in multi-zone clusters.

Caution: - Avoid using nodeName in the Pod spec with WaitForFirstConsumer, as it bypasses the scheduler, leaving the PVC in a pending state. - Use nodeSelector or affinity rules for topology constraints.

Allowed Topologies

The allowedTopologies field in a Storage Class further restricts PV provisioning to specific topological domains, enhancing control in multi-zone setups.

Example: 

allowedTopologies:
- matchLabelExpressions:
  - key: topology.kubernetes.io/zone
    values:
    - us-central-1a
    - us-central-1b

Explanation: This restricts PVs to the specified zones, ensuring they align with Pod scheduling requirements.

Supported Provisioners

Dynamic provisioning relies on provisioners specified in the Storage Class. As of Kubernetes v1.33, supported provisioners include:

Internal Provisioners

Volume Plugin Provisioner Name Status (v1.33) Example Use Case
AzureFile kubernetes.io/azure-file Deprecated Azure File shares
PortworxVolume kubernetes.io/portworx-volume Deprecated Portworx storage
VsphereVolume kubernetes.io/vsphere-volume Deprecated vSphere VMDK volumes
GCEPersistentDisk kubernetes.io/gce-pd Supported Google Cloud disks

Note: Deprecated in-tree provisioners should be replaced with CSI drivers.

External Provisioners

  • CSI Drivers: Out-of-tree drivers for AWS EBS (ebs.csi.aws.com), Azure Disk (disk.csi.azure.com), vSphere (csi.vsphere.vmware.com), etc.
  • NFS: External provisioners like nfs-subdir-external-provisioner.
  • Custom Provisioners: Third-party programs hosted in repositories like kubernetes-sigs/sig-storage-lib-external-provisioner.

Example (AWS EBS CSI): 

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-fast
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer

Explanation: This Storage Class uses the AWS EBS CSI driver to provision encrypted gp3 volumes, with topology-aware provisioning.

Best Practices

  1. Define Multiple Storage Classes:
  2. Create Storage Classes for different use cases (e.g., fast for SSDs, slow for HDDs) to give users flexibility.

  3. Use descriptive names (e.g., low-latency, high-capacity) to clarify intent.

  4. Set a Single Default Storage Class:

  5. Mark one Storage Class as default to simplify PVC creation, but avoid multiple defaults to prevent ambiguity.

  6. Regularly review the default to ensure it meets cluster needs.

  7. Use WaitForFirstConsumer:

  8. Prefer WaitForFirstConsumer for multi-zone clusters or topology-constrained storage to avoid scheduling issues.

  9. Combine with allowedTopologies for precise control.

  10. Transition to CSI Drivers:

  11. Replace deprecated in-tree provisioners with CSI drivers for better support and compatibility.

  12. Test CSI drivers in a staging environment before production use.

  13. Monitor Provisioning:

  14. Check PVC status (kubectl describe pvc) to ensure provisioning succeeds. Pending PVCs may indicate misconfigured Storage Classes or unavailable provisioners.

  15. Use cluster monitoring to track storage usage and provisioning failures.

  16. Secure Provisioning:

  17. In multi-tenant clusters, use RBAC to restrict access to Storage Classes and provisioners.

  18. Configure secrets for provisioners (e.g., Ceph RBD) in secure namespaces.

  19. Test Reclaim Policies:

  20. Use Delete for cloud-backed storage to automate cleanup.
  21. Use Retain for critical data, but establish processes for manual cleanup.

What’s Next

  • Experiment with dynamic provisioning in a lab environment, such as creating fast and slow Storage Classes and deploying applications with PVCs.
  • Explore CSI driver documentation for specific storage backends (e.g., AWS EBS, Google Cloud Persistent Disk).
  • Review the Kubernetes CSI Drivers list for compatible provisioners and their dynamic provisioning capabilities.