Kubernetes Storage Classes: A Comprehensive Guide¶

Background: Understanding Storage in Kubernetes¶

Before diving into Storage Classes, let’s revisit the foundational concepts of storage in Kubernetes. These are critical for understanding the role of Storage Classes in managing persistent storage.

Key Concepts¶

1. Pods and Volumes:
2. Pods are the smallest deployable units in Kubernetes, running one or more containers. Containers within a Pod share resources, including storage, which is provided via volumes.

3. Volumes allow containers to access and share data, either temporarily (ephemeral volumes) or persistently (backed by durable storage). They are defined in a Pod’s .spec.volumes field and mounted into containers.

4. Persistent Volumes (PVs) and Persistent Volume Claims (PVCs):

5. PVs are cluster-wide resources representing physical storage (e.g., NFS shares, cloud disks, iSCSI volumes). They can be provisioned manually (static provisioning) or automatically (dynamic provisioning) and have lifecycles independent of Pods.
6. PVCs are user requests for storage, specifying requirements like size, access modes, and optionally a Storage Class. PVCs bind to PVs, allowing Pods to use persistent storage.

7. PVs and PVCs decouple storage provisioning from consumption, enabling users to request storage without managing the underlying infrastructure.

8. Storage Challenges:

9. Different applications require different storage characteristics, such as high performance (e.g., for databases), low latency, or specific backup policies.
10. Manually provisioning PVs for each PVC is time-consuming and error-prone, especially in large clusters with diverse storage needs.
11. Cluster administrators need a way to offer standardized storage options while abstracting backend details from users.

Why Storage Classes?¶

Storage Classes address these challenges by providing a framework for administrators to define and offer different “classes” of storage. Each class represents a storage profile with specific attributes, such as performance, provisioning method, or reclaim policy. Users request a Storage Class via a PVC, and Kubernetes dynamically provisions a PV tailored to the class’s specifications.

Analogy: Think of Storage Classes as a menu at a restaurant. The chef (administrator) defines the dishes (storage profiles) available, each with specific ingredients and preparation methods (e.g., fast SSDs, replicated storage). Customers (users) choose a dish by name (Storage Class) via their order (PVC), and the kitchen (Kubernetes) prepares the meal (PV) according to the recipe.

With this background, students should understand that Storage Classes enable dynamic, scalable, and flexible storage management in Kubernetes. Now, let’s explore Storage Classes in detail.

Introduction to Storage Classes¶

A Storage Class in Kubernetes is a resource that allows cluster administrators to define different types of storage available in the cluster. Each Storage Class acts as a template for dynamically provisioning Persistent Volumes (PVs) when a Persistent Volume Claim (PVC) requests it. Storage Classes abstract the details of storage provisioning, enabling users to request storage by specifying a class name rather than configuring low-level storage details.

Key Characteristics¶

• Purpose: Provide a standardized way to offer storage with varying attributes, such as performance (e.g., SSD vs. HDD), quality-of-service (e.g., low-latency), or policies (e.g., backups, encryption).
• Dynamic Provisioning: Automatically create PVs for PVCs, reducing manual administrative overhead compared to static provisioning.
• Flexibility: Support multiple storage backends (e.g., cloud providers, NFS, CSI drivers) through provisioners.
• Cluster-Wide: Defined at the cluster level, available to all namespaces unless restricted by policies.

Explanation: Storage Classes make storage management scalable by allowing administrators to predefine storage options. Users select a class via the storageClassName field in a PVC, and Kubernetes handles the rest, provisioning a PV that matches the class’s configuration.

Storage Class Specification¶

A Storage Class is defined by a StorageClass object with the following key fields:

1. metadata.name:
2. The name of the Storage Class, used by PVCs to request this class.
3. Must be unique within the cluster.

4. Example: low-latency, standard.

5. provisioner:

6. Specifies the volume plugin or external provisioner responsible for creating PVs.
7. Internal provisioners are prefixed with kubernetes.io (e.g., kubernetes.io/aws-ebs).
8. External provisioners are third-party programs following Kubernetes specifications.

9. Example: csi.vsphere.vmware.com, efs.csi.aws.com.

10. parameters:

11. Driver-specific key-value pairs that configure the storage (e.g., filesystem type, performance settings).
12. Limited to 512 parameters, with a total size (keys and values) not exceeding 256 KiB.

13. Example: type: io1, iopsPerGB: "50" for AWS EBS.

14. reclaimPolicy:

15. Determines what happens to a PV after its PVC is deleted:
    • Delete (default): Deletes the PV and underlying storage.
    • Retain: Keeps the PV and storage, requiring manual cleanup.

16. Example: reclaimPolicy: Retain.

17. allowVolumeExpansion:

18. Boolean indicating whether PVs created by this class can be resized by editing the PVC.
19. Supported by specific volume types (e.g., CSI, Azure File, Portworx).

20. Example: allowVolumeExpansion: true.

21. mountOptions:

22. Specifies mount options for PVs (e.g., discard for TRIM support).
23. Not all volume plugins support mount options; invalid options cause provisioning to fail.

24. Example: mountOptions: [discard].

25. volumeBindingMode:

26. Controls when PV binding and provisioning occur:
    • Immediate (default): Binds/provisions the PV as soon as the PVC is created.
    • WaitForFirstConsumer: Delays binding/provisioning until a Pod using the PVC is scheduled, respecting Pod scheduling constraints.

27. Example: volumeBindingMode: WaitForFirstConsumer.

28. allowedTopologies (optional):

29. Restricts PV provisioning to specific topological domains (e.g., zones, regions).
30. Used with WaitForFirstConsumer to limit where PVs are created.
31. Example: Restrict to specific AWS availability zones.

Example Storage Class¶

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: low-latency
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: csi-driver.example-vendor.example
reclaimPolicy: Retain
allowVolumeExpansion: true
mountOptions:
  - discard
volumeBindingMode: WaitForFirstConsumer
parameters:
  guaranteedReadWriteLatency: "true"

Explanation: This Storage Class defines a low-latency storage profile using a custom CSI driver. It retains PVs after PVC deletion, supports volume expansion, and delays provisioning until a Pod is scheduled. The discard mount option enables TRIM for block storage, and the guaranteedReadWriteLatency parameter is driver-specific, ensuring low-latency performance.

Default Storage Class¶

A cluster can have a default Storage Class, which is applied to PVCs that do not specify a storageClassName. This simplifies user workflows by providing a fallback storage option.

Configuration¶

• Mark a Storage Class as default by setting the annotation:

  metadata:
    annotations:
      storageclass.kubernetes.io/is-default-class: "true"
  

• Only one Storage Class should be marked as default to avoid ambiguity. If multiple are marked, Kubernetes uses the most recently created default.

Behavior¶

• With a Default Storage Class:
• PVCs without storageClassName use the default Storage Class.
• PVCs with storageClassName: "" explicitly opt out of dynamic provisioning, binding only to PVs with no class.
• Without a Default Storage Class:
• PVCs without storageClassName remain unset until a default is created.
• Retroactive assignment (Stable, v1.28) updates existing PVCs without storageClassName to the new default, unless storageClassName: "" is set.
• Example:

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: myclaim
  spec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
    # No storageClassName; uses default if available
  

Changing the Default Storage Class¶

To migrate to a new default Storage Class: 1. Remove the storageclass.kubernetes.io/is-default-class: true annotation from the old default. 2. Add the annotation to the new Storage Class. 3. Existing PVCs without storageClassName are updated retroactively to the new default.

Recommendation: Ensure only one Storage Class is marked as default to prevent unexpected behavior.

Provisioners¶

The provisioner field determines which volume plugin or external program creates PVs for a Storage Class. Kubernetes supports both internal and external provisioners.

Internal Provisioners¶

These are built into Kubernetes and prefixed with kubernetes.io. As of Kubernetes v1.33, many in-tree provisioners are deprecated, with CSI drivers recommended instead. Supported internal provisioners include:

Note: Deprecated in-tree provisioners should be replaced with CSI drivers for future compatibility.

External Provisioners¶

External provisioners are third-party programs that follow Kubernetes’ provisioning specification. They offer flexibility for custom storage solutions or vendor-specific integrations. Examples include: - NFS: External provisioners like nfs-ganesha or nfs-subdir-external-provisioner. - CSI Drivers: Out-of-tree drivers for AWS EBS, Azure Disk, vSphere, etc. - Custom Solutions: Vendor-specific provisioners hosted in repositories like kubernetes-sigs/sig-storage-lib-external-provisioner.

Explanation: External provisioners allow Kubernetes to integrate with virtually any storage system, making Storage Classes highly extensible.

Reclaim Policy¶

The reclaimPolicy field specifies what happens to a PV after its PVC is deleted: - Delete (default): Deletes the PV and its underlying storage asset (e.g., cloud disk, NFS share). - Retain: Keeps the PV and storage, marking the PV as “released.” Administrators must manually clean up or reuse the storage. - Example:

reclaimPolicy: Retain

Notes: - Dynamically provisioned PVs inherit the Storage Class’s reclaimPolicy. - Manually created PVs retain their original reclaim policy, even if managed by a Storage Class. - Use Retain for critical data requiring manual intervention; use Delete for automated cleanup in cloud environments.

Volume Expansion¶

Storage Classes can enable volume expansion, allowing users to increase a PV’s size by editing the corresponding PVC’s resources.requests.storage. This requires: - allowVolumeExpansion: true in the Storage Class. - Support from the underlying volume plugin.

Supported Volume Types¶

Limitations: - Only expansion (growing) is supported; shrinking is not allowed. - Expansion requires the underlying storage backend and driver to support resizing.

Example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi  # Increased from 10Gi
  storageClassName: low-latency

Explanation: Volume expansion is critical for applications with growing storage needs, such as databases or log aggregators, but requires careful configuration to ensure compatibility.

Volume Binding Mode¶

The volumeBindingMode field controls when PV binding and provisioning occur, impacting scheduling and resource allocation.

Modes¶

1. Immediate (default):
2. Binds/provisions the PV immediately after PVC creation.
3. Suitable for storage backends accessible cluster-wide (e.g., cloud disks).
4. Drawback: May bind to a PV on a node where the Pod cannot be scheduled, causing scheduling failures.
5. WaitForFirstConsumer:
6. Delays binding/provisioning until a Pod using the PVC is scheduled.
7. Respects Pod scheduling constraints (e.g., node selectors, affinity rules).
8. Ideal for topology-constrained storage (e.g., local volumes, zoned cloud storage).
9. Supported Plugins:
   • CSI volumes (if the driver supports it).
   • Local volumes (for pre-created PV binding).

Example with WaitForFirstConsumer¶

apiVersion: v1
kind: Pod
metadata:
  name: task-pv-pod
spec:
  nodeSelector:
    kubernetes.io/hostname: kube-01
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
        claimName: task-pv-claim
  containers:
    - name: task-pv-container
      image: nginx
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: task-pv-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: local-storage
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Explanation: The WaitForFirstConsumer mode ensures the PV is provisioned on a node where the Pod can run, improving scheduling reliability for topology-constrained storage.

Caution: - Avoid using nodeName in the Pod spec with WaitForFirstConsumer, as it bypasses the scheduler, leaving the PVC in a pending state. - Use nodeSelector or other scheduling constraints instead.

Allowed Topologies¶

The allowedTopologies field restricts PV provisioning to specific topological domains, such as zones or regions, when using WaitForFirstConsumer. This is useful for ensuring PVs are created in locations accessible to scheduled Pods.

Example¶

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
provisioner: example.com/example
parameters:
  type: pd-standard
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: topology.kubernetes.io/zone
    values:
    - us-central-1a
    - us-central-1b

Explanation: This Storage Class restricts PV provisioning to the us-central-1a and us-central-1b zones, ensuring compatibility with Pods scheduled in those zones. It replaces older zone and zones parameters used by some plugins.

Storage Class Parameters¶

The parameters field provides driver-specific configuration for PVs. Parameters vary by provisioner and are not standardized. Below are examples for common storage backends as of Kubernetes v1.33.

AWS EBS (CSI Driver)¶

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  csi.storage.k8s.io/fstype: xfs
  type: io1
  iopsPerGB: "50"
  encrypted: "true"
  tagSpecification_1: "key1=value1"
  tagSpecification_2: "key2=value2"
allowedTopologies:
- matchLabelExpressions:
  - key: topology.ebs.csi.aws.com/zone
    values:
    - us-east-2c

Parameters: - csi.storage.k8s.io/fstype: Filesystem type (e.g., xfs, ext4). - type: EBS volume type (e.g., io1, gp3). - iopsPerGB: IOPS for provisioned IOPS volumes. - encrypted: Enables encryption. - tagSpecification_N: Tags applied to EBS volumes.

Note: The in-tree awsElasticBlockStore provisioner was removed in v1.27; use the ebs.csi.aws.com CSI driver.

AWS EFS (CSI Driver)¶

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-92107410
  directoryPerms: "700"

Parameters: - provisioningMode: Set to efs-ap for access point-based provisioning. - fileSystemId: EFS file system ID. - directoryPerms: Permissions for the root directory.

Note: Requires the efs.csi.aws.com CSI driver. See AWS EFS CSI Driver documentation.

NFS (External Provisioner)¶

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: example-nfs
provisioner: example.com/external-nfs
parameters:
  server: nfs-server.example.com
  path: /share
  readOnly: "false"

Parameters: - server: NFS server hostname or IP. - path: Exported NFS path. - readOnly: Mount as read-only (true or false).

Note: Kubernetes does not provide an internal NFS provisioner. Use external provisioners like nfs-subdir-external-provisioner.

vSphere (CSI Driver)¶

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: csi.vsphere.vmware.com
parameters:
  storagePolicyName: "vSAN Default Storage Policy"
  fstype: ext4

Parameters: - storagePolicyName: vSphere Storage Policy Based Management (SPBM) policy. - fstype: Filesystem type (e.g., ext4, xfs).

Note: The in-tree vsphere-volume provisioner is deprecated; use the csi.vsphere.vmware.com CSI driver. Supports SPBM for policy-driven storage management.

Ceph RBD (Deprecated)¶

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/rbd
parameters:
  monitors: 198.19.254.105:6789
  adminId: kube
  adminSecretName: ceph-secret
  adminSecretNamespace: kube-system
  pool: kube
  userId: kube
  userSecretName: ceph-secret-user
  userSecretNamespace: default
  fsType: ext4
  imageFormat: "2"
  imageFeatures: "layering"

Parameters: - monitors: Ceph monitor addresses. - adminId, userId: Ceph client IDs. - adminSecretName, userSecretName: Secrets for authentication. - pool: Ceph RBD pool. - fsType: Filesystem type. - imageFormat, imageFeatures: RBD image settings.

Note: Deprecated in v1.28; use the Ceph RBD CSI driver.

Azure File (Deprecated)¶

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azurefile
provisioner: kubernetes.io/azure-file
parameters:
  skuName: Standard_LRS
  location: eastus
  storageAccount: azure_storage_account_name

Parameters: - skuName: Azure storage account SKU (e.g., Standard_LRS). - location: Azure region. - storageAccount: Storage account name.

Note: Deprecated; use the file.csi.azure.com CSI driver.

Portworx (Deprecated)¶

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: portworx-io-priority-high
provisioner: kubernetes.io/portworx-volume
parameters:
  repl: "1"
  snap_interval: "70"
  priority_io: "high"

Parameters: - repl: Number of replicas (1–3). - snap_interval: Snapshot interval in minutes. - priority_io: Performance priority (high, medium, low).

Note: Deprecated; use the pxd.portworx.com CSI driver.

Local¶

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Parameters: None, as local volumes do not support dynamic provisioning.

Explanation: The no-provisioner indicates manual PV creation, but the Storage Class delays binding until Pod scheduling, improving compatibility with node-specific storage.

Best Practices¶

1. Use CSI Drivers:
2. Transition from deprecated in-tree provisioners to CSI drivers for better support and future compatibility.

3. Example: Replace kubernetes.io/aws-ebs with ebs.csi.aws.com.

4. Set a Single Default Storage Class:

5. Ensure only one Storage Class is marked as default to avoid ambiguity.

6. Regularly review and update the default as cluster needs evolve.

7. Prefer WaitForFirstConsumer:

8. Use WaitForFirstConsumer for topology-constrained storage to ensure Pods are scheduled on nodes with access to the PV.

9. Avoid nodeName in Pod specs; use nodeSelector or affinity rules.

10. Enable Volume Expansion:

11. Set allowVolumeExpansion: true for Storage Classes supporting dynamic workloads, but verify driver compatibility.

12. Configure Reclaim Policies Appropriately:

13. Use Delete for cloud-backed storage to automate cleanup.

14. Use Retain for critical data requiring manual intervention.

15. Restrict Topologies When Needed:

16. Use allowedTopologies to align PV provisioning with cluster topology, especially in multi-zone or multi-region setups.

17. Validate Parameters:

18. Ensure parameters are correctly configured per the provisioner’s documentation to avoid provisioning failures.

19. Test Storage Classes in a staging environment before production use.

20. Secure Storage Access:

21. In multi-tenant clusters, use RBAC and secrets (e.g., for Azure File) to restrict access to storage credentials.
22. Set secretNamespace explicitly for sensitive provisioners.

What’s Next¶

• Experiment with Storage Classes in a lab environment, such as creating a low-latency Storage Class for a database application.
• Explore CSI driver documentation for specific storage backends (e.g., AWS EBS, vSphere).
• Review the Kubernetes CSI Drivers list for compatible provisioners and their features.