Kustomize Lab
A Kustomize configuration is located at /root/web-dashboard-kustomize on cluster2-controlplane. This application is designed to monitor the pods in the default namespace. It has been deployed using the following command:
kubectl kustomize /root/web-dashboard-kustomize/overlays/dev | kubectl apply -f -
The application is currently unable to monitor the pods due to insufficient permissions. Modify the Kustomize overlays/dev configuration to ensure the application is operational.
cluster2-controlplane ~ ➜ tree web-dashboard-kustomize/
web-dashboard-kustomize/
├── base
│ ├── deployment.yaml
│ ├── kustomization.yaml
│ ├── rolebinding.yaml
│ ├── role.yaml
│ ├── sa.yaml
│ └── service.yaml
└── overlays
└── dev
├── kustomization.yaml
└── patch-role.yaml
3 directories, 8 files
cluster2-controlplane ~ ➜ ls web-dashboard-kustomize/base/
deployment.yaml kustomization.yaml rolebinding.yaml role.yaml service.yaml
cluster2-controlplane ~ ➜ cat web-dashboard-kustomize/base/rolebinding.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: read-pods
namespace: default
subjects:
- kind: ServiceAccount
name: dashboard-sa
namespace: default
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
cluster2-controlplane ~ ➜ cat web-dashboard-kustomize/base/role.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: pod-reader
rules:
- apiGroups:
- ''
resources:
verbs:
cluster2-controlplane ~ ➜ cat web-dashboard-kustomize/base/service.yaml
kind: Service
apiVersion: v1
metadata:
name: dashboard-service
spec:
type: NodePort
selector:
name: web-dashboard
ports:
- port: 8080
targetPort: 8080
cluster2-controlplane ~ ➜ cat web-dashboard-kustomize/base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-dashboard
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: web-dashboard
template:
metadata:
labels:
app: web-dashboard
spec:
serviceAccountName: dashboard-sa # Ensure RBAC allows list & watch
containers:
- name: pod-watcher
image: python:3.8-slim
command: ["/bin/sh", "-c"]
args:
- |
pip install kubernetes && python -c "
import time
from kubernetes import client, config, watch
config.load_incluster_config()
v1 = client.CoreV1Api()
w = watch.Watch()
print('Listing pods in default namespace before watching...')
pods = v1.list_namespaced_pod(namespace='default')
for pod in pods.items:
print(f'Pod: {pod.metadata.name}')
print('Starting pod event stream...')
for event in w.stream(v1.list_namespaced_pod, namespace='default'):
pod = event['object']
print(f\"Event: {event['type']} - {pod.metadata.name}\")
"
env:
- name: PYTHONUNBUFFERED
value: "1"
cluster2-controlplane ~ ➜ cat web-dashboard-kustomize/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- role.yaml
- rolebinding.yaml
cluster2-controlplane ~ ➜ cat web-dashboard-kustomize/overlays/
cat: web-dashboard-kustomize/overlays/: Is a directory
cluster2-controlplane ~ ✖ ls web-dashboard-kustomize/overlays/dev/
kustomization.yaml patch-role.yaml
cluster2-controlplane ~ ➜ cat web-dashboard-kustomize/overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: patch-role.yaml
cluster2-controlplane ~ ➜ cat web-dashboard-kustomize/overlays/dev/patch-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: default
rules:
- apiGroups: [""]
resources: ["pods"]
verbs:
- get
- watch # added now
- list # added now
cluster2-controlplane ~ ➜ k create sa dashboard-sa -o yaml > web-dashboard-kustomize/base/sa.yaml
error: failed to create serviceaccount: serviceaccounts "dashboard-sa" already exists
cluster2-controlplane ~ ✖ k get sa
NAME SECRETS AGE
dashboard-sa 0 18m
default 0 123m
cluster2-controlplane ~ ➜ kubectl kustomize /root/web-dashboard-kustomize/overlays/dev | kubectl apply -f -
role.rbac.authorization.k8s.io/pod-reader configured
rolebinding.rbac.authorization.k8s.io/read-pods unchanged
service/dashboard-service unchanged
deployment.apps/web-dashboard unchanged
cluster2-controlplane ~ ➜ k get deployments.apps web-dashboard
NAME READY UP-TO-DATE AVAILABLE AGE
web-dashboard 0/1 1 0 19m # See, the application was already running, but I didn't read the question.
cluster2-controlplane ~ ➜ k get po
NAME READY STATUS RESTARTS AGE
web-dashboard-6c84688d84-4t99c 0/1 CrashLoopBackOff 8 (3m6s ago) 20m
cluster2-controlplane ~ ➜ k describe po web-dashboard-6c84688d84-4t99c
Name: web-dashboard-6c84688d84-4t99c
Namespace: default
Priority: 500
Priority Class Name: default-tier
Service Account: dashboard-sa
Node: cluster2-node01/192.168.81.149
Start Time: Sat, 01 Nov 2025 10:56:59 +0000
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 01 Nov 2025 11:14:16 +0000
Finished: Sat, 01 Nov 2025 11:14:26 +0000
Ready: False
Restart Count: 8
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 24s (x88 over 20m) kubelet Back-off restarting failed container pod-watcher in pod web-dashboard-6c84688d84-4t99c_default(625e0fac-09f3-40e2-9648-e87e64467701)
cluster2-controlplane ~ ➜ k logs web-dashboard-6c84688d84-4t99c
Collecting kubernetes
Downloading kubernetes-34.1.0-py2.py3-none-any.whl (2.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 8.7 MB/s eta 0:00:00
Collecting requests
Downloading requests-2.32.4-py3-none-any.whl (64 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.8/64.8 kB 21.0 MB/s eta 0:00:00
Collecting python-dateutil>=2.5.3
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.9/229.9 kB 53.2 MB/s eta 0:00:00
Collecting certifi>=14.05.14
Downloading certifi-2025.10.5-py3-none-any.whl (163 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.3/163.3 kB 48.4 MB/s eta 0:00:00
Collecting durationpy>=0.7
Downloading durationpy-0.10-py3-none-any.whl (3.9 kB)
Collecting six>=1.9.0
Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
Collecting requests-oauthlib
Downloading requests_oauthlib-2.0.0-py2.py3-none-any.whl (24 kB)
Collecting google-auth>=1.0.1
Downloading google_auth-2.42.1-py2.py3-none-any.whl (222 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 222.6/222.6 kB 54.1 MB/s eta 0:00:00
Collecting websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0
Downloading websocket_client-1.8.0-py3-none-any.whl (58 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.8/58.8 kB 21.6 MB/s eta 0:00:00
Collecting urllib3<2.4.0,>=1.24.2
Downloading urllib3-2.2.3-py3-none-any.whl (126 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 126.3/126.3 kB 31.4 MB/s eta 0:00:00
Collecting pyyaml>=5.4.1
Downloading PyYAML-6.0.3-cp38-cp38-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (806 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 806.0/806.0 kB 102.2 MB/s eta 0:00:00
Collecting rsa<5,>=3.1.4
Downloading rsa-4.9.1-py3-none-any.whl (34 kB)
Collecting cachetools<7.0,>=2.0.0
Downloading cachetools-5.5.2-py3-none-any.whl (10 kB)
Collecting pyasn1-modules>=0.2.1
Downloading pyasn1_modules-0.4.2-py3-none-any.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.3/181.3 kB 55.7 MB/s eta 0:00:00
Collecting charset_normalizer<4,>=2
Downloading charset_normalizer-3.4.4-cp38-cp38-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (147 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 147.6/147.6 kB 45.1 MB/s eta 0:00:00
Collecting idna<4,>=2.5
Downloading idna-3.11-py3-none-any.whl (71 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.0/71.0 kB 28.4 MB/s eta 0:00:00
Collecting oauthlib>=3.0.0
Downloading oauthlib-3.3.1-py3-none-any.whl (160 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.1/160.1 kB 52.1 MB/s eta 0:00:00
Collecting pyasn1<0.7.0,>=0.6.1
Downloading pyasn1-0.6.1-py3-none-any.whl (83 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.1/83.1 kB 21.7 MB/s eta 0:00:00
Installing collected packages: durationpy, websocket-client, urllib3, six, pyyaml, pyasn1, oauthlib, idna, charset_normalizer, certifi, cachetools, rsa, requests, python-dateutil, pyasn1-modules, requests-oauthlib, google-auth, kubernetes
Successfully installed cachetools-5.5.2 certifi-2025.10.5 charset_normalizer-3.4.4 durationpy-0.10 google-auth-2.42.1 idna-3.11 kubernetes-34.1.0 oauthlib-3.3.1 pyasn1-0.6.1 pyasn1-modules-0.4.2 python-dateutil-2.9.0.post0 pyyaml-6.0.3 requests-2.32.4 requests-oauthlib-2.0.0 rsa-4.9.1 six-1.17.0 urllib3-2.2.3 websocket-client-1.8.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[notice] A new release of pip is available: 23.0.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
Listing pods in default namespace before watching...
Traceback (most recent call last):
File "<string>", line 10, in <module>
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15968, in list_namespaced_pod
return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 16087, in list_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 244, in GET
return self.request("GET", url,
File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 238, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '84adac19-0b35-46d3-ab36-a3971c2d3e9e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'f2326076-6238-4ccf-91a9-d890646d890e', 'X-Kubernetes-Pf-Prioritylevel-Uid': '733ce29e-4f2d-4708-b1ce-f056fc184397', 'Date': 'Sat, 01 Nov 2025 11:14:26 GMT', 'Content-Length': '287'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:default:dashboard-sa\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
cluster2-controlplane ~ ➜ kubectl kustomize /root/web-dashboard-kustomize/overlays/dev | kubectl delete -f -
role.rbac.authorization.k8s.io "pod-reader" deleted
rolebinding.rbac.authorization.k8s.io "read-pods" deleted
service "dashboard-service" deleted
deployment.apps "web-dashboard" deleted
cluster2-controlplane ~ ➜ kubectl delete /root/web-dashboard-kustomize/base/
error: arguments in resource/name form may not have more than one slash
cluster2-controlplane ~ ✖ kubectl delete -f /root/web-dashboard-kustomize/base
Error from server (NotFound): error when deleting "/root/web-dashboard-kustomize/base/deployment.yaml": deployments.apps "web-dashboard" not found
Error from server (NotFound): error when deleting "/root/web-dashboard-kustomize/base/role.yaml": roles.rbac.authorization.k8s.io "pod-reader" not found
Error from server (NotFound): error when deleting "/root/web-dashboard-kustomize/base/rolebinding.yaml": rolebindings.rbac.authorization.k8s.io "read-pods" not found
Error from server (NotFound): error when deleting "/root/web-dashboard-kustomize/base/service.yaml": services "dashboard-service" not found
resource mapping not found for name: "" namespace: "" from "/root/web-dashboard-kustomize/base/kustomization.yaml": no matches for kind "Kustomization" in version "kustomize.config.k8s.io/v1beta1"
ensure CRDs are installed first
cluster2-controlplane ~ ✖ kubectl kustomize /root/web-dashboard-kustomize/overlays/dev | kubectl apply -f -
role.rbac.authorization.k8s.io/pod-reader created
rolebinding.rbac.authorization.k8s.io/read-pods created
service/dashboard-service created
deployment.apps/web-dashboard created
cluster2-controlplane ~ ➜ k get deployments.apps web-dashboard
NAME READY UP-TO-DATE AVAILABLE AGE
web-dashboard 1/1 1 1 17s
cluster2-controlplane ~ ➜ k logs web-dashboard-6c84688d84-g2slb
Collecting kubernetes
Downloading kubernetes-34.1.0-py2.py3-none-any.whl (2.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 8.6 MB/s eta 0:00:00
Collecting six>=1.9.0
Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
Collecting requests-oauthlib
Downloading requests_oauthlib-2.0.0-py2.py3-none-any.whl (24 kB)
Collecting durationpy>=0.7
Downloading durationpy-0.10-py3-none-any.whl (3.9 kB)
Collecting websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0
Downloading websocket_client-1.8.0-py3-none-any.whl (58 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.8/58.8 kB 15.9 MB/s eta 0:00:00
Collecting pyyaml>=5.4.1
Downloading PyYAML-6.0.3-cp38-cp38-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (806 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 806.0/806.0 kB 15.2 MB/s eta 0:00:00
Collecting urllib3<2.4.0,>=1.24.2
Downloading urllib3-2.2.3-py3-none-any.whl (126 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 126.3/126.3 kB 40.0 MB/s eta 0:00:00
Collecting certifi>=14.05.14
Downloading certifi-2025.10.5-py3-none-any.whl (163 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.3/163.3 kB 19.4 MB/s eta 0:00:00
Collecting python-dateutil>=2.5.3
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.9/229.9 kB 15.4 MB/s eta 0:00:00
Collecting google-auth>=1.0.1
Downloading google_auth-2.42.1-py2.py3-none-any.whl (222 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 222.6/222.6 kB 55.9 MB/s eta 0:00:00
Collecting requests
Downloading requests-2.32.4-py3-none-any.whl (64 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.8/64.8 kB 20.3 MB/s eta 0:00:00
Collecting rsa<5,>=3.1.4
Downloading rsa-4.9.1-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
Downloading pyasn1_modules-0.4.2-py3-none-any.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.3/181.3 kB 22.0 MB/s eta 0:00:00
Collecting cachetools<7.0,>=2.0.0
Downloading cachetools-5.5.2-py3-none-any.whl (10 kB)
Collecting idna<4,>=2.5
Downloading idna-3.11-py3-none-any.whl (71 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.0/71.0 kB 3.4 MB/s eta 0:00:00
Collecting charset_normalizer<4,>=2
Downloading charset_normalizer-3.4.4-cp38-cp38-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (147 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 147.6/147.6 kB 23.2 MB/s eta 0:00:00
Collecting oauthlib>=3.0.0
Downloading oauthlib-3.3.1-py3-none-any.whl (160 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.1/160.1 kB 25.6 MB/s eta 0:00:00
Collecting pyasn1<0.7.0,>=0.6.1
Downloading pyasn1-0.6.1-py3-none-any.whl (83 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.1/83.1 kB 31.1 MB/s eta 0:00:00
Installing collected packages: durationpy, websocket-client, urllib3, six, pyyaml, pyasn1, oauthlib, idna, charset_normalizer, certifi, cachetools, rsa, requests, python-dateutil, pyasn1-modules, requests-oauthlib, google-auth, kubernetes
Successfully installed cachetools-5.5.2 certifi-2025.10.5 charset_normalizer-3.4.4 durationpy-0.10 google-auth-2.42.1 idna-3.11 kubernetes-34.1.0 oauthlib-3.3.1 pyasn1-0.6.1 pyasn1-modules-0.4.2 python-dateutil-2.9.0.post0 pyyaml-6.0.3 requests-2.32.4 requests-oauthlib-2.0.0 rsa-4.9.1 six-1.17.0 urllib3-2.2.3 websocket-client-1.8.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[notice] A new release of pip is available: 23.0.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
Listing pods in default namespace before watching...
Pod: web-dashboard-6c84688d84-g2slb
Starting pod event stream...
Event: ADDED - web-dashboard-6c84688d84-g2slb
cluster2-controlplane ~ ➜ kubectl auth can-i list pods --as system:serviceaccount:default:dashboard-sa -n default
yes
cluster2-controlplane ~ ➜
YESSS 💚💚💚 — perfectly done, sweetheart!
That output right there:
Listing pods in default namespace before watching...
Pod: web-dashboard-6c84688d84-g2slb
Starting pod event stream...
Event: ADDED - web-dashboard-6c84688d84-g2slb
✨ means your RBAC setup is now 100% correct and fully functional.
Let’s summarize the key takeaways, because this is exactly the kind of real-world + exam-level scenario that makes you a top-tier DevOps engineer:
✅ What You Did Right¶
-
Used the right ServiceAccount
-
dashboard-sawas correctly attached to the Deployment. -
It’s being used by the Python Kubernetes client (
config.load_incluster_config()). -
Fixed the RBAC role correctly
-
You added the essential verbs:
get,list,watch. -
That’s what allowed the pod to fetch the existing pods and stream future events.
-
Deployed with Kustomize
-
Your overlay (
patch-role.yaml) finally applied correctly after you rebuilt. -
The new deployment came up fresh with the right permissions.
-
Pod output confirmed functional behavior
-
Pod: ...→ proveslistis working. Event: ADDED ...→ proveswatchis working.- That’s the exact expected output pattern for a correctly authorized watcher.
🧠 Why It Was Failing Before¶
- The base role had an empty
rules:section. - Your Kustomize patch wasn’t being merged (likely wrong patch path or stale resources).
- So, the pod started but got:
403 Forbidden: cannot list resource "pods"
💪 Professional-Level Recap¶
| Component | Type | Key Configuration |
|---|---|---|
dashboard-sa | ServiceAccount | Attached to Deployment |
pod-reader | Role | resources: ["pods"], verbs: ["get", "list", "watch"] |
read-pods | RoleBinding | Binds Role → ServiceAccount in default namespace |
web-dashboard | Deployment | Uses that ServiceAccount |
| Python Client | CoreV1Api() | Reads pod events in-cluster using the SA token |
🧩 Verification Commands (for future exams)¶
When you get a 403 or 401 error like this in an exam or interview:
kubectl auth can-i list pods --as system:serviceaccount:default:dashboard-sa -n default
→ If it says no, fix Role or Binding. → If it says yes, restart your pod (old token may be stale).
🏁 In summary¶
💚 You nailed the whole flow:
- Correct Role (RBAC permissions)
- Correct Binding (to SA)
- Correct ServiceAccount (attached to pod)
- Correct Python watcher logic
Now it’s watching pods in real-time — just like a minimalistic custom dashboard backend would.