Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ERROR Failed to run CDSW persistent volumes claims check.

ERROR Failed to run CDSW persistent volumes claims check.

New Contributor

Has anyone else here ran into a similar issue with their worker nodes?  My master node doesnt have this error and I configured it exactly as I did to the slaves. 

 

ERROR Failed to run CDSW persistent volumes claims check.

 

 

 

2017-12-14 09:47:33,204 INFO All required pods are ready in cluster default.
2017-12-14 09:47:33,212 INFO All required Application services are configured.
2017-12-14 09:47:33,215 ERROR Failed to run CDSW config maps check.
Traceback (most recent call last):
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 42, in wrapper
result = func(*args, **kwargs)
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 284, in _check_config_maps
config_maps = api.list_config_map_for_all_namespaces()
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 11764, in list_config_map_for_all_namespaces
(data) = self.list_config_map_for_all_namespaces_with_http_info(**kwargs)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 11860, in list_config_map_for_all_namespaces_with_http_info
collection_formats=collection_formats)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
_preload_content, _request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
_request_timeout=_request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
headers=headers)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
query_params=query_params)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 240, in request
raise ApiException(http_resp=r)
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 14 Dec 2017 14:47:33 GMT', 'Content-Length': '83', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: User "system:node:cdswpoc3.silver.com" cannot list configmaps at the cluster scope.

2017-12-14 09:47:33,226 ERROR Failed to run CDSW secrets check.
Traceback (most recent call last):
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 42, in wrapper
result = func(*args, **kwargs)
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 294, in _check_secrets
secrets = api.list_secret_for_all_namespaces()
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14872, in list_secret_for_all_namespaces
(data) = self.list_secret_for_all_namespaces_with_http_info(**kwargs)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14968, in list_secret_for_all_namespaces_with_http_info
collection_formats=collection_formats)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
_preload_content, _request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
_request_timeout=_request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
headers=headers)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
query_params=query_params)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 240, in request
raise ApiException(http_resp=r)
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 14 Dec 2017 14:47:33 GMT', 'Content-Length': '80', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: User "system:node:cdswpoc3.silver.com" cannot list secrets at the cluster scope.

2017-12-14 09:47:33,229 ERROR Failed to run CDSW persistent volumes check.
Traceback (most recent call last):
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 42, in wrapper
result = func(*args, **kwargs)
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 303, in _check_persistent_volume
pvs = api.list_persistent_volume(field_selector='metadata.name=projects-share')
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14116, in list_persistent_volume
(data) = self.list_persistent_volume_with_http_info(**kwargs)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14212, in list_persistent_volume_with_http_info
collection_formats=collection_formats)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
_preload_content, _request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
_request_timeout=_request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
headers=headers)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
query_params=query_params)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 240, in request
raise ApiException(http_resp=r)
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 14 Dec 2017 14:47:33 GMT', 'Content-Length': '90', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: User "system:node:cdswpoc3.silver.com" cannot list persistentvolumes at the cluster scope.

2017-12-14 09:47:33,232 ERROR Failed to run CDSW persistent volumes claims check.
Traceback (most recent call last):
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 42, in wrapper
result = func(*args, **kwargs)
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 310, in _check_persistent_volume_claims
pvcs = api.list_persistent_volume_claim_for_all_namespaces(field_selector='metadata.name=projects-claim')
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14242, in list_persistent_volume_claim_for_all_namespaces
(data) = self.list_persistent_volume_claim_for_all_namespaces_with_http_info(**kwargs)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14338, in list_persistent_volume_claim_for_all_namespaces_with_http_info
collection_formats=collection_formats)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
_preload_content, _request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
_request_timeout=_request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
headers=headers)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
query_params=query_params)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 240, in request
raise ApiException(http_resp=r)
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 14 Dec 2017 14:47:33 GMT', 'Content-Length': '95', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: User "system:node:cdswpoc3.silver.com" cannot list persistentvolumeclaims at the cluster scope.

2017-12-14 09:47:33,235 ERROR Failed to run CDSW Ingresses check.
Traceback (most recent call last):
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 42, in wrapper
result = func(*args, **kwargs)
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 317, in _check_ingress
ingresses = client.ExtensionsV1beta1Api().list_ingress_for_all_namespaces(field_selector='metadata.name=client')
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 3189, in list_ingress_for_all_namespaces
(data) = self.list_ingress_for_all_namespaces_with_http_info(**kwargs)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 3285, in list_ingress_for_all_namespaces_with_http_info
collection_formats=collection_formats)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
_preload_content, _request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
_request_timeout=_request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
headers=headers)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
query_params=query_params)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 240, in request
raise ApiException(http_resp=r)
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 14 Dec 2017 14:47:33 GMT', 'Content-Length': '93', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: User "system:node:cdswpoc3.silver.com" cannot list ingresses.extensions at the cluster scope.

2017-12-14 09:47:33,237 INFO Checking web at url: http://cdsw.silver.com
2017-12-14 09:47:33,248 INFO OK: HTTP port check

 

 

 

Here is how my mounts are setup. 

 

[root@cdswpoc3 tmp]# pvs

  PV         VG     Fmt  Attr PSize    PFree

  /dev/vda3  rootvg lvm2 a--    49.00g     0

  /dev/vda4  rootvg lvm2 a--    50.00g 14.54g

  /dev/vdb   docker lvm2 a--  1024.00g 30.73g

[root@cdswpoc3 tmp]# vgs

  VG     #PV #LV #SN Attr   VSize    VFree

  docker   1   1   0 wz--n- 1024.00g 30.73g

  rootvg   2   7   0 wz--n-   98.99g 14.54g

[root@cdswpoc3 tmp]# lvs

  LV       VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert

  thinpool docker twi-aot--- 972.79g             1.45   0.07                            

  lvhome   rootvg -wi-ao----   2.00g                                                    

  lvopt    rootvg -wi-ao----  10.84g                                                    

  lvroot   rootvg -wi-ao----  46.92g                                                    

  lvswap   rootvg -wi-ao----   4.00g                                                    

  lvtmp    rootvg -wi-ao----   5.91g                                                    

  lvusr    rootvg -wi-ao----   4.00g                                                    

  lvvar    rootvg -wi-ao----  10.79g 

 

They seem to be connected. 

[root@cdswpoc3 tmp]# kubectl get nodes

NAME                  STATUS    AGE       VERSION

cdswpoc1.silver.com   Ready     20h       v1.6.11

cdswpoc2.silver.com   Ready     18h       v1.6.11

cdswpoc3.silver.com   Ready     18h       v1.6.11