Reply
New Contributor
Posts: 4
Registered: ‎12-14-2017

ERROR Failed to run CDSW persistent volumes claims check.

[ Edited ]

Has anyone else here ran into a similar issue with their worker nodes?  My master node doesnt have this error and I configured it exactly as I did to the slaves. 

 

ERROR Failed to run CDSW persistent volumes claims check.

 

 

 

2017-12-14 09:47:33,204 INFO All required pods are ready in cluster default.
2017-12-14 09:47:33,212 INFO All required Application services are configured.
2017-12-14 09:47:33,215 ERROR Failed to run CDSW config maps check.
Traceback (most recent call last):
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 42, in wrapper
result = func(*args, **kwargs)
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 284, in _check_config_maps
config_maps = api.list_config_map_for_all_namespaces()
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 11764, in list_config_map_for_all_namespaces
(data) = self.list_config_map_for_all_namespaces_with_http_info(**kwargs)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 11860, in list_config_map_for_all_namespaces_with_http_info
collection_formats=collection_formats)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
_preload_content, _request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
_request_timeout=_request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
headers=headers)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
query_params=query_params)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 240, in request
raise ApiException(http_resp=r)
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 14 Dec 2017 14:47:33 GMT', 'Content-Length': '83', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: User "system:node:cdswpoc3.silver.com" cannot list configmaps at the cluster scope.

2017-12-14 09:47:33,226 ERROR Failed to run CDSW secrets check.
Traceback (most recent call last):
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 42, in wrapper
result = func(*args, **kwargs)
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 294, in _check_secrets
secrets = api.list_secret_for_all_namespaces()
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14872, in list_secret_for_all_namespaces
(data) = self.list_secret_for_all_namespaces_with_http_info(**kwargs)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14968, in list_secret_for_all_namespaces_with_http_info
collection_formats=collection_formats)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
_preload_content, _request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
_request_timeout=_request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
headers=headers)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
query_params=query_params)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 240, in request
raise ApiException(http_resp=r)
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 14 Dec 2017 14:47:33 GMT', 'Content-Length': '80', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: User "system:node:cdswpoc3.silver.com" cannot list secrets at the cluster scope.

2017-12-14 09:47:33,229 ERROR Failed to run CDSW persistent volumes check.
Traceback (most recent call last):
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 42, in wrapper
result = func(*args, **kwargs)
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 303, in _check_persistent_volume
pvs = api.list_persistent_volume(field_selector='metadata.name=projects-share')
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14116, in list_persistent_volume
(data) = self.list_persistent_volume_with_http_info(**kwargs)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14212, in list_persistent_volume_with_http_info
collection_formats=collection_formats)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
_preload_content, _request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
_request_timeout=_request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
headers=headers)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
query_params=query_params)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 240, in request
raise ApiException(http_resp=r)
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 14 Dec 2017 14:47:33 GMT', 'Content-Length': '90', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: User "system:node:cdswpoc3.silver.com" cannot list persistentvolumes at the cluster scope.

2017-12-14 09:47:33,232 ERROR Failed to run CDSW persistent volumes claims check.
Traceback (most recent call last):
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 42, in wrapper
result = func(*args, **kwargs)
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 310, in _check_persistent_volume_claims
pvcs = api.list_persistent_volume_claim_for_all_namespaces(field_selector='metadata.name=projects-claim')
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14242, in list_persistent_volume_claim_for_all_namespaces
(data) = self.list_persistent_volume_claim_for_all_namespaces_with_http_info(**kwargs)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14338, in list_persistent_volume_claim_for_all_namespaces_with_http_info
collection_formats=collection_formats)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
_preload_content, _request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
_request_timeout=_request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
headers=headers)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
query_params=query_params)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 240, in request
raise ApiException(http_resp=r)
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 14 Dec 2017 14:47:33 GMT', 'Content-Length': '95', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: User "system:node:cdswpoc3.silver.com" cannot list persistentvolumeclaims at the cluster scope.

2017-12-14 09:47:33,235 ERROR Failed to run CDSW Ingresses check.
Traceback (most recent call last):
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 42, in wrapper
result = func(*args, **kwargs)
File "/etc/cdsw/cdsw_admin/cdsw/status.py", line 317, in _check_ingress
ingresses = client.ExtensionsV1beta1Api().list_ingress_for_all_namespaces(field_selector='metadata.name=client')
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 3189, in list_ingress_for_all_namespaces
(data) = self.list_ingress_for_all_namespaces_with_http_info(**kwargs)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 3285, in list_ingress_for_all_namespaces_with_http_info
collection_formats=collection_formats)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
_preload_content, _request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
_request_timeout=_request_timeout)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
headers=headers)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
query_params=query_params)
File "/etc/cdsw/cdsw_admin/env/lib64/python2.7/site-packages/kubernetes/client/rest.py", line 240, in request
raise ApiException(http_resp=r)
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 14 Dec 2017 14:47:33 GMT', 'Content-Length': '93', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: User "system:node:cdswpoc3.silver.com" cannot list ingresses.extensions at the cluster scope.

2017-12-14 09:47:33,237 INFO Checking web at url: http://cdsw.silver.com
2017-12-14 09:47:33,248 INFO OK: HTTP port check

 

 

 

Here is how my mounts are setup. 

 

[root@cdswpoc3 tmp]# pvs

  PV         VG     Fmt  Attr PSize    PFree

  /dev/vda3  rootvg lvm2 a--    49.00g     0

  /dev/vda4  rootvg lvm2 a--    50.00g 14.54g

  /dev/vdb   docker lvm2 a--  1024.00g 30.73g

[root@cdswpoc3 tmp]# vgs

  VG     #PV #LV #SN Attr   VSize    VFree

  docker   1   1   0 wz--n- 1024.00g 30.73g

  rootvg   2   7   0 wz--n-   98.99g 14.54g

[root@cdswpoc3 tmp]# lvs

  LV       VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert

  thinpool docker twi-aot--- 972.79g             1.45   0.07                            

  lvhome   rootvg -wi-ao----   2.00g                                                    

  lvopt    rootvg -wi-ao----  10.84g                                                    

  lvroot   rootvg -wi-ao----  46.92g                                                    

  lvswap   rootvg -wi-ao----   4.00g                                                    

  lvtmp    rootvg -wi-ao----   5.91g                                                    

  lvusr    rootvg -wi-ao----   4.00g                                                    

  lvvar    rootvg -wi-ao----  10.79g 

 

They seem to be connected. 

[root@cdswpoc3 tmp]# kubectl get nodes

NAME                  STATUS    AGE       VERSION

cdswpoc1.silver.com   Ready     20h       v1.6.11

cdswpoc2.silver.com   Ready     18h       v1.6.11

cdswpoc3.silver.com   Ready     18h       v1.6.11

Announcements