Hi All,
Need help in fixing the bellow errors in our newly created CDSW cluster.
all the services of the CDSW cluster are running.
I guess I need to bring the kubernetes service up. "The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port?"
health errors: CDSW StatusSuppress... Failed to run CDSW Nodes Check. * Failed to run CDSW system pods check. * Failed to run CDSW application pods check. * Failed to run CDSW services check. * Failed to run CDSW secrets check. * Failed to run CDSW persistent volumes check. * Failed to run ...
--------------------------------------------
[root@server ~]# cdsw logs -x
Generating Cloudera Data Science Workbench diagnostic bundle...
Collecting basic system info...
Collecting kernel parameters...
Collecting kernel messages...
Collecting the list of kernel modules...
Collecting the list of systemd units...
Collecting cdsw details...
Collecting application configuration...
Collecting disks information...
Collecting Hadoop configuration...
Collecting network information...
Collecting system service statuses...
Collecting nfs information...
Collecting Docker info...
Collecting Kubernetes info...
Collecting Helm info...
Collecting custom patches...
cp: cannot stat ‘/etc/cdsw/patches’: No such file or directory
Collecting Kubelet logs...
Collecting CDSW Host Controller logs...
Collecting system logs...
Collecting Kubernetes cluster info dump...
ls: cannot access cdsw-logs-cpcinchdv010813-2021-03-23--20-47-00/k8s-cluster-info/*/*/logs.txt: No such file or directory
Exporting user ids...
The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port?
The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port?
error: pod or type/name must be specified
error: pod or type/name must be specified
Collecting health logs...
Collecting event logs...
The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port?
error: Filespec must match the canonical format: [[namespace/]pod:]file/path
Exporting metrics...
The connection to the server 10.127.116.126:6443 was refused - did you specify the right host or port?
ERROR:: Unable to get service account credentials. Provide SERVICE_ACCOUNT_SECRET or run on master node.: 2
Producing logs tarball...
Logs saved to: cdsw-logs-cpcinchdv010813-2021-03-23--20-47-00.tar.gz
Cleaning up...
-------------------------------------------------------------------------------------------
[root@server ~]# cdsw validate
[Validating host configuration]
> Prechecking OS Version........[OK]
> Prechecking kernel Version........[OK]
> Prechecking that SELinux is disabled........[OK]
> Prechecking scaling limits for processes........[OK]
> Prechecking scaling limits for open files........
WARNING: Cloudera Data Science Workbench recommends that all users have a max-open-files limit set to 1048576.
It is currently set to [1024] as per 'ulimit -n'
Press enter to continue
> Loading kernel module [ip_tables]...
> Loading kernel module [iptable_nat]...
> Loading kernel module [iptable_filter]...
> Prechecking that iptables are not configured........[OK]
> Prechecking kernel parameters........[OK]
> Prechecking to ensure kernel memory accounting disabled:........[OK]
> Prechecking Java distribution and version........[OK]
> Checking unlimited Java encryption policy for AES........[OK]
> Prechecking size of root volume........
WARNING: The recommended minimum root volume size is 100G. Press enter to continue
[Validating networking setup]
> Checking if kubelet iptables rules exist
The following chains are missing from iptables: [KUBE-EXTERNAL-SERVICES, WEAVE-NPC-EGRESS, WEAVE-NPC, WEAVE-NPC-EGRESS-ACCEPT, KUBE-SERVICES, WEAVE-NPC-INGRESS, WEAVE-NPC-EGRESS-DEFAULT, WEAVE-NPC-DEFAULT, WEAVE-NPC-EGRESS-CUSTOM]
WARNING:: Verification of iptables rules failed: 1
> Checking if DNS server is running on localhost
> Checking the number of DNS servers in resolv.conf
> Checking DNS entries for CDSW main domain
> Checking reverse DNS entries for CDSW main domain
WARNING:: DNS doesn't resolve 10.127.116.126 to cdsw.cts.com; DNS is not configured properly: 1
> Checking DNS entries for CDSW wildcard domain
> Checking that firewalld is disabled
> Checking if ipv6 is enabled
[Validating Kubernetes versions]
> Checking kubernetes client version
> Checking kubernetes server version
WARNING:: Kubernetes server is not running, version couldn't be checked.: 1
[Validating NFS and Application Block Device setup]
> Checking if nfs or nfs-server is active and enabled
> Checking if rpcbind.socket is active and enabled
> Checking if rpcbind.service is active and enabled
> Checking if the project folder is exported over nfs
WARNING:: The projects folder /var/lib/cdsw/current/projects must be exported over nfs: 1
> Checking if application mountpoint exists
> Checking if the application directory is on a separate block device
> Checking the root directory (/) free space
WARNING:: The directory has less then 10% free capacity: 1
> Checking the application directory (/var/lib/cdsw) free space
WARNING:: The directory has less then 10% free capacity: 1
[Validating Kubernetes cluster state]
> Checking if we have exactly one master node
WARNING:: There must be exactly one Kubernetes node labelled 'stateful=true': 1
> Checking if the Kubernetes nodes are ready
> Checking kube-apiserver pod
WARNING: Unable to reach k8s pod kube-apiserver.
WARNING: [kube-apiserver] pod(s) are not ready under kube-system namespace.
WARNING: Unable to bring up kube-apiserver in the kube-system cluster. Skipping other checks..
[Validating CDSW application]
> Checking connectivity over ingress
WARNING:: Could not curl the application over the ingress controller: 7
--------------------------------------------------------------------------
Errors detected.
Please review the issues listed above. Further details can be collected by
capturing logs from all nodes using "cdsw logs".
--------------------------------------------------------------------------
cdsw status
Sending detailed logs to [/tmp/cdsw_status_rwPmR6.log] ...
CDSW Version: [1.9.0.7802354:5a39a73]
Installed into namespace 'default'
OK: Application running as root check
OK: NFS service check
OK: System process check for CSD install
OK: Sysctl params check
OK: Kernel memory slabs check
Failed to run CDSW Nodes Check.
Failed to run CDSW system pods check.
Failed to run CDSW application pods check.
Failed to run CDSW services check.
Failed to run CDSW secrets check.
Failed to run CDSW persistent volumes check.
Failed to run CDSW persistent volumes claims check.
Failed to run CDSW Ingresses check.
Checking web at url: http://cdsw.cts.com
Web is not yet up.
Cloudera Data Science Workbench is not ready yet
