Hi All,   Recently we are seeing problems with HBase replication.   In region server logs, I see warnings like below:   2020-05-07 16:37:46,542 WARN org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint: Peer encountered RemoteException, rechecking all sinks: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 64100 actions: Operation rpcTimeout: 64100 times, servers with issues: hostname1,16020,1588832539626 at org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:54) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1227) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:455) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.batch(ReplicationSink.java:406) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:241) at org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:178) at org.apache.hadoop.hbase.regionserver.RSRpcServices.replicateWALEntry(RSRpcServices.java:2230) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28682) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90) at org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:388) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:187) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:114) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException): org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 64100 actions: Operation rpcTimeout: 64100 times, servers with issues: hostname1,16020,1588832539626     I don't see any issue with the network,  I can list the peers, check replication status, run curl command from source region server/master to destination region server/master. All work fine.   From source HBase master Web UI, I can see that  ReplicationLag=UNKNOWN , SizeOfLogQueue=56(means oldWALs are not being replicated)    I listed oldWALs in /hbase/oldWALs and found around 106 files still not replicated to destination, dated more than a week ago.     Could anyone please share some troubleshooting steps?     ... View more

Hi Team,   I have installed CDSW 1.6.1(latest version available) on my CDH 5.12.2 cluster. "cdsw status" shows all pods/containers are running without any issues. But "cdsw validate" throws errors related DNS,ipv6 and [livelog-publisher] pod(s) are not ready.   I have successfully tested LDAP authentication. But Kerberos is failing with errors like below:   Nov 11 16:03:59 <myhostname.domain.com> dockerd[44667]: 2019-11-11 05:03:59.263#0111#011INFO #011AppServer.Models.Users #011Kerberos-Req-3 #011Start Kerberos authentication#011data = {"userId":3,"principal":"p1324642@OPTUS.COM.AU","clusterId":1,"shouldUsePassword":true} Nov 11 16:03:59 <myhostname.domain.com> dockerd[44667]: 2019-11-11 05:03:59.269#0111#011WARNING#011DS.CDHClient.Kerberos #011aba9a020-0440-11ea-8c0f-bd553e#011Finish Kerberos command, expect failed#011data = {"cmd":"/kinit -c /tmp/tgt-981376665 -f -p myID@DOMAIN.COM","err":"expect: Process not running"} Nov 11 16:03:59 <myhostname.domain.com> dockerd[44667]: 2019-11-11 05:03:59.269#0111#011ERROR #011DS.CDHClient.Kerberos #011aba9a020-0440-11ea-8c0f-bd553e#011Finish getting keytab from password, failed initial kinit#011data = {"batchRes":[{"Idx":0,"Output":"kinit: Cannot contact any KDC for realm 'DOMAIN.COM' while getting initial credentials\n","Match":null}],"err":"expect: Process not running","trace":"[25] 1573448639.268159: Getting initial credentials for myID@DOMAIN.COM\n[25] 1573448639.268161: Sending unauthenticated request\n[25] 1573448639.268162: Sending request (186 bytes) to DOMAIN.COM\n[25] 1573448639.268163: Resolving hostname DOMAIN.COM\n"} Nov 11 16:03:59 <myhostname.domain.com> dockerd[44667]: 2019-11-11 05:03:59.269#0111#011ERROR #011DS.CDHClient.Server #011aba9a020-0440-11ea-8c0f-bd553e#011Finish testKeytab, failed to get keytab#011data = {"err":"expect: Process not running"}     cdsw status:   Sending detailed logs to [/tmp/cdsw_status_LwaiQM.log] ... CDSW Version: [1.6.1.1504243:64182a2] Installed into namespace 'default' OK: Application running as root check OK: NFS service check OK: System process check for CSD install OK: Sysctl params check OK: Kernel memory slabs check --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | NAME | STATUS | CREATED-AT | VERSION | EXTERNAL-IP | OS-IMAGE | KERNEL-VERSION | GPU | STATEFUL | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | <myHostname> | True | 2019-11-11 04:40:59+00:00 | v1.13.9-1+6c8cb1a92335e2 | None | Red Hat Enterprise Linux Server 7.3 (Maipo) | 3.10.0-514.el7.x86_64 | 0 | True | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1/1 nodes are ready. -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | NAME | READY | STATUS | RESTARTS | CREATED-AT | POD-IP | HOST-IP | ROLE | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | etcd-<myHostname> | 1/1 | Running | 1 | 2019-11-11 04:42:10+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None | | kube-apiserver-<myHostname> | 1/1 | Running | 1 | 2019-11-11 04:42:29+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None | | kube-controller-manager-<myHostname> | 1/1 | Running | 1 | 2019-11-11 04:42:12+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None | | kube-dns-86b8794d97-b8djj | 3/3 | Running | 0 | 2019-11-11 04:41:18+00:00 | 100.66.0.2 | <cdsw_host_IP> | None | | kube-proxy-68q94 | 1/1 | Running | 0 | 2019-11-11 04:42:52+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None | | kube-scheduler-<myHostname> | 1/1 | Running | 5 | 2019-11-11 04:41:03+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None | | tiller-deploy-64c9844bf6-xd2dr | 1/1 | Running | 0 | 2019-11-11 04:41:18+00:00 | 100.66.0.3 | <cdsw_host_IP> | None | | weave-net-rflrn | 2/2 | Running | 0 | 2019-11-11 04:42:52+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- All required pods are ready in cluster kube-system. --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | NAME | READY | STATUS | RESTARTS | CREATED-AT | POD-IP | HOST-IP | ROLE | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | cdsw-compute-pod-evaluator-7d6b9b9f8c-bmlhf | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.16 | <cdsw_host_IP> | None | | cron-555f6b86bd-47tf9 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.4 | <cdsw_host_IP> | cron | | db-86bbb69b54-skz28 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.11 | <cdsw_host_IP> | db | | db-migrate-64182a2-57p6h | 0/1 | Succeeded | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.5 | <cdsw_host_IP> | db-migrate | | ds-cdh-client-75458df965-qhqfj | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.19 | <cdsw_host_IP> | ds-cdh-client | | ds-operator-55fb686f46-hbl7c | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.25 | <cdsw_host_IP> | ds-operator | | ds-reconciler-55945b6846-7gpzg | 1/1 | Running | 1 | 2019-11-11 04:42:20+00:00 | 100.66.0.20 | <cdsw_host_IP> | ds-reconciler | | ds-vfs-fbc8544f7-nfmtb | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.17 | <cdsw_host_IP> | ds-vfs | | image-puller-7scmp | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.23 | <cdsw_host_IP> | image-puller | | ingress-controller-f98488c49-p8bx2 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.9 | <cdsw_host_IP> | ingress-controller | | livelog-8c7d64797-psk89 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.10 | <cdsw_host_IP> | livelog | | livelog-publisher-t2p2c | 1/1 | Running | 1 | 2019-11-11 04:42:20+00:00 | 100.66.0.6 | <cdsw_host_IP> | None | | s2i-builder-6c8f884b9d-868mr | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.22 | <cdsw_host_IP> | s2i-builder | | s2i-builder-6c8f884b9d-p5j5f | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.21 | <cdsw_host_IP> | s2i-builder | | s2i-builder-6c8f884b9d-tblz8 | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.18 | <cdsw_host_IP> | s2i-builder | | s2i-client-bf9fdd6f6-vtnm2 | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.5 | <cdsw_host_IP> | s2i-client | | s2i-git-server-888954dbb-cxn7l | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.12 | <cdsw_host_IP> | s2i-git-server | | s2i-queue-677bdb59f6-dpplg | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.7 | <cdsw_host_IP> | s2i-queue | | s2i-registry-6d98cd7d87-4mpld | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.14 | <cdsw_host_IP> | s2i-registry | | s2i-registry-auth-58c6b4885-xmqp8 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.13 | <cdsw_host_IP> | s2i-registry-auth | | s2i-server-674f958dd8-djx4z | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.15 | <cdsw_host_IP> | s2i-server | | secret-generator-6b7f85776d-8cl9d | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.8 | <cdsw_host_IP> | secret-generator | | spark-port-forwarder-9bzd9 | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | spark-port-forwarder | | tcp-ingress-controller-7996966bb9-pbx9l | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.24 | <cdsw_host_IP> | tcp-ingress-controller | | web-85b99b97dc-7dcdc | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.28 | <cdsw_host_IP> | web | | web-85b99b97dc-j5p8d | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.27 | <cdsw_host_IP> | web | | web-85b99b97dc-kcg27 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.26 | <cdsw_host_IP> | web | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- All required pods are ready in cluster default. All required Application services are configured. All required secrets are available. Persistent volumes are ready. Persistent volume claims are ready. Ingresses are ready. Checking web at url: http://cdswdev.domain.com OK: HTTP port check Cloudera Data Science Workbench is ready!       CDSW validate:   [root@bdpdevedgla001 bin]# cdsw validate [Validating host configuration] > Prechecking OS Version........[OK] > Prechecking kernel Version........[OK] > Prechecking that SELinux is disabled........[OK] > Prechecking scaling limits for processes........[OK] > Prechecking scaling limits for open files........[OK] > Loading kernel module [ip_tables]... > Loading kernel module [iptable_nat]... > Loading kernel module [iptable_filter]... > Prechecking that iptables are not configured........[OK] > Prechecking kernel parameters........[OK] > Prechecking to ensure kernel memory accounting disabled:........[OK] > Prechecking Java distribution and version........[OK] > Checking unlimited Java encryption policy for AES........[OK] > Prechecking size of root volume........[OK] [Validating networking setup] > Checking if kubelet iptables rules exist > Checking if DNS server is running on localhost > Checking the number of DNS servers in resolv.conf > Checking DNS entries for CDSW main domain > Checking reverse DNS entries for CDSW main domain WARNING:: DNS doesn't resolve <cdsw_host_IP> to cdswdev.domain.com; DNS is not configured properly: 1 > Checking DNS entries for CDSW wildcard domain > Checking that firewalld is disabled > Checking if ipv6 is enabled WARNING:: ipv6 must be enabled: 1 [Validating Kubernetes versions] > Checking kubernetes client version > Checking kubernetes server version [Validating NFS and Application Block Device setup] > Checking if nfs or nfs-server is active and enabled > Checking if rpcbind.socket is active and enabled > Checking if rpcbind.service is active and enabled > Checking if the project folder is exported over nfs > Checking if application mountpoint exists > Checking if the application directory is on a separate block device > Checking the root directory (/) free space > Checking the application directory (/var/lib/cdsw) free space WARNING:: The directory has less then 10% free capacity: 1 [Validating Kubernetes cluster state] > Checking if we have exactly one master node > Checking if the Kubernetes nodes are ready > Checking kube-apiserver pod > Checking that kube-system pods are running > Checking that kube-system pods are ready > Checking that kube-system services are created > Checking application pods are running > Checking that application pods are ready WARNING: [livelog-publisher] pod(s) are not ready under default namespace. > Checking that application services are created > Checking that the application services are reachable > Checking that the web pods have access to the databases [Validating CDSW application] > Checking connectivity over ingress -------------------------------------------------------------------------- Errors detected. Please review the issues listed above. Further details can be collected by capturing logs from all nodes using "cdsw logs". [root@bdpdevedgla001 bin]#     ... View more

Hi Cloudera community,   We have a cluster of 60 datanodes and we are planning to add 20more in future. I have general question about Cloudera licensing. Our Enterprise license is accessible from Cloudera manager -> Administration->License where it shows the cluster license keys.   I don't see separate license for each node in the cluster. Is there any such licensing in Cloudera? Is there any way to check if each node added to the cluster is assigned a unique license?  Please confirm.   Regards, Nanda    ... View more