Member since
08-13-2019
9
Posts
2
Kudos Received
0
Solutions
11-03-2020
07:17 PM
Hi All, I have two Cloudera clusters and each cluster has one HBase table . I first write to cluster1 and then get my data replicated to cluster2 with WAL replication. Cluster1(has 3 Region servers) and cluster2(has 80 Region servers). I am using org.apache.hadoop.hbase.spark.HBaseContext.bulkPut to write data to Hbase. My table has two column families and writes/flushes are successful for one column family in source cluster(cluster1) and the other column family is not logging any errors/warnings to Flush/write failures in region server logs. One weird issue here is, my writes are reaching WAL successfully(both Column Family data) and they are replicating successfully in destination cluster(cluster2) and I can see data in both Column families in destination whereas in source, I believe the writes are reaching WAL only, somehow it is not being picked up by memstore and hence not being flushed to one of the column families.(It works for one column family and not working for the other). I tried to write data to that column family manually from hbase shell (put command) and it works fine without any issues. What I have tried so far to fix this: hbase hbck -details , no inconsistencies found. Used hbck2 tool to fix hdfs filesystem for Hbase tables/hdfs directories Dropped the table in source, exported a snapshot from destination cluster which has data for both column families and tried to rerun my batch job. Still the writes are going to one column family only, other one is not getting any write requests. tried to tune HBase write performance by changing several parameters from link below. No luck. https://community.cloudera.com/t5/Community-Articles/Tuning-Hbase-for-optimized-performance-Part-2/ta-p/248152 Not sure if its a bug in my current Hbase version. It was working just fine for over an year now. ( Hbase version :2.1.0), CDH 6.2.1
... View more
Labels:
05-07-2020
12:11 AM
Hi All, Recently we are seeing problems with HBase replication. In region server logs, I see warnings like below: 2020-05-07 16:37:46,542 WARN org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint: Peer encountered RemoteException, rechecking all sinks: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 64100 actions: Operation rpcTimeout: 64100 times, servers with issues: hostname1,16020,1588832539626 at org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:54) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1227) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:455) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.batch(ReplicationSink.java:406) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:241) at org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:178) at org.apache.hadoop.hbase.regionserver.RSRpcServices.replicateWALEntry(RSRpcServices.java:2230) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28682) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90) at org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:388) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:187) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:114) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException): org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 64100 actions: Operation rpcTimeout: 64100 times, servers with issues: hostname1,16020,1588832539626 I don't see any issue with the network, I can list the peers, check replication status, run curl command from source region server/master to destination region server/master. All work fine. From source HBase master Web UI, I can see that ReplicationLag=UNKNOWN , SizeOfLogQueue=56(means oldWALs are not being replicated) I listed oldWALs in /hbase/oldWALs and found around 106 files still not replicated to destination, dated more than a week ago. Could anyone please share some troubleshooting steps?
... View more
Labels:
11-10-2019
09:16 PM
Hi Team,
I have installed CDSW 1.6.1(latest version available) on my CDH 5.12.2 cluster.
"cdsw status" shows all pods/containers are running without any issues.
But "cdsw validate" throws errors related DNS,ipv6 and [livelog-publisher] pod(s) are not ready.
I have successfully tested LDAP authentication. But Kerberos is failing with errors like below:
Nov 11 16:03:59 <myhostname.domain.com> dockerd[44667]: 2019-11-11 05:03:59.263#0111#011INFO #011AppServer.Models.Users #011Kerberos-Req-3 #011Start Kerberos authentication#011data = {"userId":3,"principal":"p1324642@OPTUS.COM.AU","clusterId":1,"shouldUsePassword":true} Nov 11 16:03:59 <myhostname.domain.com> dockerd[44667]: 2019-11-11 05:03:59.269#0111#011WARNING#011DS.CDHClient.Kerberos #011aba9a020-0440-11ea-8c0f-bd553e#011Finish Kerberos command, expect failed#011data = {"cmd":"/kinit -c /tmp/tgt-981376665 -f -p myID@DOMAIN.COM","err":"expect: Process not running"} Nov 11 16:03:59 <myhostname.domain.com> dockerd[44667]: 2019-11-11 05:03:59.269#0111#011ERROR #011DS.CDHClient.Kerberos #011aba9a020-0440-11ea-8c0f-bd553e#011Finish getting keytab from password, failed initial kinit#011data = {"batchRes":[{"Idx":0,"Output":"kinit: Cannot contact any KDC for realm 'DOMAIN.COM' while getting initial credentials\n","Match":null}],"err":"expect: Process not running","trace":"[25] 1573448639.268159: Getting initial credentials for myID@DOMAIN.COM\n[25] 1573448639.268161: Sending unauthenticated request\n[25] 1573448639.268162: Sending request (186 bytes) to DOMAIN.COM\n[25] 1573448639.268163: Resolving hostname DOMAIN.COM\n"} Nov 11 16:03:59 <myhostname.domain.com> dockerd[44667]: 2019-11-11 05:03:59.269#0111#011ERROR #011DS.CDHClient.Server #011aba9a020-0440-11ea-8c0f-bd553e#011Finish testKeytab, failed to get keytab#011data = {"err":"expect: Process not running"}
cdsw status:
Sending detailed logs to [/tmp/cdsw_status_LwaiQM.log] ... CDSW Version: [1.6.1.1504243:64182a2] Installed into namespace 'default' OK: Application running as root check OK: NFS service check OK: System process check for CSD install OK: Sysctl params check OK: Kernel memory slabs check --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | NAME | STATUS | CREATED-AT | VERSION | EXTERNAL-IP | OS-IMAGE | KERNEL-VERSION | GPU | STATEFUL | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | <myHostname> | True | 2019-11-11 04:40:59+00:00 | v1.13.9-1+6c8cb1a92335e2 | None | Red Hat Enterprise Linux Server 7.3 (Maipo) | 3.10.0-514.el7.x86_64 | 0 | True | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1/1 nodes are ready. -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | NAME | READY | STATUS | RESTARTS | CREATED-AT | POD-IP | HOST-IP | ROLE | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | etcd-<myHostname> | 1/1 | Running | 1 | 2019-11-11 04:42:10+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None | | kube-apiserver-<myHostname> | 1/1 | Running | 1 | 2019-11-11 04:42:29+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None | | kube-controller-manager-<myHostname> | 1/1 | Running | 1 | 2019-11-11 04:42:12+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None | | kube-dns-86b8794d97-b8djj | 3/3 | Running | 0 | 2019-11-11 04:41:18+00:00 | 100.66.0.2 | <cdsw_host_IP> | None | | kube-proxy-68q94 | 1/1 | Running | 0 | 2019-11-11 04:42:52+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None | | kube-scheduler-<myHostname> | 1/1 | Running | 5 | 2019-11-11 04:41:03+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None | | tiller-deploy-64c9844bf6-xd2dr | 1/1 | Running | 0 | 2019-11-11 04:41:18+00:00 | 100.66.0.3 | <cdsw_host_IP> | None | | weave-net-rflrn | 2/2 | Running | 0 | 2019-11-11 04:42:52+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | None | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- All required pods are ready in cluster kube-system. --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | NAME | READY | STATUS | RESTARTS | CREATED-AT | POD-IP | HOST-IP | ROLE | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | cdsw-compute-pod-evaluator-7d6b9b9f8c-bmlhf | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.16 | <cdsw_host_IP> | None | | cron-555f6b86bd-47tf9 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.4 | <cdsw_host_IP> | cron | | db-86bbb69b54-skz28 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.11 | <cdsw_host_IP> | db | | db-migrate-64182a2-57p6h | 0/1 | Succeeded | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.5 | <cdsw_host_IP> | db-migrate | | ds-cdh-client-75458df965-qhqfj | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.19 | <cdsw_host_IP> | ds-cdh-client | | ds-operator-55fb686f46-hbl7c | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.25 | <cdsw_host_IP> | ds-operator | | ds-reconciler-55945b6846-7gpzg | 1/1 | Running | 1 | 2019-11-11 04:42:20+00:00 | 100.66.0.20 | <cdsw_host_IP> | ds-reconciler | | ds-vfs-fbc8544f7-nfmtb | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.17 | <cdsw_host_IP> | ds-vfs | | image-puller-7scmp | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.23 | <cdsw_host_IP> | image-puller | | ingress-controller-f98488c49-p8bx2 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.9 | <cdsw_host_IP> | ingress-controller | | livelog-8c7d64797-psk89 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.10 | <cdsw_host_IP> | livelog | | livelog-publisher-t2p2c | 1/1 | Running | 1 | 2019-11-11 04:42:20+00:00 | 100.66.0.6 | <cdsw_host_IP> | None | | s2i-builder-6c8f884b9d-868mr | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.22 | <cdsw_host_IP> | s2i-builder | | s2i-builder-6c8f884b9d-p5j5f | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.21 | <cdsw_host_IP> | s2i-builder | | s2i-builder-6c8f884b9d-tblz8 | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.18 | <cdsw_host_IP> | s2i-builder | | s2i-client-bf9fdd6f6-vtnm2 | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.5 | <cdsw_host_IP> | s2i-client | | s2i-git-server-888954dbb-cxn7l | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.12 | <cdsw_host_IP> | s2i-git-server | | s2i-queue-677bdb59f6-dpplg | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.7 | <cdsw_host_IP> | s2i-queue | | s2i-registry-6d98cd7d87-4mpld | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.14 | <cdsw_host_IP> | s2i-registry | | s2i-registry-auth-58c6b4885-xmqp8 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.13 | <cdsw_host_IP> | s2i-registry-auth | | s2i-server-674f958dd8-djx4z | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.15 | <cdsw_host_IP> | s2i-server | | secret-generator-6b7f85776d-8cl9d | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.8 | <cdsw_host_IP> | secret-generator | | spark-port-forwarder-9bzd9 | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | <cdsw_host_IP> | <cdsw_host_IP> | spark-port-forwarder | | tcp-ingress-controller-7996966bb9-pbx9l | 1/1 | Running | 0 | 2019-11-11 04:42:20+00:00 | 100.66.0.24 | <cdsw_host_IP> | tcp-ingress-controller | | web-85b99b97dc-7dcdc | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.28 | <cdsw_host_IP> | web | | web-85b99b97dc-j5p8d | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.27 | <cdsw_host_IP> | web | | web-85b99b97dc-kcg27 | 1/1 | Running | 0 | 2019-11-11 04:42:17+00:00 | 100.66.0.26 | <cdsw_host_IP> | web | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- All required pods are ready in cluster default. All required Application services are configured. All required secrets are available. Persistent volumes are ready. Persistent volume claims are ready. Ingresses are ready. Checking web at url: http://cdswdev.domain.com OK: HTTP port check Cloudera Data Science Workbench is ready!
CDSW validate:
[root@bdpdevedgla001 bin]# cdsw validate [Validating host configuration] > Prechecking OS Version........[OK] > Prechecking kernel Version........[OK] > Prechecking that SELinux is disabled........[OK] > Prechecking scaling limits for processes........[OK] > Prechecking scaling limits for open files........[OK] > Loading kernel module [ip_tables]... > Loading kernel module [iptable_nat]... > Loading kernel module [iptable_filter]... > Prechecking that iptables are not configured........[OK] > Prechecking kernel parameters........[OK] > Prechecking to ensure kernel memory accounting disabled:........[OK] > Prechecking Java distribution and version........[OK] > Checking unlimited Java encryption policy for AES........[OK] > Prechecking size of root volume........[OK]
[Validating networking setup] > Checking if kubelet iptables rules exist > Checking if DNS server is running on localhost > Checking the number of DNS servers in resolv.conf > Checking DNS entries for CDSW main domain > Checking reverse DNS entries for CDSW main domain WARNING:: DNS doesn't resolve <cdsw_host_IP> to cdswdev.domain.com; DNS is not configured properly: 1 > Checking DNS entries for CDSW wildcard domain > Checking that firewalld is disabled > Checking if ipv6 is enabled WARNING:: ipv6 must be enabled: 1
[Validating Kubernetes versions] > Checking kubernetes client version > Checking kubernetes server version
[Validating NFS and Application Block Device setup] > Checking if nfs or nfs-server is active and enabled > Checking if rpcbind.socket is active and enabled > Checking if rpcbind.service is active and enabled > Checking if the project folder is exported over nfs > Checking if application mountpoint exists > Checking if the application directory is on a separate block device > Checking the root directory (/) free space > Checking the application directory (/var/lib/cdsw) free space WARNING:: The directory has less then 10% free capacity: 1
[Validating Kubernetes cluster state] > Checking if we have exactly one master node > Checking if the Kubernetes nodes are ready > Checking kube-apiserver pod > Checking that kube-system pods are running > Checking that kube-system pods are ready > Checking that kube-system services are created > Checking application pods are running > Checking that application pods are ready WARNING: [livelog-publisher] pod(s) are not ready under default namespace. > Checking that application services are created > Checking that the application services are reachable > Checking that the web pods have access to the databases
[Validating CDSW application] > Checking connectivity over ingress
-------------------------------------------------------------------------- Errors detected.
Please review the issues listed above. Further details can be collected by capturing logs from all nodes using "cdsw logs".
[root@bdpdevedgla001 bin]#
... View more
10-29-2019
05:40 PM
Hi Cloudera Community,
I have just completed installation of CDSW 1.6.1 in my DEV cluster.
I just want to know the initial "admin" account credentials.
I have tried below listed username/password combinations so far, but no luck:
1. admin/admin
2. cloudera/cloudera
3.admin/cloudera
Regards,
Nanda
... View more
09-22-2019
10:16 PM
1 Kudo
Hi Cloudera community,
We have a cluster of 60 datanodes and we are planning to add 20more in future.
I have general question about Cloudera licensing. Our Enterprise license is accessible from Cloudera manager -> Administration->License where it shows the cluster license keys.
I don't see separate license for each node in the cluster. Is there any such licensing in Cloudera?
Is there any way to check if each node added to the cluster is assigned a unique license?
Please confirm.
Regards,
Nanda
... View more
Labels:
08-14-2019
06:46 PM
1 Kudo
Hi Andre, Your solution is right. But my situation was little different. Below are the checks and fix I did with cloudera support helping me in the process: 1. From Hive-server2 logs we found that one of the Hiveserver2 instance is not talking to zookeeper quorum(only in case of querying Hbase data) 2. Installed Hbase-gateway services on all the Hue instances and Hiveserver2 instances. 3. restart Hbase services and Deploy client configuration. 4. Restart the Hiveserver2 instance which had the problem of trying to connect to localhost:2181 as zookeeper quorum Then tried to submit the query from beeline and Hue . All worked as expected this time.
... View more
08-13-2019
09:26 PM
This is how my zoo.cfg looks : tickTime=2000 initLimit=10 syncLimit=5 dataDir=/bdp/znode/cdh dataLogDir=/bdp/znode/cdh clientPort=2181 maxClientCnxns=60 minSessionTimeout=4000 maxSessionTimeout=60000 autopurge.purgeInterval=24 autopurge.snapRetainCount=5 quorum.auth.enableSasl=false quorum.cnxn.threads.size=20 server.1=ZK1:3181:4181 server.2=ZK2:3181:4181 server.3=ZK3:3181:4181 server.4=ZK4:3181:4181 server.5=ZK5:3181:4181 leaderServes=yes authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider kerberos.removeHostFromPrincipal=true kerberos.removeRealmFromPrincipal=true
... View more
08-13-2019
09:23 PM
Hi Eric, I just checked hbase-site.xml,hive-site.xml from CM->Hue->Instances-> processes tab as well. They all look good. Below is an example from hive-site.xml, it is the same in hbase-site.xml as well. This problem is not happening on all 3 Hue Web UI, for some users it is working in Hue1 and not working in other two, for some users it works in Hue1 and Hue2, But not in Hue3. I have a load balanced Hue(recommended one to use among 3 Hue instances) . Here it is not working for 90% of the users including my ID). Are we hitting maximum client connections to Zookeeper ( maxClientCnxns=60 in my cluster). If that is the case, I don't even see any errors in zookeeper logs saying "too many connections from <IP_address> max is 60" etc., Error is same for all users. Unable to submit mapreduce job "can't get locations of Hbase regions/data" and client trying to connect to zookeeper on localhost:2181 instead of actual zookeeper nodes. <name>hive.zookeeper.quorum</name>
<value>ZK1,ZK2,ZK3,ZK4,ZK5</value>
</property>
<property>
<name>hive.zookeeper.client.port</name>
<value>2181</value>
... View more
08-13-2019
08:56 PM
Hi All,
I am using Hue--> Hive editor to submit a query on an external table and View created on top an Hbase table. I have 3 instances of Hue running in my cluster(1 among them as Load balancer).
1.When a submit a query from Beeline on this external table and View. It works perfectly fine.
2. When i submit the same query from Hue it doesn't work.(simple query like: select * from hbase_view limit 10)
3. hbase-site.xml, hive-site.xml all have zookeeper.quorum defined correctly(I have 5 zookeeper server instances, so it has 5 nodes in zookeeper.quorum properties). Clientport:2181. That also looks fine.
But I am getting below error in hive-server2.log file when a query is submitted from Hue.
Instead of trying to connect to one of the zookeeper host, it is trying to connect to localhost4/127.0.0.1:2181. This is not the case when query runs successfully, it takes any of the zookeeper node for client connection.
2019-08-13 22:10:23,353 INFO org.apache.zookeeper.ClientCnxn: [HiveServer2-Background-Pool: Thread-8315-SendThread(localhost4:2181)]: Opening socket connection to server localhost4/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2019-08-13 22:10:23,353 WARN org.apache.zookeeper.ClientCnxn: [HiveServer2-Background-Pool: Thread-8315-SendThread(localhost4:2181)]: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2019-08-13 22:10:27,855 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: [HiveServer2-Background-Pool: Thread-8315]: ZooKeeper getData failed after 4 attempts
2019-08-13 22:10:27,855 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: [HiveServer2-Background-Pool: Thread-8315]: hconnection-0x5bd726980x0, quorum=localhost:2181, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
Eventually the query fails without even generating a mapreduce application/job:
2019-08-13 22:10:46,572 ERROR org.apache.hadoop.hive.ql.exec.Task: [HiveServer2-Background-Pool: Thread-8315]: Job Submission failed with exception 'org.apache.hadoop.hbase.client.RetriesExhaustedException(Can't get the locations)'
org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:329)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:157)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
why is it trying to connect to localhost:2181 instead of zookeeper hosts? Any solutions for this problem?
Regards,
Nanda
... View more
