Cloudera Community
scratch28
user rank
Contributor

hbase Master failed to become active master

by Contributor in Support Questions
‎08-16-2017 03:54 PM
‎08-16-2017 03:54 PM
My primary master died due to scheduled maintenance and my backup master failed to kick in  CM Agent has tried to start it up but could not initialize the namespace table  After several manual efforts where I followed https://community.cloudera.com/t5/Storage-Random-Access-HDFS/HBase-Master-Failed-to-become-active-master/m-p/27186#M1225   I did the following only  rmr /hbase/meta-region-server rmr /hbase/rs rmr /hbase/splitWAL rmr /hbase/backup-masters rmr /hbase/table-lock rmr /hbase/flush-table-proc rmr /hbase/region-in-transition rmr /hbase/running rmr /hbase/balancer rmr /hbase/recovering-regions rmr /hbase/draining rmr /hbase/namespace rmr /hbase/hbaseid rmr /hbase/table I got the a master to come up after setting hbase.master.namespace.init.timeout to some absurd value I see the master registering dead region servers (though I cannot find where it pick them up, not in the WAL, Archive or data) and I see the master registering the following  Starting namespace manager (since 1hrs, 20mins, 5sec ago) even though cloudera manager shows healthy   list the catalog in hbase shell gives me the following error   hbase(main):004:0> list TABLE ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2373) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableNames(MasterRpcServices.java:907) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55650) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2182) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165) fsck /hbase -files -blocks shows healthy hbck  shows zero inconsistencies   I am on version  Hadoop 2.6.0+cdh5.11.1+2400 HBase 1.2.0+cdh5.11.1+319 I did have a master colocated with a region server and was wondering if I ran into this https://issues.apache.org/jira/browse/HBASE-14861 and then this  https://issues.apache.org/jira/browse/HBASE-14664 as the cause of the failed backup kicking in    But I cannot determine why I am getting namespace manager would take so long to initialize   ... View more
Labels:

Impala running slow after upgrade

by Contributor in Support Questions
‎08-10-2017 11:39 AM
‎08-10-2017 11:39 AM
Recently upgraded Cloudera from 5.4 to 5.12 but running 5.11 components so Impala 2.8.0+cdh5.11.1+0   Also recently moved Hive Metastore database from postgres to MySQL Currently I have found that Impala is reacting slowly to executing queiries and executing them at all I have a 7 node cluster and only 5 real users using Impala  In searching the logs  I am getting the following error    E0810 18:01:54.606694 25271 ShortCircuitCache.java:215] ShortCircuitCache(0xba3834e): failed to release short-circuit shared memory slot Slot(slotIdx=17, shm=DfsClientShm(5db0b20a1ec13fd9af815db2d22cd894)) by sending ReleaseShortCircuitAccessRequestProto to /var/run/hdfs-sockets/dn. Closing shared memory segment. Java exception follows: java.io.IOException: ERROR_INVALID: there is no shared memory segment registered with shmId 5db0b20a1ec13fd9af815db2d22cd894 at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) I also see the following as well cc:278 channel send status: Sender timed out waiting for receiver fragment instance: 3d4e8ea794a486e7:a74d612000000010 ... View more
Labels:

Re: HUE no longer sees impala or have after restar...

by Contributor in Support Questions
‎04-05-2017 06:57 PM
‎04-05-2017 06:57 PM
Fixed Romain posted on Hue user group that I had a conflict in python libraries and indeed I did I had Impyla and Airflow installed as part of the main lib/distro instead of a controlled "virtualized" environment This caused the  conflict.     Unfortunately both Impyla and Airflow could not be uninstalled with a simple pip uninstall command Had to manually compare directories from my staging environment and remove several of the directories that could be potentially conflicting These include: hive_metastore hive_thrift_py pyhive hive_serde impala PyHive TCLIService hive_service impyla thrift Once the directories where removed from /usr/local/lib/python2.7/dist-packages and restarted HUE, it was querying again.    Many kudos to Romain ... View more

HUE no longer sees impala or have after restart

by Contributor in Support Questions
‎04-03-2017 06:39 AM
‎04-03-2017 06:39 AM
I had Hive and Impala working on HUE before  Versions HUE 3.7 Clouderan Manager 5.4.11   I recently restarted HUE and HUE no longer sees Impala or hive       Configuration files located in /run/cloudera-scm-agent/process/7850-hue-HUE_SERVER Potential misconfiguration detected. Fix and restart Hue. Hive Editor The application won't work without a running HiveServer2. Impala Editor No available Impalad to send queries to.      When I run a query I get cannot import name TGetLogReq     Error log shows the following    [03/Apr/2017 00:16:53 -0700] wsgiserver ERROR WSGI (<WorkerThread(CP WSGIServer Thread-19, started 140018618201856)>) error: [('SSL routines', 'SSL23_WRITE', 'ssl handshake failure')] Traceback (most recent call last): File "/opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 1304, in run conn.communicate() File "/opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 1214, in communicate req.simple_response("408 Request Timeout") File "/opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 618, in simple_response self.wfile.sendall("".join(buf)) File "/opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 1139, in sendall return self._safe_call(False, super(SSL_fileobject, self).sendall, *args, **kwargs) File "/opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 1091, in _safe_call return call(*args, **kwargs) File "/opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 753, in sendall bytes_sent = self.send(data) File "/opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 1142, in send return self._safe_call(False, super(SSL_fileobject, self).send, *args, **kwargs) File "/opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 1121, in _safe_call raise FatalSSLAlert(*e.args) FatalSSLAlert: [('SSL routines', 'SSL23_WRITE', 'ssl handshake failure')]   ... View more

Re: Sentry No valid privileges Required privileges...

by Contributor in Support Questions
‎03-30-2017 05:03 PM
1 Kudo
‎03-30-2017 05:03 PM
1 Kudo
do you mind expanding on the answer a bit I have the same problem  is the group setting in CM or on hdfs or os files system?   ... View more

Re: unhex in Impala

by Contributor in Support Questions
‎06-21-2016 10:21 AM
‎06-21-2016 10:21 AM
Thanks very helpful but the ASCII function only translates the first character and the decode function (in hive) is in conflict with Impala interpretation of decode UDF to be similar to case (a poor decision IMHO since case is much more readable than decode)   Anyway opinions aside How do I convert a large hex value into a large number?   select ascii(unhex('130F9C6D6A2972')) largenumber , ascii(unhex('13')) firstpart both yeild the value of 19 when the first one suppose to yeild 5365189082491250 ... View more

unhex in Impala

by Contributor in Support Questions
‎06-20-2016 01:32 PM
‎06-20-2016 01:32 PM
Running in Impala I try the following  select unhex('4B') and I get a string result of K   If I go to Hex to decimal converter to convert hex value '4B' I get the number 75 which is what I expect   Is unhex not a hexadecimal to decimal converter? according to http://www.cloudera.com/documentation/enterprise/5-5-x/topics/impala_math_functions.html unhex(string a) Purpose: Returns a string of characters with ASCII values corresponding to pairs of hexadecimal digits in the argument. Return type: string   I don't even know where to begin to start troubleshooting this   ... View more
Labels:

Re: HUE no longer show databases in Impala after m...

by Contributor in Support Questions
‎02-05-2016 05:13 PM
‎02-05-2016 05:13 PM
Per support in another thread   The last time I saw this issue, it was due to something cached in the browser used to access Hue.  Can you try using another browser to see if the problem happens there? If so, you might try logging out of Hue, clearing your cache, then logging into Hue to see if that helps.     ... View more

Re: HUE no longer show databases in Impala after m...

by Contributor in Support Questions
‎02-05-2016 04:22 PM
‎02-05-2016 04:22 PM
typing in the command show databases in HUE under the Impala GUI yeilds the following error message AnalysisException: Syntax error in line 1: USE `` ^ Encountered: EMPTY IDENTIFIER Expected: IDENTIFIER CAUSED BY: Exception: Syntax error   works in hive, works on Impala Command line ... View more

Re: HUE no longer show databases in Impala after m...

by Contributor in Support Questions
‎02-05-2016 04:18 PM
‎02-05-2016 04:18 PM
I restarted HUE --Impala via HUEdoes not see databases then I went to Cloudera Manager disabled Impala restarted HUE enabled Impala restarted HUE -Impala via HUE does not see databases then I went to Cloudera Manager restarted Impala, Hive and HUE - Impala via HUE does not see databases ... View more
Contact Me
Online
Offline
Last Visited
‎01-14-2019 03:17 PM
Community Statistics
Member Since ‎10-15-2014 04:20 PM
Last Visited ‎01-14-2019 03:17 PM
Posts 126
Kudos received 2