Hue(beewax) hive error: Internal error processing query in PROD server

[#1]

Hi..i am using HDP 2.1.2 and Hue(beewax hive UI) ..while i run simple hive query(show databases) its show internal error processing query error

But i could see all data bases and tables in hue and also i could run hive queries in command line..please help me get out of this..

Hive: 0.13.0.2.1

Tez:0.4.0.2.1

[#2]

@sankar rao

can you please share the logs from /var/log/hue/*. It could be a simple version issue. What version of HUE beeswax are you using? Does it support the hive version you are using?

[#3]

@mqureshi

Hue: 2.3.1-402

Hive: 0.13.0.2.1

I think so its support because previously until one day back i have successfully run hive queries in hue, today only i have this issue...I have same conf and versions in TEST here can run hive queries in HUE..but problem in PROD only.

comes logs i just wondering which logs should check i have below logs

-rw-r--r-- 1 hue hue 989989 Jul 30 00:04 access.log

-rw-r--r-- 1 hue hue 999960 Jul 27 08:18 access.log.1

-rw-r--r-- 1 hue hue 999961 Jul 21 08:38 access.log.2

-rw-r--r-- 1 hue hue 999978 Jul 20 05:35 access.log.3

-rw-r--r-- 1 hue hue 27700 Sep 23 2015 beeswax_server.log

-rw-r--r-- 1 hue hue 628497762 Jul 29 22:36 beeswax_server.out

-rw-r--r-- 1 hue hue 986613 Jul 29 22:36 error.log

-rw-r--r-- 1 hue hue 999246 Jun 1 11:53 error.log.1

-rw-r--r-- 1 hue hue 991759 Feb 15 09:54 error.log.2

-rw-r--r-- 1 hue hue 999264 Dec 8 2015 error.log.3

-rw-r--r-- 1 hue hue 12691 Sep 23 2015 kt_renewer.log

-rw-r--r-- 1 hue hue 0 Aug 26 2014 kt_renewer.out

-rw-r--r-- 1 hue hue 65644 Jul 30 00:04 runcpserver.log

-rw-r--r-- 1 hue hue 999967 Jul 29 22:34 runcpserver.log.1

-rw-r--r-- 1 hue hue 999942 Jul 28 12:36 runcpserver.log.2

-rw-r--r-- 1 hue hue 999160 Jul 28 08:30 runcpserver.log.3

-rw-r--r-- 1 hue hue 114927 Jul 19 10:33 runcpserver.out

-rw-r--r-- 1 hue hue 6603 Jun 27 05:37 shell_input.log

-rw-r--r-- 1 hue hue 36501 Jun 27 05:37 shell_output.log

-rw-r--r-- 1 hue hue 11758 Sep 23 2015 supervisor.log

-rw-r--r-- 1 hue hue 0 Aug 26 2014 supervisor.out

[#4]

@sankar rao

check on the timestamp when you run the query and open that log. i think you should look at beeswax_server.out as well as error.log.

Please share what you see.

[#5]

@mqureshi

Previously i had error: Internal error processing query but now i reciving time out error

i post the error.log & beeswax_server.out

TApplicationException: Internal error processing query [30/Jul/2016 22:46:05 +0000] thrift_util ERROR Thrift saw exception (this may be expected). Traceback (most recent call last): File "/usr/lib/hue/desktop/core/src/desktop/lib/thrift_util.py", line 348, in wrapper ret = res(*args, **kwargs) File "/usr/lib/hue/apps/beeswax/gen-py/beeswaxd/BeeswaxService.py", line 142, in query return self.recv_query() File "/usr/lib/hue/apps/beeswax/gen-py/beeswaxd/BeeswaxService.py", line 158, in recv_query raise x TApplicationException: Internal error processing query

[01/Aug/2016 00:16:49 +0000] views ERROR Failed to send query completion notification via e-mail: QueryHistory matching query does not exist.

[01/Aug/2016 01:09:25 +0000] views ERROR Failed to send query completion notification via e-mail: QueryHistory matching query does not exist.

beeswax_server.out

16/08/01 00:48:15 INFO tez.TezSessionState: User of session id 5cb11a09-71fb-4970-95d6-b1ac73a0c804 is y919122 16/08/01 00:53:25 INFO tez.DagUtils: Jar dir is null/directory doesn't exist. Choosing HIVE_INSTALL_DIR - hdfs:/user/y919122/.hiveJars 16/08/01 00:56:33 ERROR thrift.ProcessFunction: Internal error processing query java.lang.OutOfMemoryError: Java heap space 16/08/01 00:56:33 INFO log.PerfLogger: </PERFLOG method=Driver.execute start=1470036998452 end=1470038193846 duration=1195394 from=org.apache.hadoop.hive.ql.Driver> 16/08/01 00:56:34 INFO tez.DagUtils: Localizing resource because it does not exist: file:/usr/lib/hive/lib/hive-exec-0.13.0.2.1.2.2-516.jar to dest: hdfs://ABIHADOOP1/user/y919122/.hiveJars/hive-exec-0.13.0.2.1.2.2-516-82bbaf519912703e421e55e9826552910417b9fcfae3f865e2e07f826a9f260a.jar 16/08/01 00:56:50 INFO tez.DagUtils: Looks like another thread is writing the same file will wait. 16/08/01 00:56:50 INFO tez.DagUtils: Number of wait attempts: 5. Wait interval: 5000 16/08/01 00:57:06 INFO tez.DagUtils: Resource modification time: 1469367036707 16/08/01 00:57:08 INFO tez.TezSessionState: Opening new Tez Session (id: 5cb11a09-71fb-4970-95d6-b1ac73a0c804, scratch dir: hdfs://ABIHADOOP1/tmp/hive-beeswax-y919122/_tez_session_dir/5cb11a09-71fb-4970-95d6-b1ac73a0c804) 16/08/01 00:57:49 INFO client.RMProxy: Connecting to ResourceManager at stlpr8712/39.6.64.3:8050 16/08/01 00:59:25 INFO impl.YarnClientImpl: Submitted application application_1443521267046_220241 16/08/01 00:59:25 INFO log.PerfLogger: <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver> 16/08/01 00:59:25 INFO ql.Driver: Starting command: show databases 16/08/01 01:04:22 INFO impl.TimelineClientImpl: Timeline service address: http://s11:8188/ws/v1/timeline/ 16/08/01 01:04:22 INFO hooks.ATSHook: Created ATS Hook 16/08/01 01:04:22 INFO log.PerfLogger: <PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver> 16/08/01 01:04:22 INFO log.PerfLogger: </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1470038662694 end=1470038662694 duration=0 from=org.apache.hadoop.hive.ql.Driver> 16/08/01 01:04:22 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit start=1470035682189 end=1470038662695 duration=2980506 from=org.apache.hadoop.hive.ql.Driver> 16/08/01 01:04:22 INFO log.PerfLogger: <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver> 16/08/01 01:04:22 INFO log.PerfLogger: <PERFLOG method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver> 16/08/01 01:04:22 INFO exec.DDLTask: results : 48 16/08/01 01:04:24 INFO log.PerfLogger: </PERFLOG method=runTasks start=1470038662695 end=1470038664516 duration=1821 from=org.apache.hadoop.hive.ql.Driver> 16/08/01 01:09:25 INFO impl.TimelineClientImpl: Timeline service address: http://s1:8188/ws/v1/timeline/ 16/08/01 01:09:25 INFO hooks.ATSHook: Created ATS Hook 16/08/01 01:09:25 INFO log.PerfLogger: <PERFLOG method=PostHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver> 16/08/01 01:09:25 INFO log.PerfLogger: </PERFLOG method=PostHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1470038965108 end=1470038965108 duration=0 from=org.apache.hadoop.hive.ql.Driver> 16/08/01 01:09:25 INFO log.PerfLogger: </PERFLOG method=Driver.execute start=1470038365875 end=1470038965108 duration=599233 from=org.apache.hadoop.hive.ql.Driver> 16/08/01 01:09:25 INFO ql.Driver: OK 16/08/01 04:22:15 ERROR thrift.ProcessFunction: Internal error processing query java.lang.OutOfMemoryError: Java heap space

[#6 · Accepted solution]

@sankar rao

Is your beeswax remote from cluster? It would be better if it's on one of the edge nodes. You are running out of memory. Not sure what the cause is but it could be low memory on your machine. how much memory do you have for beeswax?

[#7]

Ok ..but i could able to run same query from different users(all are new users) but error comes with one single user(old user) ..all are in same group as well

[#8]

@sankar rao what's the user name? does this old user have a directory under /user/<username>? Also, can this user run queries outside of HUE, using beeline?

[#9]

@mqureshi

Thank you so much i really appreciate your response..

.user access in DEV is cleared but only still issue there PROD...

Its has below error log

16/08/02 01:09:12 INFO tez.TezSessionState: User of session id baacb86a-e54d-4fae-8c16-fff9834d4d8a is y919122 16/08/02 01:11:03 INFO tez.DagUtils: Jar dir is null/directory doesn't exist. Choosing HIVE_INSTALL_DIR - hdfs:/user/y919122/.hiveJars 16/08/02 01:12:52 ERROR thrift.ProcessFunction: Internal error processing query java.lang.OutOfMemoryError: Java heap space

its is says that java heap space ..but i could run same query in CLI so what should i do in HUE(hive beewax)?