Using HIVE VIEW in Ambari hangs and/or throws errors at the simplest thing

[#1]

Good day! My developer team is just starting to want to use Hive View (via ambari). I'm the sysadmin. The first report was "my query isn't working" (hangs, no result). After a bunch of fiddling around and simplifying the tests, I discovered that pretty much everything in Hive View is not functioning properly. I am desperate for help.... My simplest test: creating two empty databases. Boom.

Example: Log into ambari. go into Hive View. In the work window try a very simple operation: "create database test;" As soon as I hit "execute", I get an error at the top of the window. (see image)

BUT..... if I wait a few seconds...... the error vanishes and well, the database DOES get created. There is no explanatory text regarding that error. Here is where everything goes (further) south. As soon as I try anything else, I don't get the same error above. Instead, after clicking "Execute", I never get a result back. The green button at the bottom stays in an executing state (green button turns to orange "Stop Execution" button.,......... and it never comes back.

In like fashion-- the database IS apparently created, which is a partial success. But the GUI gets locked up.

Meanwhile, in the hive view log, I see a repeated series of messages over and over and over.

Example:

22 Mar 2017 16:13:07,851 INFO [ambari-client-thread-11095] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] DDLDelegatorImpl:163 - Executing query: show databases like '*', for user: kcb 22 Mar 2017 16:13:07,852 INFO [HiveViewActorSystem-akka.actor.result-dispatcher-50] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] StatementExecutor:86 - Statement executor is executing statement: show databases like '*', Statement id: 0, JobId: SYNC JOB 22 Mar 2017 16:13:07,873 INFO [HiveViewActorSystem-akka.actor.result-dispatcher-50] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] StatementExecutor:88 - Finished executing statement: show databases like '*', Statement id: 0, JobId: SYNC JOB 22 Mar 2017 16:13:07,874 INFO [HiveViewActorSystem-akka.actor.jdbc-connector-dispatcher-69] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] JdbcConnector:281 - Finished processing SQL statements for Job id : SYNC JOB 22 Mar 2017 16:13:07,880 INFO [HiveViewActorSystem-akka.actor.default-dispatcher-34] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] OperationController:328 - About to free sync connector for user kcb 22 Mar 2017 16:13:19,271 INFO [HiveViewActorSystem-akka.actor.default-dispatcher-34] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] OperationController:319 - About to free connector for job 9 and user kcb 22 Mar 2017 16:13:22,868 INFO [ambari-client-thread-11096] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] DDLDelegatorImpl:163 - Executing query: show databases like '*', for user: kcb 22 Mar 2017 16:13:22,869 INFO [HiveViewActorSystem-akka.actor.result-dispatcher-50] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] StatementExecutor:86 - Statement executor is executing statement: show databases like '*', Statement id: 0, JobId: SYNC JOB 22 Mar 2017 16:13:22,891 INFO [HiveViewActorSystem-akka.actor.result-dispatcher-50] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] StatementExecutor:88 - Finished executing statement: show databases like '*', Statement id: 0, JobId: SYNC JOB 22 Mar 2017 16:13:22,891 INFO [HiveViewActorSystem-akka.actor.jdbc-connector-dispatcher-69] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] JdbcConnector:281 - Finished processing SQL statements for Job id : SYNC JOB 22 Mar 2017 16:13:22,897 INFO [HiveViewActorSystem-akka.actor.default-dispatcher-70] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] OperationController:328 - About to free sync connector for user kcb 22 Mar 2017 16:13:37,870 INFO [ambari-client-thread-11095] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] DDLDelegatorImpl:163 - Executing query: show databases like '*', for user: kcb 22 Mar 2017 16:13:37,871 INFO [HiveViewActorSystem-akka.actor.result-dispatcher-50] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] StatementExecutor:86 - Statement executor is executing statement: show databases like '*', Statement id: 0, JobId: SYNC JOB 22 Mar 2017 16:13:37,894 INFO [HiveViewActorSystem-akka.actor.result-dispatcher-50] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] StatementExecutor:88 - Finished executing statement: show databases like '*', Statement id: 0, JobId: SYNC JOB 22 Mar 2017 16:13:37,894 INFO [HiveViewActorSystem-akka.actor.jdbc-connector-dispatcher-69] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] JdbcConnector:281 - Finished processing SQL statements for Job id : SYNC JOB 22 Mar 2017 16:13:37,900 INFO [HiveViewActorSystem-akka.actor.default-dispatcher-70] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] OperationController:328 - About to free sync connector for user kcb 22 Mar 2017 16:13:52,873 INFO [ambari-client-thread-11095] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] DDLDelegatorImpl:163 - Executing query: show databases like '*', for user: kcb 22 Mar 2017 16:13:52,875 INFO [HiveViewActorSystem-akka.actor.result-dispatcher-50] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] StatementExecutor:86 - Statement executor is executing statement: show databases like '*', Statement id: 0, JobId: SYNC JOB 22 Mar 2017 16:13:52,896 INFO [HiveViewActorSystem-akka.actor.result-dispatcher-50] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] StatementExecutor:88 - Finished executing statement: show databases like '*', Statement id: 0, JobId: SYNC JOB 22 Mar 2017 16:13:52,897 INFO [HiveViewActorSystem-akka.actor.jdbc-connector-dispatcher-69] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] JdbcConnector:281 - Finished processing SQL statements for Job id : SYNC JOB 22 Mar 2017 16:13:52,903 INFO [HiveViewActorSystem-akka.actor.default-dispatcher-34] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] OperationController:328 - About to free sync connector for user kcb 22 Mar 2017 16:14:07,876 INFO [ambari-client-thread-11005] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] DDLDelegatorImpl:163 - Executing query: show databases like '*', for user: kcb 22 Mar 2017 16:14:07,877 INFO [HiveViewActorSystem-akka.actor.result-dispatcher-50] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] StatementExecutor:86 - Statement executor is executing statement: show databases like '*', Statement id: 0, JobId: SYNC JOB 22 Mar 2017 16:14:07,897 INFO [HiveViewActorSystem-akka.actor.result-dispatcher-50] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] StatementExecutor:88 - Finished executing statement: show databases like '*', Statement id: 0, JobId: SYNC JOB 22 Mar 2017 16:14:07,897 INFO [HiveViewActorSystem-akka.actor.jdbc-connector-dispatcher-69] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] JdbcConnector:281 - Finished processing SQL statements for Job id : SYNC JOB 22 Mar 2017 16:14:07,904 INFO [HiveViewActorSystem-akka.actor.default-dispatcher-70] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] OperationController:328 - About to free sync connector for user kcb 22 Mar 2017 16:14:22,897 INFO [ambari-client-thread-11095] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] DDLDelegatorImpl:163 - Executing query: show databases like '*', for user: kcb 22 Mar 2017 16:14:22,898 INFO [HiveViewActorSystem-akka.actor.result-dispatcher-50] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] StatementExecutor:86 - Statement executor is executing statement: show databases like '*', Statement id: 0, JobId: SYNC JOB 22 Mar 2017 16:14:22,919 INFO [HiveViewActorSystem-akka.actor.result-dispatcher-50] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] StatementExecutor:88 - Finished executing statement: show databases like '*', Statement id: 0, JobId: SYNC JOB 22 Mar 2017 16:14:22,919 INFO [HiveViewActorSystem-akka.actor.jdbc-connector-dispatcher-69] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] JdbcConnector:281 - Finished processing SQL statements for Job id : SYNC JOB 22 Mar 2017 16:14:22,925 INFO [HiveViewActorSystem-akka.actor.default-dispatcher-34] [HIVE 1.5.0 AUTO_HIVE_INSTANCE] OperationController:328 - About to free sync connector for user kcb

Finally.. hitting "Stop Execution" on the bottom gets me no further; that too hangs seemingly forever.

So that's where I am. I have a relatively new install of HDP (latest- 2.5), and as I currently have it, the whole hive view is unusable. I even tried uninstalling and reinstalling HIVE completely; no change.

I am desperate for help here all... I appreciate any and all support the community can give. I will monitor this closely and will provide any other config info anyone needs to know about.

In related debugging- I have found that I have NO issues with the hive CLI. There I can do queries and all sorts of things, no issues. My problem seems limited to the Ambari Hive View front end.

[#2]

@Kent Brodie

In Hive Views setting section can you make changes to below parameter and try?

Hive Session Parameters= transportMode=binary;httpPath=cliservice;hive.server2.proxy.user=${username}

Hope this helps you.

[#3]

@SBandaru I tried the change above and it has made no difference. I still have the same behavior.

[#4 · Accepted solution]

@Kent Brodie

I can reproduce the similar issue as yours by applying an apache httpd webserver in front of Ambari.

I have written an article on PigView sometimes back and i see that you might be facing the same issue with HiveView. https://community.hortonworks.com/content/kbentry/68951/when-webserver-is-installed-in-front-to-the-...

I see that your URL looks like using port 80 (default http port) which makes me feel that you are accessing ambari server using some front end WebServer like (apache httpd).

Check your webserver setting to find out if it allows "AllowEncodedSlashes" or not? For example refer to: http://httpd.apache.org/docs/current/mod/core.html#allowencodedslashes

The AllowEncodedSlashes directive allows URLs which contain encoded path separators (%2F for / and additionally %5C for \ on accordant systems) to be used in the path info.

- With the default value, "Off", such URLs are refused with a 404 (Not found) error.

- With the value "On", such URLs are accepted, and encoded slashes are decoded like all other encoded characters.

- With the value "NoDecode", such URLs are accepted, but encoded slashes are not decoded but left in their encoded state.

.

Or for quick testing it will be best if you directly access the Ambari on port 8080 instead of using the webserver/frontend.

Following fixed the issue for me:

# cat  /etc/httpd/conf/httpd.conf | grep -A 2 -B 3 AllowEncodedSlashes
<VirtualHost erie1.example.com:80>
    ServerAdmin erie1.example.com
    ProxyPass / http://erie1.example.com:8080/  nocanon
    AllowEncodedSlashes NoDecode
</VirtualHost>


[root@erie1 ~]# 
[root@erie1 ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

.

[#5]

@Kent Brodie

Great to hear that your issue is resolved. It will be wonderful if you can mark the answer of this thread as "Accepted" so that it will be useful for community.

[#6]

Oh my gosh, I totally forgot that yes, this is front-ended by apache. Setting the AllowEncodedSlashes to "On" in my environment solved the issue. (I have apache 2.2.x which does not yet have the "NoDecode" value, but that's OK as "On" seems to work). thankyouthankyyouthankyouthankyouthankyou.