Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

problem running query through webhcat - CDH 5

problem running query through webhcat - CDH 5

Champion Alumni

Hello,

 

I need to expose a simple webservice (get only the lines that have a certain ID from one table). Therefore I tried to use WebHCat. 

curl -v -d user.name=agherman -d execute="SELECT * FROM myDatabase.myTable+*+WHERE+key=52;" -d statusdir="/user/agherman/agherman.output" 'http://<ip>:50111/templeton/v1/hive'

 Firstly I had a similar error with this post : http://community.cloudera.com/t5/Batch-SQL-Apache-Hive/problem-running-hive-query-through-webhcat/td... , therefore I folowed the given steps (and I addaped them to CDH5 version). 

 

  • ln -s ../../../../../../jars/hive-exec-0.13.1-cdh5.3.0.jar /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/hive-hcatalog/sbin/../share/webhcat/svr/lib/

  • Service-Wide Environment Safety Valve (Cloudera Management Service Environment Advanced Configuration Snippet (Safety Valve)): TEMPLETON_HOME=/opt/cloudera/parcels/CDH/lib/hive-hcatalog

  • WebHCat Server Configuration Safety Valve for webhcat-site.xml

<property>
    <name>templeton.libjars</name>
    <value>${env.TEMPLETON_HOME}/share/webhcat/svr/lib/zookeeper-3.4.5-cdh5.3.0.jar</value>
    <description>Jars to add to the classpath.</description>
  </property>
<property>
    <name>templeton.hive.path</name>
    <value>hive-0.13.1-cdh5.3.0.tar.gz/hive/bin/hive</value>
    <description>The path to the Hive executable.</description>
  </property>
  
  <property>
    <name>templeton.hive.archive</name>
    <value>hdfs:///apps/templeton/hive-0.13.1-cdh5.3.0.tar.gz</value>
    <description>The path to the Hive executable.</description>
  </property>
  •  Create tar an upload
cd /opt/cloudera/parcels/CDH/lib
tar -zcvf /tmp/hive-0.13.1-cdh5.3.0.tar.gz ./
cd /tmp
sudo -u hdfs hadoop fs -mkdir /apps/templeton
sudo -u hdfs hadoop fs -mkdir /apps
sudo -u hdfs hadoop fs -mkdir /apps/templeton
sudo -u hdfs hadoop fs -put /tmp/hive-0.13.0-cdh5.3.0.tar.gz /apps/templeton
sudo -u hdfs hadoop dfs -chmod -R 777 /apps/templeton
 
  • create needed directory

 

hdfs dfs -mkdir /templeton-hadoop

hdfs dfs -mkdir /templeton-hadoop/jobs/

hdfs dfs -chown -R hive /templeton-hadoop

 

  • However now, the jobs run and has a sucessful state but when I try to see my result file I have a missing jar error:
-	Missing Hive Execution Jar: /yarn/nm/usercache/agherman/appcache/application_1422872444323_0062/container_1422872444323_0062_01_000002/hive-0.13.1-cdh5.3.0.tar.gz/hive/lib/hive-exec-*.jar => add this jar into our jar

 I checked the created hive-0.13.1-cdh5.3.0.tar.gz and it contains 2 hive-exec-*.jar in the hive/lib directory...

 

What should I check? Where should I add this jars?

 

Thank you!

 

 

GHERMAN Alina
2 REPLIES 2

Re: problem running query through webhcat - CDH 5

New Contributor

I tried your procedure and the file hive-0.13.1-cdh5.3.0.tar.gz contains only a symbolic link to hive-exec.jar.  You need to dereference the symbolic links.  I used the following to create the tarball:

 

tar -hzcvf /tmp/hive-1.1.0-cdh5.5.1.tar.gz ./hive

 

This gets past the error you were experiencing although I now get the following exception:

 

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:512)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1492)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:64)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2934)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:493)
... 8 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1490)
... 13 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:419)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:234)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1490)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:64)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:74)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2915)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2934)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:493)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:466)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:234)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
... 18 more

Highlighted

Re: problem running query through webhcat - CDH 5

New Contributor

You need to update the metastore thrift server port information in your WebHCat configuration. Adding the below property to webhcat-site.xml will resolve the issue. 

 

<property>
<name>templeton.hive.properties</name>
<value>hive.metastore.uris=thrift://metastore-host:9083</value>
<description>Properties to set when running hive.</description>
</property>

 

Thanks!