Support Questions
Find answers, ask questions, and share your expertise

Tez execution error during Hive query in HDP tutorial

Solved Go to solution
Highlighted

Tez execution error during Hive query in HDP tutorial

Contributor

I am following the tutorial to visualize and analyze sensor data but am running into problems during the create table stored as orc section. The log is saying there is a file that doesn't exist. I have included the hive log below. Anyone know how to fix this?

4056-hive-error.jpg

INFO : Session is already openINFO : Dag name: CREATE TABLE hvac STORED AS ORC A...hvac_raw(Stage-1)INFO : Tez session was closed. Reopening...ERROR : Failed to execute tez graph.java.io.FileNotFoundException: File does not exist: /hdp/apps/2.3.4.7-4/tez/tez.tar.gzat org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1319)at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1311)at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1311)at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:751)at org.apache.tez.client.TezClientUtils.getLRFileStatus(TezClientUtils.java:132)at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:845)at org.apache.tez.client.TezClient.start(TezClient.java:380)at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeAndOpen(TezSessionPoolManager.java:311)at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:453)at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:180)at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1720)at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1477)at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1254)at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118)at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Tez execution error during Hive query in HDP tutorial

Guru

If you are using latest sandbox, you should see /hdp/apps/2.4.0.0-169/tez/tez.tar.gz folder in HDFS

But if you are using a 2.3.4 version of sandbox, you should see the version in HDFS. Can you look at /hdp/apps/ in hdfs and paste the contents? If the file does not exist, you can manually put it there by following below steps

su - $HDFS_USER
hdfs dfs -mkdir -p /hdp/apps/<hdp_version>/tez/
hdfs dfs -put /usr/hdp/<hdp_version>/tez/lib/tez.tar.gz /hdp/apps/<hdp_version>/tez/
hdfs dfs -chown -R $HDFS_USER:$HADOOP_USER /hdp
hdfs dfs -chmod -R 555 /hdp/apps/<hdp_version>/tez
hdfs dfs -chmod -R 444 /hdp/apps/<hdp_version>/tez/tez.tar.gz

View solution in original post

3 REPLIES 3
Highlighted

Re: Tez execution error during Hive query in HDP tutorial

Guru

If you are using latest sandbox, you should see /hdp/apps/2.4.0.0-169/tez/tez.tar.gz folder in HDFS

But if you are using a 2.3.4 version of sandbox, you should see the version in HDFS. Can you look at /hdp/apps/ in hdfs and paste the contents? If the file does not exist, you can manually put it there by following below steps

su - $HDFS_USER
hdfs dfs -mkdir -p /hdp/apps/<hdp_version>/tez/
hdfs dfs -put /usr/hdp/<hdp_version>/tez/lib/tez.tar.gz /hdp/apps/<hdp_version>/tez/
hdfs dfs -chown -R $HDFS_USER:$HADOOP_USER /hdp
hdfs dfs -chmod -R 555 /hdp/apps/<hdp_version>/tez
hdfs dfs -chmod -R 444 /hdp/apps/<hdp_version>/tez/tez.tar.gz

View solution in original post

Highlighted

Re: Tez execution error during Hive query in HDP tutorial

Contributor

Hey @Ravi Mutyala, I am using an older version of HDP (2.3.4) in a 3 node cluster. I followed the steps you provided and I was able to execute the query! Can you explain why moving that file allowed the query to be successful? Thanks for your help.

Highlighted

Re: Tez execution error during Hive query in HDP tutorial

Guru

Not sure how your sandbox missed that folder in the first place. I sent you the steps from manual install. Should have been there in the first place, but since it was not there, we followed manual steps to put them there. Tez and mapreduce use the tar.gz files from /hdp/apps/<hdp_version>/ for submitting applications.

Don't have an account?