Tez execution error during Hive query in HDP tutorial

[#1]

I am following the tutorial to visualize and analyze sensor data but am running into problems during the create table stored as orc section. The log is saying there is a file that doesn't exist. I have included the hive log below. Anyone know how to fix this?

INFO : Session is already openINFO : Dag name: CREATE TABLE hvac STORED AS ORC A...hvac_raw(Stage-1)INFO : Tez session was closed. Reopening...ERROR : Failed to execute tez graph.java.io.FileNotFoundException: File does not exist: /hdp/apps/2.3.4.7-4/tez/tez.tar.gzat org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1319)at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1311)at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1311)at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:751)at org.apache.tez.client.TezClientUtils.getLRFileStatus(TezClientUtils.java:132)at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:845)at org.apache.tez.client.TezClient.start(TezClient.java:380)at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeAndOpen(TezSessionPoolManager.java:311)at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:453)at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:180)at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1720)at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1477)at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1254)at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118)at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)

[#2 · Accepted solution]

If you are using latest sandbox, you should see /hdp/apps/2.4.0.0-169/tez/tez.tar.gz folder in HDFS

But if you are using a 2.3.4 version of sandbox, you should see the version in HDFS. Can you look at /hdp/apps/ in hdfs and paste the contents? If the file does not exist, you can manually put it there by following below steps

su - $HDFS_USER
hdfs dfs -mkdir -p /hdp/apps/<hdp_version>/tez/
hdfs dfs -put /usr/hdp/<hdp_version>/tez/lib/tez.tar.gz /hdp/apps/<hdp_version>/tez/
hdfs dfs -chown -R $HDFS_USER:$HADOOP_USER /hdp
hdfs dfs -chmod -R 555 /hdp/apps/<hdp_version>/tez
hdfs dfs -chmod -R 444 /hdp/apps/<hdp_version>/tez/tez.tar.gz

[#3]

Hey @Ravi Mutyala, I am using an older version of HDP (2.3.4) in a 3 node cluster. I followed the steps you provided and I was able to execute the query! Can you explain why moving that file allowed the query to be successful? Thanks for your help.

[#4]

Not sure how your sandbox missed that folder in the first place. I sent you the steps from manual install. Should have been there in the first place, but since it was not there, we followed manual steps to put them there. Tez and mapreduce use the tar.gz files from /hdp/apps/<hdp_version>/ for submitting applications.