question Re: Tez execution error during Hive query in HDP tutorial in Archives of Support Questions (Read Only)

Tez execution error during Hive query in HDP tutorial

marka_thorson — Mon, 19 Aug 2019 09:09:57 GMT

I am following the tutorial to visualize and analyze sensor data but am running into problems during the create table stored as orc section. The log is saying there is a file that doesn't exist. I have included the hive log below. Anyone know how to fix this?

INFO : Session is already openINFO : Dag name: CREATE TABLE hvac STORED AS ORC A...hvac_raw(Stage-1)INFO : Tez session was closed. Reopening...ERROR : Failed to execute tez graph.java.io.FileNotFoundException: File does not exist: /hdp/apps/2.3.4.7-4/tez/tez.tar.gzat org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1319)at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1311)at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1311)at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:751)at org.apache.tez.client.TezClientUtils.getLRFileStatus(TezClientUtils.java:132)at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:845)at org.apache.tez.client.TezClient.start(TezClient.java:380)at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeAndOpen(TezSessionPoolManager.java:311)at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:453)at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:180)at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1720)at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1477)at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1254)at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118)at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)

Re: Tez execution error during Hive query in HDP tutorial

ravi1 — Fri, 06 May 2016 00:10:00 GMT

If you are using latest sandbox, you should see /hdp/apps/2.4.0.0-169/tez/tez.tar.gz folder in HDFS

But if you are using a 2.3.4 version of sandbox, you should see the version in HDFS. Can you look at /hdp/apps/ in hdfs and paste the contents? If the file does not exist, you can manually put it there by following below steps

su - $HDFS_USER
hdfs dfs -mkdir -p /hdp/apps/<hdp_version>/tez/
hdfs dfs -put /usr/hdp/<hdp_version>/tez/lib/tez.tar.gz /hdp/apps/<hdp_version>/tez/
hdfs dfs -chown -R $HDFS_USER:$HADOOP_USER /hdp
hdfs dfs -chmod -R 555 /hdp/apps/<hdp_version>/tez
hdfs dfs -chmod -R 444 /hdp/apps/<hdp_version>/tez/tez.tar.gz

Re: Tez execution error during Hive query in HDP tutorial

marka_thorson — Fri, 06 May 2016 00:35:33 GMT

Hey @Ravi Mutyala, I am using an older version of HDP (2.3.4) in a 3 node cluster. I followed the steps you provided and I was able to execute the query! Can you explain why moving that file allowed the query to be successful? Thanks for your help.

Re: Tez execution error during Hive query in HDP tutorial

ravi1 — Fri, 06 May 2016 00:43:03 GMT

Not sure how your sandbox missed that folder in the first place. I sent you the steps from manual install. Should have been there in the first place, but since it was not there, we followed manual steps to put them there. Tez and mapreduce use the tar.gz files from /hdp/apps/<hdp_version>/ for submitting applications.