[Adding details to the original case]
To rectify this issue I tried refering the tar file directly from S3 without changing fs.defaultFS to s3://bucketName.
I did the following changes in tez-site.xml:
- set tez.lib.uris property to s3://bucketName
In hive-site.xml:
- set hive.execution.engine=tez
- fs.defaultFS=hdfs://
But still it is searching for tar file in hdfs.
S3 location where tez.tar.gz has been stored: s3://srev-emr-dev/F129/tez/tez.tar.gz where s3 bucket name is 'srev-emr-dev'
Facing below error:
java.io.FileNotFoundException: File does not exist: /F129/tez/tez.tar.gz
java.io.FileNotFoundException: File does not exist: /F129/tez/tez.tar.gz at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1441) ~[hadoop-hdfs-client-2.8.3-amzn-1.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1434) ~[hadoop-hdfs-client-2.8.3-amzn-1.jar:?] at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.8.3-amzn-1.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1449) ~[hadoop-hdfs-client-2.8.3-amzn-1.jar:?] at org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:1031) ~[tez-api-0.8.4.jar:0.8.4] at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:275) ~[tez-api-0.8.4.jar:0.8.4] at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183) ~[tez-api-0.8.4.jar:0.8.4] at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057) ~[tez-api-0.8.4.jar:0.8.4] at org.apache.tez.client.TezClient.start(TezClient.java:447) ~[tez-api-0.8.4.jar:0.8.4] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:376) ~[hive-exec-2.3.2-amzn-0.jar:2.3.2-amzn-0] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:87) ~[hive-exec-2.3.2-amzn-0.jar:2.3.2-amzn-0] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:330) [hive-exec-2.3.2-amzn-0.jar:2.3.2-amzn-0] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:326) [hive-exec-2.3.2-amzn-0.jar:2.3.2-amzn-0] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]