This is fixed in Hortonworks Cloud. Is this on-prime cluster or Hortonworks Cloud?. ... View more

Can you share the details of the HDP/HDC version used and the s3a connector version being used?. If it is recent versions you can enable "fs.s3a.experimental.input.fadvise=random" for ORC dataset, which reduces the number of connection establishments and breakages to S3. Not sure if you are using EC2 instances or on-prem for accessing S3. But in case the machine itself has network connectivity issues to S3, easiest option could be to eliminate that node or fix the n/w inconsistency. ... View more

>>> 2017-03-18 22:43:27,002 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1489855879213_0002 2017-03-18 22:43:27,005 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - The url to track the Tez Session: http://pc-1.thenet.edu:8088/proxy/application_1489855879213_0002/ >>> I agree with zyang. Looking at the logs, it appears that there isn't enough capacity to launch. Can you verify the queue configs and cluster capacity? ... View more

Stack trace points to "s3native.NativeS3FileSystem" which is pretty much deprecated. Suggest to use "s3a://" in table location "(show create table ex_httpvendorstats" would reveal the details) ... View more

@PJ - Would it be possible to share the hive.log when you observed this? ... View more

Can you please check if "hive.mv.files.thread=0" helps in this case? ... View more

You can remove "hive.tez.java.opts=-Xmx4096m". Tez automatically takes up 80% of the container size allocated to it. As per your hive-site.xml and yarn-site.xml, "hive.tez.container.size=6144", "yarn.scheduler.minimum-allocation-mb=6144". So ~4915MB should be automatically be assigned without specifying any Xmx value in hive.tez.java.opts. Remove "--hiveconf hive.tez.container.size=2048" from hive cli command. By specifying 2048, it would end up under utilizing the memory. After incorporating #1, with the current config in hive-site, yarn-site you have posted, can you run "hive" cli without specifying any option and run the query? For simple like select count(*) from table, it should not launch tez job if it had enough information about the rows in metastore. Run "analyze table tweets compute statistics" and re-run this select count(*) statement. It should fetch information from metastore directly as opposed to launching the tez job. ... View more

For beeline, can you "set hive.llap.client.consistent.splits=false;" and run your query. ... View more