ls: cannot access /hadoop/yarn/local/usercache/subrata/appcache/application_1489855879213_0001/container_e24_1489855879213_0001_01_000002/hive.tar.gz/hive/lib/slf4j-api-*.jar: No such file or directory ls: cannot access /hadoop/yarn/local/usercache/subrata/appcache/application_1489855879213_0001/container_e24_1489855879213_0001_01_000002/hive.tar.gz/hive/hcatalog/lib/*hbase-storage-handler-*.jar: No such file or directory WARNING: Use "yarn jar" to launch YARN applications. 17/03/18 22:43:16 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL 17/03/18 22:43:16 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE 17/03/18 22:43:16 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL 17/03/18 22:43:16 INFO pig.ExecTypeProvider: Trying ExecType : TEZ 17/03/18 22:43:16 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType 2017-03-18 22:43:16,323 [main] INFO org.apache.pig.Main - Apache Pig version 0.16.0.2.5.3.0-37 (rexported) compiled Nov 30 2016, 03:39:58 2017-03-18 22:43:16,323 [main] INFO org.apache.pig.Main - Logging error messages to: /hadoop/yarn/local/usercache/subrata/appcache/application_1489855879213_0001/container_e24_1489855879213_0001_01_000002/pig_1489857196322.log 2017-03-18 22:43:16,883 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/yarn/.pigbootup not found 2017-03-18 22:43:17,002 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://pc-1.thenet.edu:8020 2017-03-18 22:43:17,713 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-script.pig-684fa29b-1138-49f2-a6be-3b4ebd7babd5 2017-03-18 22:43:18,097 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/ 2017-03-18 22:43:18,201 [main] INFO org.apache.pig.backend.hadoop.PigATSClient - Created ATS Hook 2017-03-18 22:43:19,004 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2017-03-18 22:43:19,056 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083 2017-03-18 22:43:19,090 [main] INFO hive.metastore - Connected to metastore. 2017-03-18 22:43:20,386 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2017-03-18 22:43:20,390 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083 2017-03-18 22:43:20,394 [main] INFO hive.metastore - Connected to metastore. 2017-03-18 22:43:20,731 [main] WARN org.apache.pig.newplan.BaseOperatorPlan - Encountered Warning IMPLICIT_CAST_TO_FLOAT 1 time(s). 2017-03-18 22:43:20,788 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2017-03-18 22:43:20,822 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083 2017-03-18 22:43:20,823 [main] INFO hive.metastore - Connected to metastore. 2017-03-18 22:43:21,120 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2017-03-18 22:43:21,149 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083 2017-03-18 22:43:21,150 [main] INFO hive.metastore - Connected to metastore. 2017-03-18 22:43:21,270 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: HASH_JOIN,GROUP_BY,FILTER 2017-03-18 22:43:21,293 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code. 2017-03-18 22:43:21,318 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]} 2017-03-18 22:43:21,350 [main] INFO org.apache.pig.impl.util.SpillableMemoryManager - Selected heap (PS Old Gen) of size 716177408 to monitor. collectionUsageThreshold = 501324192, usageThreshold = 501324192 2017-03-18 22:43:21,393 [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for a: $0, $3, $4, $5, $6, $7, $8, $9 2017-03-18 22:43:21,569 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Tez staging directory is /tmp/yarn/staging and resources directory is /tmp/temp-816712545 2017-03-18 22:43:21,641 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.plan.TezCompiler - File concatenation threshold: 100 optimistic? false 2017-03-18 22:43:21,712 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil - Choosing to move algebraic foreach to combiner 2017-03-18 22:43:21,830 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2017-03-18 22:43:21,836 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083 2017-03-18 22:43:21,837 [main] INFO hive.metastore - Connected to metastore. 2017-03-18 22:43:22,114 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - 2017-03-18 22:43:22,123 [main] INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat - ORC pushdown predicate: null 2017-03-18 22:43:22,146 [main] INFO org.apache.hadoop.hive.ql.io.orc.OrcUtils - Using schema evolution configuration variables schema.evolution.columns [truckid, driverid, event, latitude, longitude, city, state, velocity, event_ind, idling_ind] / schema.evolution.columns.types [string, string, string, string, string, string, string, string, string, string] (isAcid false) 2017-03-18 22:43:22,492 [main] INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat - FooterCacheHitRatio: 0/2 2017-03-18 22:43:22,492 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - 2017-03-18 22:43:22,494 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1 2017-03-18 22:43:22,630 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2017-03-18 22:43:22,644 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083 2017-03-18 22:43:22,653 [main] INFO hive.metastore - Connected to metastore. 2017-03-18 22:43:22,767 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - 2017-03-18 22:43:22,767 [main] INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat - ORC pushdown predicate: null 2017-03-18 22:43:22,767 [main] INFO org.apache.hadoop.hive.ql.io.orc.OrcUtils - Using schema evolution configuration variables schema.evolution.columns [driverid, totmiles] / schema.evolution.columns.types [string, bigint] (isAcid false) 2017-03-18 22:43:22,788 [main] INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat - FooterCacheHitRatio: 0/2 2017-03-18 22:43:22,788 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - 2017-03-18 22:43:22,789 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1 2017-03-18 22:43:24,523 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: libfb303-0.9.3.jar 2017-03-18 22:43:24,523 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: pig-0.16.0.2.5.3.0-37-core-h2.jar 2017-03-18 22:43:24,523 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: libthrift-0.9.3.jar 2017-03-18 22:43:24,523 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: jdo-api-3.0.1.jar 2017-03-18 22:43:24,523 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: hive-hcatalog-pig-adapter-1.2.1000.2.5.3.0-37.jar 2017-03-18 22:43:24,523 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: automaton-1.11-8.jar 2017-03-18 22:43:24,523 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: hive-hbase-handler-1.2.1000.2.5.3.0-37.jar 2017-03-18 22:43:24,523 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: hive-exec-1.2.1000.2.5.3.0-37.jar 2017-03-18 22:43:24,523 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: hive-hcatalog-core-1.2.1000.2.5.3.0-37.jar 2017-03-18 22:43:24,523 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: antlr-runtime-3.4.jar 2017-03-18 22:43:24,523 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: hive-metastore-1.2.1000.2.5.3.0-37.jar 2017-03-18 22:43:24,668 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - For vertex - scope-51: parallelism=1, memory=1024, java opts=-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB 2017-03-18 22:43:24,669 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Processing aliases: a,b,c,d,e 2017-03-18 22:43:24,669 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Detailed locations: a[1,4],b[2,4],c[3,4],e[5,4],d[4,4] 2017-03-18 22:43:24,669 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Pig features in the vertex: 2017-03-18 22:43:24,727 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Set auto parallelism for vertex scope-52 2017-03-18 22:43:24,728 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - For vertex - scope-52: parallelism=1, memory=1024, java opts=-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB 2017-03-18 22:43:24,728 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Processing aliases: e,h 2017-03-18 22:43:24,728 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Detailed locations: e[5,4],h[7,4] 2017-03-18 22:43:24,728 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Pig features in the vertex: GROUP_BY 2017-03-18 22:43:24,784 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - For vertex - scope-53: parallelism=1, memory=1024, java opts=-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB 2017-03-18 22:43:24,784 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Processing aliases: g,h 2017-03-18 22:43:24,784 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Detailed locations: g[6,4],h[7,4] 2017-03-18 22:43:24,784 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Pig features in the vertex: 2017-03-18 22:43:24,845 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Set auto parallelism for vertex scope-54 2017-03-18 22:43:24,846 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - For vertex - scope-54: parallelism=2, memory=1024, java opts=-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB 2017-03-18 22:43:24,846 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Processing aliases: final_data,h 2017-03-18 22:43:24,846 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Detailed locations: h[7,4],final_data[8,13] 2017-03-18 22:43:24,846 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Pig features in the vertex: HASH_JOIN 2017-03-18 22:43:24,918 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2017-03-18 22:43:24,923 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083 2017-03-18 22:43:24,924 [main] INFO hive.metastore - Connected to metastore. 2017-03-18 22:43:24,949 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Total estimated parallelism is 5 2017-03-18 22:43:25,010 [PigTezLauncher-0] INFO org.apache.pig.tools.pigstats.tez.TezScriptState - Pig script settings are added to the job 2017-03-18 22:43:25,104 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Tez Client Version: [ component=tez-api, version=0.7.0.2.5.3.0-37, revision=2269be0a12263567ecfea0705e3c72244baa5830, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=2016-11-29T18:40:01Z ] 2017-03-18 22:43:25,259 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/ 2017-03-18 22:43:25,266 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050 2017-03-18 22:43:25,445 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200 2017-03-18 22:43:25,458 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Using org.apache.tez.dag.history.ats.acls.ATSV15HistoryACLPolicyManager to manage Timeline ACLs 2017-03-18 22:43:25,631 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/ 2017-03-18 22:43:25,640 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Session mode. Starting session. 2017-03-18 22:43:25,648 [PigTezLauncher-0] INFO org.apache.tez.common.security.TokenCache - Merging additional tokens from binary file, binaryFileName=/hadoop/yarn/local/usercache/subrata/appcache/application_1489855879213_0001/container_e24_1489855879213_0001_01_000002/container_tokens 2017-03-18 22:43:25,649 [PigTezLauncher-0] INFO org.apache.tez.client.TezClientUtils - Using tez.lib.uris value from configuration: /hdp/apps/2.5.3.0-37/tez/tez.tar.gz 2017-03-18 22:43:25,828 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Tez system stage directory hdfs://pc-1.thenet.edu:8020/tmp/yarn/staging/.tez/application_1489855879213_0002 doesn't exist and is created 2017-03-18 22:43:26,569 [PigTezLauncher-0] INFO org.apache.tez.dag.history.ats.acls.ATSV15HistoryACLPolicyManager - Created Timeline Domain for History ACLs, domainId=Tez_ATS_application_1489855879213_0002 2017-03-18 22:43:27,002 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1489855879213_0002 2017-03-18 22:43:27,005 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - The url to track the Tez Session: http://pc-1.thenet.edu:8088/proxy/application_1489855879213_0002/ ------------------------------------------------------------------------------------------------------------ Following portion of the log is after I hit the 'KILL' button -->>>> ------------------------------------------------------------------------------------------------------------ 2017-03-18 23:31:13,301 [Thread-3] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Shutting down thread pool 2017-03-18 23:31:13,302 [PigTezLauncher-0] ERROR org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager - Exception while waiting for Tez client to be ready java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:785) at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:105) at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:232) at org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:203) at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:210) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(Fu