WARNING: Use "yarn jar" to launch YARN applications. 16/07/19 19:11:51 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL 16/07/19 19:11:51 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE 16/07/19 19:11:51 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL 16/07/19 19:11:51 INFO pig.ExecTypeProvider: Trying ExecType : TEZ 16/07/19 19:11:51 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType 2016-07-19 19:11:52,092 [main] INFO org.apache.pig.Main - Apache Pig version 0.15.0.2.4.0.0-169 (rexported) compiled Feb 10 2016, 07:50:04 2016-07-19 19:11:52,093 [main] INFO org.apache.pig.Main - Logging error messages to: /hadoop/yarn/local/usercache/maria_dev/appcache/application_1468940847411_0003/container_e16_1468940847411_0003_01_000004/pig_1468955512051.log 2016-07-19 19:11:57,824 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/yarn/.pigbootup not found 2016-07-19 19:11:58,158 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox.hortonworks.com:8020 2016-07-19 19:12:02,498 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-script.pig-3910d5d6-b7d6-4739-b8d6-a42d82bd50d8 2016-07-19 19:12:04,474 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/ 2016-07-19 19:12:06,105 [main] INFO org.apache.pig.backend.hadoop.ATSService - Created ATS Hook 2016-07-19 19:12:08,930 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2016-07-19 19:12:09,112 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083 2016-07-19 19:12:09,222 [main] WARN org.apache.hadoop.security.UserGroupInformation - No groups available for user maria_dev 2016-07-19 19:12:09,283 [main] INFO hive.metastore - Connected to metastore. 2016-07-19 19:12:22,305 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2016-07-19 19:12:22,309 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083 2016-07-19 19:12:22,314 [main] WARN org.apache.hadoop.security.UserGroupInformation - No groups available for user maria_dev 2016-07-19 19:12:22,459 [main] INFO hive.metastore - Connected to metastore. 2016-07-19 19:12:26,584 [main] WARN org.apache.pig.newplan.BaseOperatorPlan - Encountered Warning IMPLICIT_CAST_TO_FLOAT 1 time(s). 2016-07-19 19:12:26,867 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2016-07-19 19:12:26,934 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083 2016-07-19 19:12:26,935 [main] WARN org.apache.hadoop.security.UserGroupInformation - No groups available for user maria_dev 2016-07-19 19:12:26,986 [main] INFO hive.metastore - Connected to metastore. 2016-07-19 19:12:28,887 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2016-07-19 19:12:28,928 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083 2016-07-19 19:12:28,931 [main] WARN org.apache.hadoop.security.UserGroupInformation - No groups available for user maria_dev 2016-07-19 19:12:28,957 [main] INFO hive.metastore - Connected to metastore. 2016-07-19 19:12:32,609 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: HASH_JOIN,GROUP_BY,FILTER 2016-07-19 19:12:32,737 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code. 2016-07-19 19:12:32,926 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]} 2016-07-19 19:12:33,123 [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for a: $0, $3, $4, $5, $6, $7, $8, $9 2016-07-19 19:12:33,418 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Tez staging directory is /tmp/yarn/staging and resources directory is /tmp/temp1542924427 2016-07-19 19:12:33,574 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.plan.TezCompiler - File concatenation threshold: 100 optimistic? false 2016-07-19 19:12:33,636 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil - Choosing to move algebraic foreach to combiner 2016-07-19 19:12:33,840 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2016-07-19 19:12:33,860 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083 2016-07-19 19:12:33,862 [main] WARN org.apache.hadoop.security.UserGroupInformation - No groups available for user maria_dev 2016-07-19 19:12:33,903 [main] INFO hive.metastore - Connected to metastore. 2016-07-19 19:12:35,058 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - 2016-07-19 19:12:35,095 [main] INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat - ORC pushdown predicate: null 2016-07-19 19:12:35,413 [main] INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat - FooterCacheHitRatio: 0/0 2016-07-19 19:12:35,414 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - 2016-07-19 19:12:35,422 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1 2016-07-19 19:12:35,513 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2016-07-19 19:12:35,535 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083 2016-07-19 19:12:35,542 [main] WARN org.apache.hadoop.security.UserGroupInformation - No groups available for user maria_dev 2016-07-19 19:12:35,544 [main] INFO hive.metastore - Connected to metastore. 2016-07-19 19:12:35,887 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - 2016-07-19 19:12:35,888 [main] INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat - ORC pushdown predicate: null 2016-07-19 19:12:35,895 [main] INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat - FooterCacheHitRatio: 0/0 2016-07-19 19:12:35,895 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - 2016-07-19 19:12:35,895 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1 2016-07-19 19:12:47,101 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: hive-exec-1.2.1000.2.4.0.0-169.jar 2016-07-19 19:12:47,102 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: libfb303-0.9.2.jar 2016-07-19 19:12:47,111 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: libthrift-0.9.2.jar 2016-07-19 19:12:47,111 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: jdo-api-3.0.1.jar 2016-07-19 19:12:47,112 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: hive-metastore-1.2.1000.2.4.0.0-169.jar 2016-07-19 19:12:47,112 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: antlr-runtime-3.4.jar 2016-07-19 19:12:47,112 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: hive-hbase-handler-1.2.1000.2.4.0.0-169.jar 2016-07-19 19:12:47,113 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: hive-hcatalog-pig-adapter-1.2.1000.2.4.0.0-169.jar 2016-07-19 19:12:47,113 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: pig-0.15.0.2.4.0.0-169-core-h2.jar 2016-07-19 19:12:47,113 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: automaton-1.11-8.jar 2016-07-19 19:12:47,113 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local resource: hive-hcatalog-core-1.2.1000.2.4.0.0-169.jar 2016-07-19 19:12:47,735 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.merge.percent to 0.66 from MR setting mapreduce.reduce.shuffle.merge.percent 2016-07-19 19:12:47,735 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.fetch.buffer.percent to 0.7 from MR setting mapreduce.reduce.shuffle.input.buffer.percent 2016-07-19 19:12:47,735 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.mb to 64 from MR setting mapreduce.task.io.sort.mb 2016-07-19 19:12:47,735 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.memory.limit.percent to 0.25 from MR setting mapreduce.reduce.shuffle.memory.limit.percent 2016-07-19 19:12:47,736 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.factor to 100 from MR setting mapreduce.task.io.sort.factor 2016-07-19 19:12:47,736 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.connect.timeout to 180000 from MR setting mapreduce.reduce.shuffle.connect.timeout 2016-07-19 19:12:47,736 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.internal.sorter.class to org.apache.hadoop.util.QuickSort from MR setting map.sort.class 2016-07-19 19:12:47,736 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.merge.progress.records to 10000 from MR setting mapreduce.task.merge.progress.records 2016-07-19 19:12:47,736 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress to false from MR setting mapreduce.map.output.compress 2016-07-19 19:12:47,737 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.sort.spill.percent to 0.7 from MR setting mapreduce.map.sort.spill.percent 2016-07-19 19:12:47,737 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.ssl.enable to false from MR setting mapreduce.shuffle.ssl.enabled 2016-07-19 19:12:47,737 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead to true from MR setting mapreduce.ifile.readahead 2016-07-19 19:12:47,737 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.parallel.copies to 30 from MR setting mapreduce.reduce.shuffle.parallelcopies 2016-07-19 19:12:47,737 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead.bytes to 4194304 from MR setting mapreduce.ifile.readahead.bytes 2016-07-19 19:12:47,737 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.task.input.post-merge.buffer.percent to 0.0 from MR setting mapreduce.reduce.input.buffer.percent 2016-07-19 19:12:47,737 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.credentials.path to /hadoop/yarn/local/usercache/maria_dev/appcache/application_1468940847411_0003/container_e16_1468940847411_0003_01_000004/container_tokens from MR setting mapreduce.job.credentials.binary 2016-07-19 19:12:47,738 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.read.timeout to 180000 from MR setting mapreduce.reduce.shuffle.read.timeout 2016-07-19 19:12:47,738 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress.codec to org.apache.hadoop.io.compress.DefaultCodec from MR setting mapreduce.map.output.compress.codec 2016-07-19 19:12:48,026 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - For vertex - scope-51: parallelism=1, memory=250, java opts=-Xmx256m 2016-07-19 19:12:48,274 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.merge.percent to 0.66 from MR setting mapreduce.reduce.shuffle.merge.percent 2016-07-19 19:12:48,274 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.fetch.buffer.percent to 0.7 from MR setting mapreduce.reduce.shuffle.input.buffer.percent 2016-07-19 19:12:48,275 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.mb to 64 from MR setting mapreduce.task.io.sort.mb 2016-07-19 19:12:48,275 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.memory.limit.percent to 0.25 from MR setting mapreduce.reduce.shuffle.memory.limit.percent 2016-07-19 19:12:48,275 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.factor to 100 from MR setting mapreduce.task.io.sort.factor 2016-07-19 19:12:48,275 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.connect.timeout to 180000 from MR setting mapreduce.reduce.shuffle.connect.timeout 2016-07-19 19:12:48,275 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.internal.sorter.class to org.apache.hadoop.util.QuickSort from MR setting map.sort.class 2016-07-19 19:12:48,275 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.merge.progress.records to 10000 from MR setting mapreduce.task.merge.progress.records 2016-07-19 19:12:48,275 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress to false from MR setting mapreduce.map.output.compress 2016-07-19 19:12:48,275 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.sort.spill.percent to 0.7 from MR setting mapreduce.map.sort.spill.percent 2016-07-19 19:12:48,276 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.ssl.enable to false from MR setting mapreduce.shuffle.ssl.enabled 2016-07-19 19:12:48,276 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead to true from MR setting mapreduce.ifile.readahead 2016-07-19 19:12:48,282 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.parallel.copies to 30 from MR setting mapreduce.reduce.shuffle.parallelcopies 2016-07-19 19:12:48,282 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead.bytes to 4194304 from MR setting mapreduce.ifile.readahead.bytes 2016-07-19 19:12:48,282 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.task.input.post-merge.buffer.percent to 0.0 from MR setting mapreduce.reduce.input.buffer.percent 2016-07-19 19:12:48,282 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.credentials.path to /hadoop/yarn/local/usercache/maria_dev/appcache/application_1468940847411_0003/container_e16_1468940847411_0003_01_000004/container_tokens from MR setting mapreduce.job.credentials.binary 2016-07-19 19:12:48,282 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.read.timeout to 180000 from MR setting mapreduce.reduce.shuffle.read.timeout 2016-07-19 19:12:48,283 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress.codec to org.apache.hadoop.io.compress.DefaultCodec from MR setting mapreduce.map.output.compress.codec 2016-07-19 19:12:48,510 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Set auto parallelism for vertex scope-52 2016-07-19 19:12:48,511 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - For vertex - scope-52: parallelism=1, memory=250, java opts=-Xmx256m 2016-07-19 19:12:48,627 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.merge.percent to 0.66 from MR setting mapreduce.reduce.shuffle.merge.percent 2016-07-19 19:12:48,627 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.fetch.buffer.percent to 0.7 from MR setting mapreduce.reduce.shuffle.input.buffer.percent 2016-07-19 19:12:48,627 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.mb to 64 from MR setting mapreduce.task.io.sort.mb 2016-07-19 19:12:48,627 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.memory.limit.percent to 0.25 from MR setting mapreduce.reduce.shuffle.memory.limit.percent 2016-07-19 19:12:48,627 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.factor to 100 from MR setting mapreduce.task.io.sort.factor 2016-07-19 19:12:48,628 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.connect.timeout to 180000 from MR setting mapreduce.reduce.shuffle.connect.timeout 2016-07-19 19:12:48,628 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.internal.sorter.class to org.apache.hadoop.util.QuickSort from MR setting map.sort.class 2016-07-19 19:12:48,628 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.merge.progress.records to 10000 from MR setting mapreduce.task.merge.progress.records 2016-07-19 19:12:48,628 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress to false from MR setting mapreduce.map.output.compress 2016-07-19 19:12:48,628 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.sort.spill.percent to 0.7 from MR setting mapreduce.map.sort.spill.percent 2016-07-19 19:12:48,628 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.ssl.enable to false from MR setting mapreduce.shuffle.ssl.enabled 2016-07-19 19:12:48,628 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead to true from MR setting mapreduce.ifile.readahead 2016-07-19 19:12:48,628 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.parallel.copies to 30 from MR setting mapreduce.reduce.shuffle.parallelcopies 2016-07-19 19:12:48,628 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead.bytes to 4194304 from MR setting mapreduce.ifile.readahead.bytes 2016-07-19 19:12:48,629 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.task.input.post-merge.buffer.percent to 0.0 from MR setting mapreduce.reduce.input.buffer.percent 2016-07-19 19:12:48,629 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.credentials.path to /hadoop/yarn/local/usercache/maria_dev/appcache/application_1468940847411_0003/container_e16_1468940847411_0003_01_000004/container_tokens from MR setting mapreduce.job.credentials.binary 2016-07-19 19:12:48,629 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.read.timeout to 180000 from MR setting mapreduce.reduce.shuffle.read.timeout 2016-07-19 19:12:48,629 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress.codec to org.apache.hadoop.io.compress.DefaultCodec from MR setting mapreduce.map.output.compress.codec 2016-07-19 19:12:48,772 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.merge.percent to 0.66 from MR setting mapreduce.reduce.shuffle.merge.percent 2016-07-19 19:12:48,772 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.fetch.buffer.percent to 0.7 from MR setting mapreduce.reduce.shuffle.input.buffer.percent 2016-07-19 19:12:48,772 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.mb to 64 from MR setting mapreduce.task.io.sort.mb 2016-07-19 19:12:48,772 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.memory.limit.percent to 0.25 from MR setting mapreduce.reduce.shuffle.memory.limit.percent 2016-07-19 19:12:48,772 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.factor to 100 from MR setting mapreduce.task.io.sort.factor 2016-07-19 19:12:48,773 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.connect.timeout to 180000 from MR setting mapreduce.reduce.shuffle.connect.timeout 2016-07-19 19:12:48,773 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.internal.sorter.class to org.apache.hadoop.util.QuickSort from MR setting map.sort.class 2016-07-19 19:12:48,773 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.merge.progress.records to 10000 from MR setting mapreduce.task.merge.progress.records 2016-07-19 19:12:48,773 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress to false from MR setting mapreduce.map.output.compress 2016-07-19 19:12:48,773 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.sort.spill.percent to 0.7 from MR setting mapreduce.map.sort.spill.percent 2016-07-19 19:12:48,773 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.ssl.enable to false from MR setting mapreduce.shuffle.ssl.enabled 2016-07-19 19:12:48,773 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead to true from MR setting mapreduce.ifile.readahead 2016-07-19 19:12:48,773 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.parallel.copies to 30 from MR setting mapreduce.reduce.shuffle.parallelcopies 2016-07-19 19:12:48,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead.bytes to 4194304 from MR setting mapreduce.ifile.readahead.bytes 2016-07-19 19:12:48,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.task.input.post-merge.buffer.percent to 0.0 from MR setting mapreduce.reduce.input.buffer.percent 2016-07-19 19:12:48,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.credentials.path to /hadoop/yarn/local/usercache/maria_dev/appcache/application_1468940847411_0003/container_e16_1468940847411_0003_01_000004/container_tokens from MR setting mapreduce.job.credentials.binary 2016-07-19 19:12:48,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.read.timeout to 180000 from MR setting mapreduce.reduce.shuffle.read.timeout 2016-07-19 19:12:48,774 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress.codec to org.apache.hadoop.io.compress.DefaultCodec from MR setting mapreduce.map.output.compress.codec 2016-07-19 19:12:48,822 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - For vertex - scope-53: parallelism=1, memory=250, java opts=-Xmx256m 2016-07-19 19:12:48,960 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.merge.percent to 0.66 from MR setting mapreduce.reduce.shuffle.merge.percent 2016-07-19 19:12:48,960 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.fetch.buffer.percent to 0.7 from MR setting mapreduce.reduce.shuffle.input.buffer.percent 2016-07-19 19:12:48,960 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.mb to 64 from MR setting mapreduce.task.io.sort.mb 2016-07-19 19:12:48,960 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.memory.limit.percent to 0.25 from MR setting mapreduce.reduce.shuffle.memory.limit.percent 2016-07-19 19:12:48,961 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.factor to 100 from MR setting mapreduce.task.io.sort.factor 2016-07-19 19:12:48,961 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.connect.timeout to 180000 from MR setting mapreduce.reduce.shuffle.connect.timeout 2016-07-19 19:12:48,961 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.internal.sorter.class to org.apache.hadoop.util.QuickSort from MR setting map.sort.class 2016-07-19 19:12:48,961 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.merge.progress.records to 10000 from MR setting mapreduce.task.merge.progress.records 2016-07-19 19:12:48,961 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress to false from MR setting mapreduce.map.output.compress 2016-07-19 19:12:48,961 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.sort.spill.percent to 0.7 from MR setting mapreduce.map.sort.spill.percent 2016-07-19 19:12:48,961 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.ssl.enable to false from MR setting mapreduce.shuffle.ssl.enabled 2016-07-19 19:12:48,961 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead to true from MR setting mapreduce.ifile.readahead 2016-07-19 19:12:48,962 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.parallel.copies to 30 from MR setting mapreduce.reduce.shuffle.parallelcopies 2016-07-19 19:12:48,962 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead.bytes to 4194304 from MR setting mapreduce.ifile.readahead.bytes 2016-07-19 19:12:48,962 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.task.input.post-merge.buffer.percent to 0.0 from MR setting mapreduce.reduce.input.buffer.percent 2016-07-19 19:12:48,962 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.credentials.path to /hadoop/yarn/local/usercache/maria_dev/appcache/application_1468940847411_0003/container_e16_1468940847411_0003_01_000004/container_tokens from MR setting mapreduce.job.credentials.binary 2016-07-19 19:12:48,962 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.read.timeout to 180000 from MR setting mapreduce.reduce.shuffle.read.timeout 2016-07-19 19:12:48,962 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress.codec to org.apache.hadoop.io.compress.DefaultCodec from MR setting mapreduce.map.output.compress.codec 2016-07-19 19:12:49,049 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Set auto parallelism for vertex scope-54 2016-07-19 19:12:49,050 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - For vertex - scope-54: parallelism=2, memory=250, java opts=-Xmx256m 2016-07-19 19:12:49,258 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist 2016-07-19 19:12:49,277 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083 2016-07-19 19:12:49,318 [main] WARN org.apache.hadoop.security.UserGroupInformation - No groups available for user maria_dev 2016-07-19 19:12:49,329 [main] INFO hive.metastore - Connected to metastore. 2016-07-19 19:12:49,881 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.merge.percent to 0.66 from MR setting mapreduce.reduce.shuffle.merge.percent 2016-07-19 19:12:49,881 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.fetch.buffer.percent to 0.7 from MR setting mapreduce.reduce.shuffle.input.buffer.percent 2016-07-19 19:12:49,881 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.mb to 64 from MR setting mapreduce.task.io.sort.mb 2016-07-19 19:12:49,882 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.memory.limit.percent to 0.25 from MR setting mapreduce.reduce.shuffle.memory.limit.percent 2016-07-19 19:12:49,882 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.factor to 100 from MR setting mapreduce.task.io.sort.factor 2016-07-19 19:12:49,882 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.connect.timeout to 180000 from MR setting mapreduce.reduce.shuffle.connect.timeout 2016-07-19 19:12:49,882 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.internal.sorter.class to org.apache.hadoop.util.QuickSort from MR setting map.sort.class 2016-07-19 19:12:49,882 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.merge.progress.records to 10000 from MR setting mapreduce.task.merge.progress.records 2016-07-19 19:12:49,882 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress to false from MR setting mapreduce.map.output.compress 2016-07-19 19:12:49,884 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.sort.spill.percent to 0.7 from MR setting mapreduce.map.sort.spill.percent 2016-07-19 19:12:49,884 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.ssl.enable to false from MR setting mapreduce.shuffle.ssl.enabled 2016-07-19 19:12:49,884 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead to true from MR setting mapreduce.ifile.readahead 2016-07-19 19:12:49,884 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.parallel.copies to 30 from MR setting mapreduce.reduce.shuffle.parallelcopies 2016-07-19 19:12:49,884 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead.bytes to 4194304 from MR setting mapreduce.ifile.readahead.bytes 2016-07-19 19:12:49,885 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.task.input.post-merge.buffer.percent to 0.0 from MR setting mapreduce.reduce.input.buffer.percent 2016-07-19 19:12:49,885 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.credentials.path to /hadoop/yarn/local/usercache/maria_dev/appcache/application_1468940847411_0003/container_e16_1468940847411_0003_01_000004/container_tokens from MR setting mapreduce.job.credentials.binary 2016-07-19 19:12:49,885 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.read.timeout to 180000 from MR setting mapreduce.reduce.shuffle.read.timeout 2016-07-19 19:12:49,885 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress.codec to org.apache.hadoop.io.compress.DefaultCodec from MR setting mapreduce.map.output.compress.codec 2016-07-19 19:12:49,996 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.merge.percent to 0.66 from MR setting mapreduce.reduce.shuffle.merge.percent 2016-07-19 19:12:49,996 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.fetch.buffer.percent to 0.7 from MR setting mapreduce.reduce.shuffle.input.buffer.percent 2016-07-19 19:12:49,996 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.mb to 64 from MR setting mapreduce.task.io.sort.mb 2016-07-19 19:12:49,996 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.memory.limit.percent to 0.25 from MR setting mapreduce.reduce.shuffle.memory.limit.percent 2016-07-19 19:12:49,996 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.io.sort.factor to 100 from MR setting mapreduce.task.io.sort.factor 2016-07-19 19:12:49,996 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.connect.timeout to 180000 from MR setting mapreduce.reduce.shuffle.connect.timeout 2016-07-19 19:12:49,997 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.internal.sorter.class to org.apache.hadoop.util.QuickSort from MR setting map.sort.class 2016-07-19 19:12:49,997 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.merge.progress.records to 10000 from MR setting mapreduce.task.merge.progress.records 2016-07-19 19:12:49,997 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress to false from MR setting mapreduce.map.output.compress 2016-07-19 19:12:49,997 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.sort.spill.percent to 0.7 from MR setting mapreduce.map.sort.spill.percent 2016-07-19 19:12:49,997 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.ssl.enable to false from MR setting mapreduce.shuffle.ssl.enabled 2016-07-19 19:12:49,997 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead to true from MR setting mapreduce.ifile.readahead 2016-07-19 19:12:49,997 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.parallel.copies to 30 from MR setting mapreduce.reduce.shuffle.parallelcopies 2016-07-19 19:12:49,997 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.ifile.readahead.bytes to 4194304 from MR setting mapreduce.ifile.readahead.bytes 2016-07-19 19:12:49,997 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.task.input.post-merge.buffer.percent to 0.0 from MR setting mapreduce.reduce.input.buffer.percent 2016-07-19 19:12:49,998 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.credentials.path to /hadoop/yarn/local/usercache/maria_dev/appcache/application_1468940847411_0003/container_e16_1468940847411_0003_01_000004/container_tokens from MR setting mapreduce.job.credentials.binary 2016-07-19 19:12:49,998 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.shuffle.read.timeout to 180000 from MR setting mapreduce.reduce.shuffle.read.timeout 2016-07-19 19:12:49,998 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - Setting tez.runtime.compress.codec to org.apache.hadoop.io.compress.DefaultCodec from MR setting mapreduce.map.output.compress.codec 2016-07-19 19:12:50,168 [PigTezLauncher-0] INFO org.apache.pig.tools.pigstats.tez.TezScriptState - Pig script settings are added to the job 2016-07-19 19:12:50,337 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Tez Client Version: [ component=tez-api, version=0.7.0.2.4.0.0-169, revision=3c1431f45faaca982ecc8dad13a107787b834696, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160210-0711 ] 2016-07-19 19:12:51,260 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/ 2016-07-19 19:12:51,388 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at sandbox.hortonworks.com/192.168.228.128:8050 2016-07-19 19:12:52,675 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Using org.apache.tez.dag.history.ats.acls.ATSV15HistoryACLPolicyManager to manage Timeline ACLs 2016-07-19 19:12:52,859 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/ 2016-07-19 19:12:52,889 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Session mode. Starting session. 2016-07-19 19:12:52,909 [PigTezLauncher-0] INFO org.apache.tez.common.security.TokenCache - Merging additional tokens from binary file, binaryFileName=/hadoop/yarn/local/usercache/maria_dev/appcache/application_1468940847411_0003/container_e16_1468940847411_0003_01_000004/container_tokens 2016-07-19 19:12:52,929 [PigTezLauncher-0] INFO org.apache.tez.client.TezClientUtils - Using tez.lib.uris value from configuration: /hdp/apps/2.4.0.0-169/tez/tez.tar.gz 2016-07-19 19:12:53,335 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Tez system stage directory hdfs://sandbox.hortonworks.com:8020/tmp/yarn/staging/.tez/application_1468940847411_0006 doesn't exist and is created 2016-07-19 19:12:53,654 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Writing domains for appattempt_1468940847411_0006_000001 to /ats/active/application_1468940847411_0006/appattempt_1468940847411_0006_000001/domainlog-appattempt_1468940847411_0006_000001 2016-07-19 19:12:53,839 [PigTezLauncher-0] INFO org.apache.tez.dag.history.ats.acls.ATSV15HistoryACLPolicyManager - Created Timeline Domain for History ACLs, domainId=Tez_ATS_application_1468940847411_0006 2016-07-19 19:12:58,719 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1468940847411_0006 2016-07-19 19:12:58,762 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - The url to track the Tez Session: http://sandbox.hortonworks.com:8088/proxy/application_1468940847411_0006/ 2016-07-19 19:15:30,911 [PigTezLauncher-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - Submitting DAG PigLatin:script.pig-0_scope-0 2016-07-19 19:15:30,912 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Submitting dag to TezSession, sessionName=PigLatin:script.pig, applicationId=application_1468940847411_0006, dagName=PigLatin:script.pig-0_scope-0, callerContext={ context=PIG, callerType=PIG_SCRIPT_ID, callerId=PIG-script.pig-3910d5d6-b7d6-4739-b8d6-a42d82bd50d8 } 2016-07-19 19:15:47,268 [PigTezLauncher-0] INFO org.apache.tez.client.TezClient - Submitted dag to TezSession, sessionName=PigLatin:script.pig, applicationId=application_1468940847411_0006, dagName=PigLatin:script.pig-0_scope-0 2016-07-19 19:15:48,493 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/ 2016-07-19 19:15:48,493 [PigTezLauncher-0] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at sandbox.hortonworks.com/192.168.228.128:8050 2016-07-19 19:15:48,500 [PigTezLauncher-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - Submitted DAG PigLatin:script.pig-0_scope-0. Application id: application_1468940847411_0006 2016-07-19 19:15:49,393 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - HadoopJobId: job_1468940847411_0006 2016-07-19 19:16:29,510 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:16:49,505 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:17:09,506 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:17:29,506 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:17:49,506 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:18:09,507 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:18:29,508 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:18:49,508 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:19:09,509 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:19:29,509 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:19:49,509 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:20:09,509 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:20:43,063 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:21:03,062 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:21:23,063 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:21:43,064 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:22:03,063 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:22:23,064 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:22:43,065 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:23:03,064 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:23:23,065 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:23:43,066 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:24:03,066 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:24:23,066 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:24:43,067 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:25:03,067 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:25:23,067 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:25:43,068 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:26:16,042 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:26:28,084 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:26:48,084 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:27:08,084 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:27:28,085 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:27:48,086 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:28:08,086 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:28:28,087 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:28:48,088 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:29:08,089 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:29:29,003 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:29:49,004 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:30:09,005 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:30:29,005 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:30:49,006 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:31:09,005 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:31:29,006 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:31:49,006 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:32:09,006 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:32:29,007 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:32:49,007 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:33:09,007 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:33:32,135 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 2 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:34:05,752 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 5 Succeeded: 0 Running: 1 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:34:25,751 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 4 Succeeded: 2 Running: 1 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:34:45,751 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 4 Succeeded: 2 Running: 1 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:35:05,751 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 4 Succeeded: 3 Running: 1 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:35:25,752 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 4 Succeeded: 3 Running: 1 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:35:45,752 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 4 Succeeded: 3 Running: 1 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:36:05,753 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:36:25,843 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:36:45,754 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:37:05,754 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=SUCCEEDED, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:37:25,754 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=SUCCEEDED, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:37:45,756 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=SUCCEEDED, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=null 2016-07-19 19:39:59,207 [PigTezLauncher-0] INFO org.apache.tez.common.counters.Limits - Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=3000, COUNTER_NAME_MAX=64, MAX_COUNTERS=10000 2016-07-19 19:40:08,100 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=SUCCEEDED, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=Counters: 171 org.apache.tez.common.counters.DAGCounter NUM_SUCCEEDED_TASKS=4 TOTAL_LAUNCHED_TASKS=4 DATA_LOCAL_TASKS=2 AM_CPU_MILLISECONDS=30700 AM_GC_TIME_MILLIS=12384 File System Counters FILE_BYTES_READ=3397 FILE_BYTES_WRITTEN=2335 FILE_READ_OPS=0 FILE_LARGE_READ_OPS=0 FILE_WRITE_OPS=0 HDFS_BYTES_READ=25911 HDFS_BYTES_WRITTEN=1455 HDFS_READ_OPS=9 HDFS_LARGE_READ_OPS=0 HDFS_WRITE_OPS=2 org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=298 REDUCE_INPUT_RECORDS=298 COMBINE_INPUT_RECORDS=0 SPILLED_RECORDS=596 NUM_SHUFFLED_INPUTS=5 NUM_SKIPPED_INPUTS=0 NUM_FAILED_SHUFFLE_INPUTS=0 MERGED_MAP_OUTPUTS=5 GC_TIME_MILLIS=186160 CPU_MILLISECONDS=60430 PHYSICAL_MEMORY_BYTES=915931136 VIRTUAL_MEMORY_BYTES=3519447040 COMMITTED_HEAP_BYTES=915931136 INPUT_RECORDS_PROCESSED=8100 OUTPUT_RECORDS=755 OUTPUT_BYTES=7458 OUTPUT_BYTES_WITH_OVERHEAD=4265 OUTPUT_BYTES_PHYSICAL=2191 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILLS_BYTES_READ=2191 ADDITIONAL_SPILL_COUNT=0 SHUFFLE_CHUNK_COUNT=3 SHUFFLE_BYTES=2191 SHUFFLE_BYTES_DECOMPRESSED=4265 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_DISK_DIRECT=2191 NUM_MEM_TO_DISK_MERGES=0 NUM_DISK_TO_DISK_MERGES=0 SHUFFLE_PHASE_TIME=405 MERGE_PHASE_TIME=1339 FIRST_EVENT_RECEIVED=159 LAST_EVENT_RECEIVED=160 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_52_INPUT_scope_51 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_52 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_53 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 TaskCounter_scope_51_INPUT_scope_2 INPUT_RECORDS_PROCESSED=8000 TaskCounter_scope_51_OUTPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=4992 OUTPUT_BYTES_PHYSICAL=605 OUTPUT_BYTES_WITH_OVERHEAD=1377 OUTPUT_RECORDS=457 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_52_INPUT_scope_51 ADDITIONAL_SPILLS_BYTES_READ=605 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=106 LAST_EVENT_RECEIVED=106 MERGED_MAP_OUTPUTS=1 MERGE_PHASE_TIME=1241 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=1 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=605 SHUFFLE_BYTES_DECOMPRESSED=1377 SHUFFLE_BYTES_DISK_DIRECT=605 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=322 SPILLED_RECORDS=99 TaskCounter_scope_52_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1074 OUTPUT_BYTES_PHYSICAL=683 OUTPUT_BYTES_WITH_OVERHEAD=1284 OUTPUT_RECORDS=99 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_53_INPUT_scope_24 INPUT_RECORDS_PROCESSED=100 TaskCounter_scope_53_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1392 OUTPUT_BYTES_PHYSICAL=903 OUTPUT_BYTES_WITH_OVERHEAD=1604 OUTPUT_RECORDS=100 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=100 TaskCounter_scope_54_INPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=683 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=28 LAST_EVENT_RECEIVED=28 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=43 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=683 SHUFFLE_BYTES_DECOMPRESSED=1284 SHUFFLE_BYTES_DISK_DIRECT=683 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=34 SPILLED_RECORDS=99 TaskCounter_scope_54_INPUT_scope_53 ADDITIONAL_SPILLS_BYTES_READ=903 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=25 LAST_EVENT_RECEIVED=26 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=55 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=100 REDUCE_INPUT_RECORDS=100 SHUFFLE_BYTES=903 SHUFFLE_BYTES_DECOMPRESSED=1604 SHUFFLE_BYTES_DISK_DIRECT=903 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=49 SPILLED_RECORDS=100 TaskCounter_scope_54_OUTPUT_scope_50 OUTPUT_RECORDS=99 org.apache.hadoop.mapreduce.TaskCounter COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_51_OUTPUT_scope_52 COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_52_INPUT_scope_51 COMBINE_INPUT_RECORDS=0 COMBINE_OUTPUT_RECORDS=0 2016-07-19 19:40:20,660 [PigTezLauncher-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=SUCCEEDED, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=Counters: 171 org.apache.tez.common.counters.DAGCounter NUM_SUCCEEDED_TASKS=4 TOTAL_LAUNCHED_TASKS=4 DATA_LOCAL_TASKS=2 AM_CPU_MILLISECONDS=30700 AM_GC_TIME_MILLIS=12384 File System Counters FILE_BYTES_READ=3397 FILE_BYTES_WRITTEN=2335 FILE_READ_OPS=0 FILE_LARGE_READ_OPS=0 FILE_WRITE_OPS=0 HDFS_BYTES_READ=25911 HDFS_BYTES_WRITTEN=1455 HDFS_READ_OPS=9 HDFS_LARGE_READ_OPS=0 HDFS_WRITE_OPS=2 org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=298 REDUCE_INPUT_RECORDS=298 COMBINE_INPUT_RECORDS=0 SPILLED_RECORDS=596 NUM_SHUFFLED_INPUTS=5 NUM_SKIPPED_INPUTS=0 NUM_FAILED_SHUFFLE_INPUTS=0 MERGED_MAP_OUTPUTS=5 GC_TIME_MILLIS=186160 CPU_MILLISECONDS=60430 PHYSICAL_MEMORY_BYTES=915931136 VIRTUAL_MEMORY_BYTES=3519447040 COMMITTED_HEAP_BYTES=915931136 INPUT_RECORDS_PROCESSED=8100 OUTPUT_RECORDS=755 OUTPUT_BYTES=7458 OUTPUT_BYTES_WITH_OVERHEAD=4265 OUTPUT_BYTES_PHYSICAL=2191 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILLS_BYTES_READ=2191 ADDITIONAL_SPILL_COUNT=0 SHUFFLE_CHUNK_COUNT=3 SHUFFLE_BYTES=2191 SHUFFLE_BYTES_DECOMPRESSED=4265 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_DISK_DIRECT=2191 NUM_MEM_TO_DISK_MERGES=0 NUM_DISK_TO_DISK_MERGES=0 SHUFFLE_PHASE_TIME=405 MERGE_PHASE_TIME=1339 FIRST_EVENT_RECEIVED=159 LAST_EVENT_RECEIVED=160 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_52_INPUT_scope_51 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_52 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_53 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 TaskCounter_scope_51_INPUT_scope_2 INPUT_RECORDS_PROCESSED=8000 TaskCounter_scope_51_OUTPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=4992 OUTPUT_BYTES_PHYSICAL=605 OUTPUT_BYTES_WITH_OVERHEAD=1377 OUTPUT_RECORDS=457 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_52_INPUT_scope_51 ADDITIONAL_SPILLS_BYTES_READ=605 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=106 LAST_EVENT_RECEIVED=106 MERGED_MAP_OUTPUTS=1 MERGE_PHASE_TIME=1241 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=1 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=605 SHUFFLE_BYTES_DECOMPRESSED=1377 SHUFFLE_BYTES_DISK_DIRECT=605 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=322 SPILLED_RECORDS=99 TaskCounter_scope_52_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1074 OUTPUT_BYTES_PHYSICAL=683 OUTPUT_BYTES_WITH_OVERHEAD=1284 OUTPUT_RECORDS=99 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_53_INPUT_scope_24 INPUT_RECORDS_PROCESSED=100 TaskCounter_scope_53_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1392 OUTPUT_BYTES_PHYSICAL=903 OUTPUT_BYTES_WITH_OVERHEAD=1604 OUTPUT_RECORDS=100 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=100 TaskCounter_scope_54_INPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=683 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=28 LAST_EVENT_RECEIVED=28 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=43 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=683 SHUFFLE_BYTES_DECOMPRESSED=1284 SHUFFLE_BYTES_DISK_DIRECT=683 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=34 SPILLED_RECORDS=99 TaskCounter_scope_54_INPUT_scope_53 ADDITIONAL_SPILLS_BYTES_READ=903 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=25 LAST_EVENT_RECEIVED=26 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=55 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=100 REDUCE_INPUT_RECORDS=100 SHUFFLE_BYTES=903 SHUFFLE_BYTES_DECOMPRESSED=1604 SHUFFLE_BYTES_DISK_DIRECT=903 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=49 SPILLED_RECORDS=100 TaskCounter_scope_54_OUTPUT_scope_50 OUTPUT_RECORDS=99 org.apache.hadoop.mapreduce.TaskCounter COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_51_OUTPUT_scope_52 COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_52_INPUT_scope_51 COMBINE_INPUT_RECORDS=0 COMBINE_OUTPUT_RECORDS=0 2016-07-19 19:42:52,853 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=SUCCEEDED, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=Counters: 171 org.apache.tez.common.counters.DAGCounter NUM_SUCCEEDED_TASKS=4 TOTAL_LAUNCHED_TASKS=4 DATA_LOCAL_TASKS=2 AM_CPU_MILLISECONDS=30700 AM_GC_TIME_MILLIS=12384 File System Counters FILE_BYTES_READ=3397 FILE_BYTES_WRITTEN=2335 FILE_READ_OPS=0 FILE_LARGE_READ_OPS=0 FILE_WRITE_OPS=0 HDFS_BYTES_READ=25911 HDFS_BYTES_WRITTEN=1455 HDFS_READ_OPS=9 HDFS_LARGE_READ_OPS=0 HDFS_WRITE_OPS=2 org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=298 REDUCE_INPUT_RECORDS=298 COMBINE_INPUT_RECORDS=0 SPILLED_RECORDS=596 NUM_SHUFFLED_INPUTS=5 NUM_SKIPPED_INPUTS=0 NUM_FAILED_SHUFFLE_INPUTS=0 MERGED_MAP_OUTPUTS=5 GC_TIME_MILLIS=186160 CPU_MILLISECONDS=60430 PHYSICAL_MEMORY_BYTES=915931136 VIRTUAL_MEMORY_BYTES=3519447040 COMMITTED_HEAP_BYTES=915931136 INPUT_RECORDS_PROCESSED=8100 OUTPUT_RECORDS=755 OUTPUT_BYTES=7458 OUTPUT_BYTES_WITH_OVERHEAD=4265 OUTPUT_BYTES_PHYSICAL=2191 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILLS_BYTES_READ=2191 ADDITIONAL_SPILL_COUNT=0 SHUFFLE_CHUNK_COUNT=3 SHUFFLE_BYTES=2191 SHUFFLE_BYTES_DECOMPRESSED=4265 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_DISK_DIRECT=2191 NUM_MEM_TO_DISK_MERGES=0 NUM_DISK_TO_DISK_MERGES=0 SHUFFLE_PHASE_TIME=405 MERGE_PHASE_TIME=1339 FIRST_EVENT_RECEIVED=159 LAST_EVENT_RECEIVED=160 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_52_INPUT_scope_51 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_52 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_53 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 TaskCounter_scope_51_INPUT_scope_2 INPUT_RECORDS_PROCESSED=8000 TaskCounter_scope_51_OUTPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=4992 OUTPUT_BYTES_PHYSICAL=605 OUTPUT_BYTES_WITH_OVERHEAD=1377 OUTPUT_RECORDS=457 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_52_INPUT_scope_51 ADDITIONAL_SPILLS_BYTES_READ=605 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=106 LAST_EVENT_RECEIVED=106 MERGED_MAP_OUTPUTS=1 MERGE_PHASE_TIME=1241 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=1 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=605 SHUFFLE_BYTES_DECOMPRESSED=1377 SHUFFLE_BYTES_DISK_DIRECT=605 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=322 SPILLED_RECORDS=99 TaskCounter_scope_52_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1074 OUTPUT_BYTES_PHYSICAL=683 OUTPUT_BYTES_WITH_OVERHEAD=1284 OUTPUT_RECORDS=99 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_53_INPUT_scope_24 INPUT_RECORDS_PROCESSED=100 TaskCounter_scope_53_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1392 OUTPUT_BYTES_PHYSICAL=903 OUTPUT_BYTES_WITH_OVERHEAD=1604 OUTPUT_RECORDS=100 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=100 TaskCounter_scope_54_INPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=683 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=28 LAST_EVENT_RECEIVED=28 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=43 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=683 SHUFFLE_BYTES_DECOMPRESSED=1284 SHUFFLE_BYTES_DISK_DIRECT=683 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=34 SPILLED_RECORDS=99 TaskCounter_scope_54_INPUT_scope_53 ADDITIONAL_SPILLS_BYTES_READ=903 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=25 LAST_EVENT_RECEIVED=26 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=55 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=100 REDUCE_INPUT_RECORDS=100 SHUFFLE_BYTES=903 SHUFFLE_BYTES_DECOMPRESSED=1604 SHUFFLE_BYTES_DISK_DIRECT=903 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=49 SPILLED_RECORDS=100 TaskCounter_scope_54_OUTPUT_scope_50 OUTPUT_RECORDS=99 org.apache.hadoop.mapreduce.TaskCounter COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_51_OUTPUT_scope_52 COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_52_INPUT_scope_51 COMBINE_INPUT_RECORDS=0 COMBINE_OUTPUT_RECORDS=0 2016-07-19 19:42:55,586 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=SUCCEEDED, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=Counters: 171 org.apache.tez.common.counters.DAGCounter NUM_SUCCEEDED_TASKS=4 TOTAL_LAUNCHED_TASKS=4 DATA_LOCAL_TASKS=2 AM_CPU_MILLISECONDS=30700 AM_GC_TIME_MILLIS=12384 File System Counters FILE_BYTES_READ=3397 FILE_BYTES_WRITTEN=2335 FILE_READ_OPS=0 FILE_LARGE_READ_OPS=0 FILE_WRITE_OPS=0 HDFS_BYTES_READ=25911 HDFS_BYTES_WRITTEN=1455 HDFS_READ_OPS=9 HDFS_LARGE_READ_OPS=0 HDFS_WRITE_OPS=2 org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=298 REDUCE_INPUT_RECORDS=298 COMBINE_INPUT_RECORDS=0 SPILLED_RECORDS=596 NUM_SHUFFLED_INPUTS=5 NUM_SKIPPED_INPUTS=0 NUM_FAILED_SHUFFLE_INPUTS=0 MERGED_MAP_OUTPUTS=5 GC_TIME_MILLIS=186160 CPU_MILLISECONDS=60430 PHYSICAL_MEMORY_BYTES=915931136 VIRTUAL_MEMORY_BYTES=3519447040 COMMITTED_HEAP_BYTES=915931136 INPUT_RECORDS_PROCESSED=8100 OUTPUT_RECORDS=755 OUTPUT_BYTES=7458 OUTPUT_BYTES_WITH_OVERHEAD=4265 OUTPUT_BYTES_PHYSICAL=2191 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILLS_BYTES_READ=2191 ADDITIONAL_SPILL_COUNT=0 SHUFFLE_CHUNK_COUNT=3 SHUFFLE_BYTES=2191 SHUFFLE_BYTES_DECOMPRESSED=4265 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_DISK_DIRECT=2191 NUM_MEM_TO_DISK_MERGES=0 NUM_DISK_TO_DISK_MERGES=0 SHUFFLE_PHASE_TIME=405 MERGE_PHASE_TIME=1339 FIRST_EVENT_RECEIVED=159 LAST_EVENT_RECEIVED=160 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_52_INPUT_scope_51 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_52 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_53 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 TaskCounter_scope_51_INPUT_scope_2 INPUT_RECORDS_PROCESSED=8000 TaskCounter_scope_51_OUTPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=4992 OUTPUT_BYTES_PHYSICAL=605 OUTPUT_BYTES_WITH_OVERHEAD=1377 OUTPUT_RECORDS=457 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_52_INPUT_scope_51 ADDITIONAL_SPILLS_BYTES_READ=605 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=106 LAST_EVENT_RECEIVED=106 MERGED_MAP_OUTPUTS=1 MERGE_PHASE_TIME=1241 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=1 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=605 SHUFFLE_BYTES_DECOMPRESSED=1377 SHUFFLE_BYTES_DISK_DIRECT=605 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=322 SPILLED_RECORDS=99 TaskCounter_scope_52_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1074 OUTPUT_BYTES_PHYSICAL=683 OUTPUT_BYTES_WITH_OVERHEAD=1284 OUTPUT_RECORDS=99 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_53_INPUT_scope_24 INPUT_RECORDS_PROCESSED=100 TaskCounter_scope_53_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1392 OUTPUT_BYTES_PHYSICAL=903 OUTPUT_BYTES_WITH_OVERHEAD=1604 OUTPUT_RECORDS=100 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=100 TaskCounter_scope_54_INPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=683 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=28 LAST_EVENT_RECEIVED=28 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=43 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=683 SHUFFLE_BYTES_DECOMPRESSED=1284 SHUFFLE_BYTES_DISK_DIRECT=683 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=34 SPILLED_RECORDS=99 TaskCounter_scope_54_INPUT_scope_53 ADDITIONAL_SPILLS_BYTES_READ=903 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=25 LAST_EVENT_RECEIVED=26 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=55 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=100 REDUCE_INPUT_RECORDS=100 SHUFFLE_BYTES=903 SHUFFLE_BYTES_DECOMPRESSED=1604 SHUFFLE_BYTES_DISK_DIRECT=903 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=49 SPILLED_RECORDS=100 TaskCounter_scope_54_OUTPUT_scope_50 OUTPUT_RECORDS=99 org.apache.hadoop.mapreduce.TaskCounter COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_51_OUTPUT_scope_52 COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_52_INPUT_scope_51 COMBINE_INPUT_RECORDS=0 COMBINE_OUTPUT_RECORDS=0 2016-07-19 19:43:15,097 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=SUCCEEDED, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=Counters: 171 org.apache.tez.common.counters.DAGCounter NUM_SUCCEEDED_TASKS=4 TOTAL_LAUNCHED_TASKS=4 DATA_LOCAL_TASKS=2 AM_CPU_MILLISECONDS=30700 AM_GC_TIME_MILLIS=12384 File System Counters FILE_BYTES_READ=3397 FILE_BYTES_WRITTEN=2335 FILE_READ_OPS=0 FILE_LARGE_READ_OPS=0 FILE_WRITE_OPS=0 HDFS_BYTES_READ=25911 HDFS_BYTES_WRITTEN=1455 HDFS_READ_OPS=9 HDFS_LARGE_READ_OPS=0 HDFS_WRITE_OPS=2 org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=298 REDUCE_INPUT_RECORDS=298 COMBINE_INPUT_RECORDS=0 SPILLED_RECORDS=596 NUM_SHUFFLED_INPUTS=5 NUM_SKIPPED_INPUTS=0 NUM_FAILED_SHUFFLE_INPUTS=0 MERGED_MAP_OUTPUTS=5 GC_TIME_MILLIS=186160 CPU_MILLISECONDS=60430 PHYSICAL_MEMORY_BYTES=915931136 VIRTUAL_MEMORY_BYTES=3519447040 COMMITTED_HEAP_BYTES=915931136 INPUT_RECORDS_PROCESSED=8100 OUTPUT_RECORDS=755 OUTPUT_BYTES=7458 OUTPUT_BYTES_WITH_OVERHEAD=4265 OUTPUT_BYTES_PHYSICAL=2191 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILLS_BYTES_READ=2191 ADDITIONAL_SPILL_COUNT=0 SHUFFLE_CHUNK_COUNT=3 SHUFFLE_BYTES=2191 SHUFFLE_BYTES_DECOMPRESSED=4265 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_DISK_DIRECT=2191 NUM_MEM_TO_DISK_MERGES=0 NUM_DISK_TO_DISK_MERGES=0 SHUFFLE_PHASE_TIME=405 MERGE_PHASE_TIME=1339 FIRST_EVENT_RECEIVED=159 LAST_EVENT_RECEIVED=160 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_52_INPUT_scope_51 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_52 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_53 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 TaskCounter_scope_51_INPUT_scope_2 INPUT_RECORDS_PROCESSED=8000 TaskCounter_scope_51_OUTPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=4992 OUTPUT_BYTES_PHYSICAL=605 OUTPUT_BYTES_WITH_OVERHEAD=1377 OUTPUT_RECORDS=457 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_52_INPUT_scope_51 ADDITIONAL_SPILLS_BYTES_READ=605 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=106 LAST_EVENT_RECEIVED=106 MERGED_MAP_OUTPUTS=1 MERGE_PHASE_TIME=1241 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=1 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=605 SHUFFLE_BYTES_DECOMPRESSED=1377 SHUFFLE_BYTES_DISK_DIRECT=605 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=322 SPILLED_RECORDS=99 TaskCounter_scope_52_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1074 OUTPUT_BYTES_PHYSICAL=683 OUTPUT_BYTES_WITH_OVERHEAD=1284 OUTPUT_RECORDS=99 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_53_INPUT_scope_24 INPUT_RECORDS_PROCESSED=100 TaskCounter_scope_53_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1392 OUTPUT_BYTES_PHYSICAL=903 OUTPUT_BYTES_WITH_OVERHEAD=1604 OUTPUT_RECORDS=100 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=100 TaskCounter_scope_54_INPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=683 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=28 LAST_EVENT_RECEIVED=28 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=43 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=683 SHUFFLE_BYTES_DECOMPRESSED=1284 SHUFFLE_BYTES_DISK_DIRECT=683 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=34 SPILLED_RECORDS=99 TaskCounter_scope_54_INPUT_scope_53 ADDITIONAL_SPILLS_BYTES_READ=903 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=25 LAST_EVENT_RECEIVED=26 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=55 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=100 REDUCE_INPUT_RECORDS=100 SHUFFLE_BYTES=903 SHUFFLE_BYTES_DECOMPRESSED=1604 SHUFFLE_BYTES_DISK_DIRECT=903 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=49 SPILLED_RECORDS=100 TaskCounter_scope_54_OUTPUT_scope_50 OUTPUT_RECORDS=99 org.apache.hadoop.mapreduce.TaskCounter COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_51_OUTPUT_scope_52 COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_52_INPUT_scope_51 COMBINE_INPUT_RECORDS=0 COMBINE_OUTPUT_RECORDS=0 2016-07-19 19:43:26,201 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:28,080 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:29,123 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:30,172 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:31,173 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:32,202 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:33,301 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:34,402 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:35,209 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=SUCCEEDED, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=Counters: 171 org.apache.tez.common.counters.DAGCounter NUM_SUCCEEDED_TASKS=4 TOTAL_LAUNCHED_TASKS=4 DATA_LOCAL_TASKS=2 AM_CPU_MILLISECONDS=30700 AM_GC_TIME_MILLIS=12384 File System Counters FILE_BYTES_READ=3397 FILE_BYTES_WRITTEN=2335 FILE_READ_OPS=0 FILE_LARGE_READ_OPS=0 FILE_WRITE_OPS=0 HDFS_BYTES_READ=25911 HDFS_BYTES_WRITTEN=1455 HDFS_READ_OPS=9 HDFS_LARGE_READ_OPS=0 HDFS_WRITE_OPS=2 org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=298 REDUCE_INPUT_RECORDS=298 COMBINE_INPUT_RECORDS=0 SPILLED_RECORDS=596 NUM_SHUFFLED_INPUTS=5 NUM_SKIPPED_INPUTS=0 NUM_FAILED_SHUFFLE_INPUTS=0 MERGED_MAP_OUTPUTS=5 GC_TIME_MILLIS=186160 CPU_MILLISECONDS=60430 PHYSICAL_MEMORY_BYTES=915931136 VIRTUAL_MEMORY_BYTES=3519447040 COMMITTED_HEAP_BYTES=915931136 INPUT_RECORDS_PROCESSED=8100 OUTPUT_RECORDS=755 OUTPUT_BYTES=7458 OUTPUT_BYTES_WITH_OVERHEAD=4265 OUTPUT_BYTES_PHYSICAL=2191 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILLS_BYTES_READ=2191 ADDITIONAL_SPILL_COUNT=0 SHUFFLE_CHUNK_COUNT=3 SHUFFLE_BYTES=2191 SHUFFLE_BYTES_DECOMPRESSED=4265 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_DISK_DIRECT=2191 NUM_MEM_TO_DISK_MERGES=0 NUM_DISK_TO_DISK_MERGES=0 SHUFFLE_PHASE_TIME=405 MERGE_PHASE_TIME=1339 FIRST_EVENT_RECEIVED=159 LAST_EVENT_RECEIVED=160 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_52_INPUT_scope_51 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_52 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_53 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 TaskCounter_scope_51_INPUT_scope_2 INPUT_RECORDS_PROCESSED=8000 TaskCounter_scope_51_OUTPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=4992 OUTPUT_BYTES_PHYSICAL=605 OUTPUT_BYTES_WITH_OVERHEAD=1377 OUTPUT_RECORDS=457 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_52_INPUT_scope_51 ADDITIONAL_SPILLS_BYTES_READ=605 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=106 LAST_EVENT_RECEIVED=106 MERGED_MAP_OUTPUTS=1 MERGE_PHASE_TIME=1241 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=1 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=605 SHUFFLE_BYTES_DECOMPRESSED=1377 SHUFFLE_BYTES_DISK_DIRECT=605 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=322 SPILLED_RECORDS=99 TaskCounter_scope_52_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1074 OUTPUT_BYTES_PHYSICAL=683 OUTPUT_BYTES_WITH_OVERHEAD=1284 OUTPUT_RECORDS=99 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_53_INPUT_scope_24 INPUT_RECORDS_PROCESSED=100 TaskCounter_scope_53_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1392 OUTPUT_BYTES_PHYSICAL=903 OUTPUT_BYTES_WITH_OVERHEAD=1604 OUTPUT_RECORDS=100 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=100 TaskCounter_scope_54_INPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=683 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=28 LAST_EVENT_RECEIVED=28 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=43 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=683 SHUFFLE_BYTES_DECOMPRESSED=1284 SHUFFLE_BYTES_DISK_DIRECT=683 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=34 SPILLED_RECORDS=99 TaskCounter_scope_54_INPUT_scope_53 ADDITIONAL_SPILLS_BYTES_READ=903 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=25 LAST_EVENT_RECEIVED=26 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=55 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=100 REDUCE_INPUT_RECORDS=100 SHUFFLE_BYTES=903 SHUFFLE_BYTES_DECOMPRESSED=1604 SHUFFLE_BYTES_DISK_DIRECT=903 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=49 SPILLED_RECORDS=100 TaskCounter_scope_54_OUTPUT_scope_50 OUTPUT_RECORDS=99 org.apache.hadoop.mapreduce.TaskCounter COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_51_OUTPUT_scope_52 COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_52_INPUT_scope_51 COMBINE_INPUT_RECORDS=0 COMBINE_OUTPUT_RECORDS=0 2016-07-19 19:43:35,885 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:37,076 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:38,078 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 10 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:39,079 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 11 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:40,080 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 12 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:41,081 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 13 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:42,085 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 14 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:43,113 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 15 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:44,114 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 16 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:45,115 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 17 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:46,117 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 18 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:47,118 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 19 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:48,120 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 20 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:49,130 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 21 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:50,131 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 22 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:51,149 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 23 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:52,152 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 24 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:53,154 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 25 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:54,155 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 26 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:54,967 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=SUCCEEDED, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=Counters: 171 org.apache.tez.common.counters.DAGCounter NUM_SUCCEEDED_TASKS=4 TOTAL_LAUNCHED_TASKS=4 DATA_LOCAL_TASKS=2 AM_CPU_MILLISECONDS=30700 AM_GC_TIME_MILLIS=12384 File System Counters FILE_BYTES_READ=3397 FILE_BYTES_WRITTEN=2335 FILE_READ_OPS=0 FILE_LARGE_READ_OPS=0 FILE_WRITE_OPS=0 HDFS_BYTES_READ=25911 HDFS_BYTES_WRITTEN=1455 HDFS_READ_OPS=9 HDFS_LARGE_READ_OPS=0 HDFS_WRITE_OPS=2 org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=298 REDUCE_INPUT_RECORDS=298 COMBINE_INPUT_RECORDS=0 SPILLED_RECORDS=596 NUM_SHUFFLED_INPUTS=5 NUM_SKIPPED_INPUTS=0 NUM_FAILED_SHUFFLE_INPUTS=0 MERGED_MAP_OUTPUTS=5 GC_TIME_MILLIS=186160 CPU_MILLISECONDS=60430 PHYSICAL_MEMORY_BYTES=915931136 VIRTUAL_MEMORY_BYTES=3519447040 COMMITTED_HEAP_BYTES=915931136 INPUT_RECORDS_PROCESSED=8100 OUTPUT_RECORDS=755 OUTPUT_BYTES=7458 OUTPUT_BYTES_WITH_OVERHEAD=4265 OUTPUT_BYTES_PHYSICAL=2191 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILLS_BYTES_READ=2191 ADDITIONAL_SPILL_COUNT=0 SHUFFLE_CHUNK_COUNT=3 SHUFFLE_BYTES=2191 SHUFFLE_BYTES_DECOMPRESSED=4265 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_DISK_DIRECT=2191 NUM_MEM_TO_DISK_MERGES=0 NUM_DISK_TO_DISK_MERGES=0 SHUFFLE_PHASE_TIME=405 MERGE_PHASE_TIME=1339 FIRST_EVENT_RECEIVED=159 LAST_EVENT_RECEIVED=160 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_52_INPUT_scope_51 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_52 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_53 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 TaskCounter_scope_51_INPUT_scope_2 INPUT_RECORDS_PROCESSED=8000 TaskCounter_scope_51_OUTPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=4992 OUTPUT_BYTES_PHYSICAL=605 OUTPUT_BYTES_WITH_OVERHEAD=1377 OUTPUT_RECORDS=457 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_52_INPUT_scope_51 ADDITIONAL_SPILLS_BYTES_READ=605 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=106 LAST_EVENT_RECEIVED=106 MERGED_MAP_OUTPUTS=1 MERGE_PHASE_TIME=1241 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=1 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=605 SHUFFLE_BYTES_DECOMPRESSED=1377 SHUFFLE_BYTES_DISK_DIRECT=605 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=322 SPILLED_RECORDS=99 TaskCounter_scope_52_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1074 OUTPUT_BYTES_PHYSICAL=683 OUTPUT_BYTES_WITH_OVERHEAD=1284 OUTPUT_RECORDS=99 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_53_INPUT_scope_24 INPUT_RECORDS_PROCESSED=100 TaskCounter_scope_53_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1392 OUTPUT_BYTES_PHYSICAL=903 OUTPUT_BYTES_WITH_OVERHEAD=1604 OUTPUT_RECORDS=100 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=100 TaskCounter_scope_54_INPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=683 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=28 LAST_EVENT_RECEIVED=28 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=43 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=683 SHUFFLE_BYTES_DECOMPRESSED=1284 SHUFFLE_BYTES_DISK_DIRECT=683 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=34 SPILLED_RECORDS=99 TaskCounter_scope_54_INPUT_scope_53 ADDITIONAL_SPILLS_BYTES_READ=903 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=25 LAST_EVENT_RECEIVED=26 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=55 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=100 REDUCE_INPUT_RECORDS=100 SHUFFLE_BYTES=903 SHUFFLE_BYTES_DECOMPRESSED=1604 SHUFFLE_BYTES_DISK_DIRECT=903 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=49 SPILLED_RECORDS=100 TaskCounter_scope_54_OUTPUT_scope_50 OUTPUT_RECORDS=99 org.apache.hadoop.mapreduce.TaskCounter COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_51_OUTPUT_scope_52 COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_52_INPUT_scope_51 COMBINE_INPUT_RECORDS=0 COMBINE_OUTPUT_RECORDS=0 2016-07-19 19:43:55,157 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 27 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:56,157 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 28 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:57,164 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 29 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:58,165 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 30 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:43:59,166 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 31 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:00,167 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 32 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:01,168 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 33 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:02,175 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 34 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:03,176 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 35 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:04,177 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 36 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:05,187 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 37 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:06,188 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 38 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:07,190 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 39 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:08,200 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 40 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:09,202 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 41 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:10,203 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 42 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:11,214 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 43 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:12,215 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 44 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:13,225 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 45 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:14,226 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 46 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:15,060 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=SUCCEEDED, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=Counters: 171 org.apache.tez.common.counters.DAGCounter NUM_SUCCEEDED_TASKS=4 TOTAL_LAUNCHED_TASKS=4 DATA_LOCAL_TASKS=2 AM_CPU_MILLISECONDS=30700 AM_GC_TIME_MILLIS=12384 File System Counters FILE_BYTES_READ=3397 FILE_BYTES_WRITTEN=2335 FILE_READ_OPS=0 FILE_LARGE_READ_OPS=0 FILE_WRITE_OPS=0 HDFS_BYTES_READ=25911 HDFS_BYTES_WRITTEN=1455 HDFS_READ_OPS=9 HDFS_LARGE_READ_OPS=0 HDFS_WRITE_OPS=2 org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=298 REDUCE_INPUT_RECORDS=298 COMBINE_INPUT_RECORDS=0 SPILLED_RECORDS=596 NUM_SHUFFLED_INPUTS=5 NUM_SKIPPED_INPUTS=0 NUM_FAILED_SHUFFLE_INPUTS=0 MERGED_MAP_OUTPUTS=5 GC_TIME_MILLIS=186160 CPU_MILLISECONDS=60430 PHYSICAL_MEMORY_BYTES=915931136 VIRTUAL_MEMORY_BYTES=3519447040 COMMITTED_HEAP_BYTES=915931136 INPUT_RECORDS_PROCESSED=8100 OUTPUT_RECORDS=755 OUTPUT_BYTES=7458 OUTPUT_BYTES_WITH_OVERHEAD=4265 OUTPUT_BYTES_PHYSICAL=2191 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILLS_BYTES_READ=2191 ADDITIONAL_SPILL_COUNT=0 SHUFFLE_CHUNK_COUNT=3 SHUFFLE_BYTES=2191 SHUFFLE_BYTES_DECOMPRESSED=4265 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_DISK_DIRECT=2191 NUM_MEM_TO_DISK_MERGES=0 NUM_DISK_TO_DISK_MERGES=0 SHUFFLE_PHASE_TIME=405 MERGE_PHASE_TIME=1339 FIRST_EVENT_RECEIVED=159 LAST_EVENT_RECEIVED=160 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_52_INPUT_scope_51 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_52 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_53 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 TaskCounter_scope_51_INPUT_scope_2 INPUT_RECORDS_PROCESSED=8000 TaskCounter_scope_51_OUTPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=4992 OUTPUT_BYTES_PHYSICAL=605 OUTPUT_BYTES_WITH_OVERHEAD=1377 OUTPUT_RECORDS=457 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_52_INPUT_scope_51 ADDITIONAL_SPILLS_BYTES_READ=605 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=106 LAST_EVENT_RECEIVED=106 MERGED_MAP_OUTPUTS=1 MERGE_PHASE_TIME=1241 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=1 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=605 SHUFFLE_BYTES_DECOMPRESSED=1377 SHUFFLE_BYTES_DISK_DIRECT=605 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=322 SPILLED_RECORDS=99 TaskCounter_scope_52_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1074 OUTPUT_BYTES_PHYSICAL=683 OUTPUT_BYTES_WITH_OVERHEAD=1284 OUTPUT_RECORDS=99 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_53_INPUT_scope_24 INPUT_RECORDS_PROCESSED=100 TaskCounter_scope_53_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1392 OUTPUT_BYTES_PHYSICAL=903 OUTPUT_BYTES_WITH_OVERHEAD=1604 OUTPUT_RECORDS=100 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=100 TaskCounter_scope_54_INPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=683 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=28 LAST_EVENT_RECEIVED=28 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=43 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=683 SHUFFLE_BYTES_DECOMPRESSED=1284 SHUFFLE_BYTES_DISK_DIRECT=683 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=34 SPILLED_RECORDS=99 TaskCounter_scope_54_INPUT_scope_53 ADDITIONAL_SPILLS_BYTES_READ=903 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=25 LAST_EVENT_RECEIVED=26 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=55 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=100 REDUCE_INPUT_RECORDS=100 SHUFFLE_BYTES=903 SHUFFLE_BYTES_DECOMPRESSED=1604 SHUFFLE_BYTES_DISK_DIRECT=903 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=49 SPILLED_RECORDS=100 TaskCounter_scope_54_OUTPUT_scope_50 OUTPUT_RECORDS=99 org.apache.hadoop.mapreduce.TaskCounter COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_51_OUTPUT_scope_52 COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_52_INPUT_scope_51 COMBINE_INPUT_RECORDS=0 COMBINE_OUTPUT_RECORDS=0 2016-07-19 19:44:15,262 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 47 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:16,275 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 48 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:17,276 [PigTezLauncher-0] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: sandbox.hortonworks.com/192.168.228.128:52539. Already tried 49 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-07-19 19:44:17,592 [PigTezLauncher-0] WARN org.apache.pig.tools.pigstats.tez.TezDAGStats - Failed to get status for vertex: scope-54 2016-07-19 19:44:28,127 [PigTezLauncher-0] WARN org.apache.pig.tools.pigstats.tez.TezDAGStats - Failed to get status for vertex: scope-51 2016-07-19 19:44:35,282 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=SUCCEEDED, progress=TotalTasks: 4 Succeeded: 4 Running: 0 Failed: 0 Killed: 0, diagnostics=, counters=Counters: 171 org.apache.tez.common.counters.DAGCounter NUM_SUCCEEDED_TASKS=4 TOTAL_LAUNCHED_TASKS=4 DATA_LOCAL_TASKS=2 AM_CPU_MILLISECONDS=30700 AM_GC_TIME_MILLIS=12384 File System Counters FILE_BYTES_READ=3397 FILE_BYTES_WRITTEN=2335 FILE_READ_OPS=0 FILE_LARGE_READ_OPS=0 FILE_WRITE_OPS=0 HDFS_BYTES_READ=25911 HDFS_BYTES_WRITTEN=1455 HDFS_READ_OPS=9 HDFS_LARGE_READ_OPS=0 HDFS_WRITE_OPS=2 org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=298 REDUCE_INPUT_RECORDS=298 COMBINE_INPUT_RECORDS=0 SPILLED_RECORDS=596 NUM_SHUFFLED_INPUTS=5 NUM_SKIPPED_INPUTS=0 NUM_FAILED_SHUFFLE_INPUTS=0 MERGED_MAP_OUTPUTS=5 GC_TIME_MILLIS=186160 CPU_MILLISECONDS=60430 PHYSICAL_MEMORY_BYTES=915931136 VIRTUAL_MEMORY_BYTES=3519447040 COMMITTED_HEAP_BYTES=915931136 INPUT_RECORDS_PROCESSED=8100 OUTPUT_RECORDS=755 OUTPUT_BYTES=7458 OUTPUT_BYTES_WITH_OVERHEAD=4265 OUTPUT_BYTES_PHYSICAL=2191 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILLS_BYTES_READ=2191 ADDITIONAL_SPILL_COUNT=0 SHUFFLE_CHUNK_COUNT=3 SHUFFLE_BYTES=2191 SHUFFLE_BYTES_DECOMPRESSED=4265 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_DISK_DIRECT=2191 NUM_MEM_TO_DISK_MERGES=0 NUM_DISK_TO_DISK_MERGES=0 SHUFFLE_PHASE_TIME=405 MERGE_PHASE_TIME=1339 FIRST_EVENT_RECEIVED=159 LAST_EVENT_RECEIVED=160 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_52_INPUT_scope_51 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_52 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Shuffle Errors_scope_54_INPUT_scope_53 BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 TaskCounter_scope_51_INPUT_scope_2 INPUT_RECORDS_PROCESSED=8000 TaskCounter_scope_51_OUTPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=4992 OUTPUT_BYTES_PHYSICAL=605 OUTPUT_BYTES_WITH_OVERHEAD=1377 OUTPUT_RECORDS=457 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_52_INPUT_scope_51 ADDITIONAL_SPILLS_BYTES_READ=605 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=106 LAST_EVENT_RECEIVED=106 MERGED_MAP_OUTPUTS=1 MERGE_PHASE_TIME=1241 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=1 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=605 SHUFFLE_BYTES_DECOMPRESSED=1377 SHUFFLE_BYTES_DISK_DIRECT=605 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=322 SPILLED_RECORDS=99 TaskCounter_scope_52_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1074 OUTPUT_BYTES_PHYSICAL=683 OUTPUT_BYTES_WITH_OVERHEAD=1284 OUTPUT_RECORDS=99 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=99 TaskCounter_scope_53_INPUT_scope_24 INPUT_RECORDS_PROCESSED=100 TaskCounter_scope_53_OUTPUT_scope_54 ADDITIONAL_SPILLS_BYTES_READ=0 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 ADDITIONAL_SPILL_COUNT=0 OUTPUT_BYTES=1392 OUTPUT_BYTES_PHYSICAL=903 OUTPUT_BYTES_WITH_OVERHEAD=1604 OUTPUT_RECORDS=100 SHUFFLE_CHUNK_COUNT=1 SPILLED_RECORDS=100 TaskCounter_scope_54_INPUT_scope_52 ADDITIONAL_SPILLS_BYTES_READ=683 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=28 LAST_EVENT_RECEIVED=28 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=43 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=99 REDUCE_INPUT_RECORDS=99 SHUFFLE_BYTES=683 SHUFFLE_BYTES_DECOMPRESSED=1284 SHUFFLE_BYTES_DISK_DIRECT=683 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=34 SPILLED_RECORDS=99 TaskCounter_scope_54_INPUT_scope_53 ADDITIONAL_SPILLS_BYTES_READ=903 ADDITIONAL_SPILLS_BYTES_WRITTEN=0 COMBINE_INPUT_RECORDS=0 FIRST_EVENT_RECEIVED=25 LAST_EVENT_RECEIVED=26 MERGED_MAP_OUTPUTS=2 MERGE_PHASE_TIME=55 NUM_DISK_TO_DISK_MERGES=0 NUM_FAILED_SHUFFLE_INPUTS=0 NUM_MEM_TO_DISK_MERGES=0 NUM_SHUFFLED_INPUTS=2 NUM_SKIPPED_INPUTS=0 REDUCE_INPUT_GROUPS=100 REDUCE_INPUT_RECORDS=100 SHUFFLE_BYTES=903 SHUFFLE_BYTES_DECOMPRESSED=1604 SHUFFLE_BYTES_DISK_DIRECT=903 SHUFFLE_BYTES_TO_DISK=0 SHUFFLE_BYTES_TO_MEM=0 SHUFFLE_PHASE_TIME=49 SPILLED_RECORDS=100 TaskCounter_scope_54_OUTPUT_scope_50 OUTPUT_RECORDS=99 org.apache.hadoop.mapreduce.TaskCounter COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_51_OUTPUT_scope_52 COMBINE_INPUT_RECORDS=99 COMBINE_OUTPUT_RECORDS=457 org.apache.hadoop.mapreduce.TaskCounter_scope_52_INPUT_scope_51 COMBINE_INPUT_RECORDS=0 COMBINE_OUTPUT_RECORDS=0 2016-07-19 19:44:41,372 [PigTezLauncher-0] WARN org.apache.pig.tools.pigstats.tez.TezDAGStats - Failed to get status for vertex: scope-53 2016-07-19 19:44:43,376 [PigTezLauncher-0] WARN org.apache.pig.tools.pigstats.tez.TezDAGStats - Failed to get status for vertex: scope-52 2016-07-19 19:44:43,946 [main] WARN org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Counters are not available for vertex scope-51. Not computing warning aggregates. 2016-07-19 19:44:44,360 [main] WARN org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Counters are not available for vertex scope-52. Not computing warning aggregates. 2016-07-19 19:44:44,366 [main] WARN org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Counters are not available for vertex scope-53. Not computing warning aggregates. 2016-07-19 19:44:44,366 [main] WARN org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Counters are not available for vertex scope-54. Not computing warning aggregates. 2016-07-19 19:44:44,912 [main] INFO org.apache.pig.tools.pigstats.tez.TezPigScriptStats - Script Statistics: HadoopVersion: 2.7.1.2.4.0.0-169 PigVersion: 0.15.0.2.4.0.0-169 TezVersion: 0.7.0.2.4.0.0-169 UserId: yarn FileName: script.pig StartedAt: 2016-07-19 19:12:33 FinishedAt: 2016-07-19 19:44:44 Features: HASH_JOIN,GROUP_BY,FILTER Success! DAG PigLatin:script.pig-0_scope-0: ApplicationId: job_1468940847411_0006 TotalLaunchedTasks: 4 FileBytesRead: 3397 FileBytesWritten: 2335 HdfsBytesRead: 25911 HdfsBytesWritten: 1455 Input(s): Output(s): 2016-07-19 19:44:48,509 [main] INFO org.apache.pig.Main - Pig script completed in 32 minutes, 54 seconds and 51 milliseconds (1974051 ms) 2016-07-19 19:44:48,510 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Shutting down thread pool 2016-07-19 19:44:48,948 [Thread-34] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager - Tez session is already shutdown org.apache.tez.client.TezClient@167df515 2016-07-19 19:44:49,079 [Thread-1] ERROR org.apache.pig.impl.io.FileLocalizer - java.io.IOException: Filesystem closed