ls: cannot access /hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/lib/slf4j-api-*.jar: No such file or directory
ls: cannot access /hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/hcatalog/lib/*hbase-storage-handler-*.jar: No such file or directory
WARNING: Use "yarn jar" to launch YARN applications.
17/03/19 20:43:33 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
17/03/19 20:43:33 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
17/03/19 20:43:33 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
2017-03-19 20:43:33,949 [main] INFO org.apache.pig.Main - Apache Pig version 0.16.0.2.5.3.0-37 (rexported) compiled Nov 30 2016, 03:39:58
2017-03-19 20:43:33,950 [main] INFO org.apache.pig.Main - Logging error messages to: /hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/pig_1489936413948.log
2017-03-19 20:43:34,644 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/yarn/.pigbootup not found
2017-03-19 20:43:34,793 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://pc-1.thenet.edu:8020
2017-03-19 20:43:35,635 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-script.pig-d7322ec7-3161-486b-8b7d-4e02dd61d5db
2017-03-19 20:43:36,106 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:43:36,225 [main] INFO org.apache.pig.backend.hadoop.PigATSClient - Created ATS Hook
2017-03-19 20:43:37,264 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2017-03-19 20:43:37,308 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083
2017-03-19 20:43:37,343 [main] INFO hive.metastore - Connected to metastore.
2017-03-19 20:43:38,260 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2017-03-19 20:43:38,268 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083
2017-03-19 20:43:38,269 [main] INFO hive.metastore - Connected to metastore.
2017-03-19 20:43:38,553 [main] WARN org.apache.pig.newplan.BaseOperatorPlan - Encountered Warning IMPLICIT_CAST_TO_FLOAT 1 time(s).
2017-03-19 20:43:38,703 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2017-03-19 20:43:38,745 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083
2017-03-19 20:43:38,754 [main] INFO hive.metastore - Connected to metastore.
2017-03-19 20:43:39,197 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2017-03-19 20:43:39,226 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083
2017-03-19 20:43:39,229 [main] INFO hive.metastore - Connected to metastore.
2017-03-19 20:43:39,269 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: HASH_JOIN,GROUP_BY,FILTER
2017-03-19 20:43:39,296 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
2017-03-19 20:43:39,486 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
2017-03-19 20:43:39,518 [main] INFO org.apache.pig.impl.util.SpillableMemoryManager - Selected heap (PS Old Gen) of size 716177408 to monitor. collectionUsageThreshold = 501324192, usageThreshold = 501324192
2017-03-19 20:43:39,564 [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for a: $0, $3, $4, $5, $6, $7, $8, $9
2017-03-19 20:43:39,629 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2017-03-19 20:43:39,694 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil - Choosing to move algebraic foreach to combiner
2017-03-19 20:43:39,716 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage->POForEach to POPackage(JoinPackager)
2017-03-19 20:43:39,723 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 2
2017-03-19 20:43:39,723 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 2
2017-03-19 20:43:39,950 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:43:39,983 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:43:40,121 [main] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:43:40,212 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job
2017-03-19 20:43:40,230 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2017-03-19 20:43:40,356 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2017-03-19 20:43:40,375 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083
2017-03-19 20:43:40,377 [main] INFO hive.metastore - Connected to metastore.
2017-03-19 20:43:40,574 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers.
2017-03-19 20:43:40,576 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2017-03-19 20:43:40,709 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=0
2017-03-19 20:43:40,709 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2017-03-19 20:43:40,709 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - This job cannot be converted run in-process
2017-03-19 20:43:41,235 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/lib/hive-metastore-1.2.1000.2.5.3.0-37.jar to DistributedCache through /tmp/temp530065216/tmp1080125993/hive-metastore-1.2.1000.2.5.3.0-37.jar
2017-03-19 20:43:41,289 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/lib/libthrift-0.9.3.jar to DistributedCache through /tmp/temp530065216/tmp-2135387749/libthrift-0.9.3.jar
2017-03-19 20:43:41,480 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/lib/hive-exec-1.2.1000.2.5.3.0-37.jar to DistributedCache through /tmp/temp530065216/tmp1103910850/hive-exec-1.2.1000.2.5.3.0-37.jar
2017-03-19 20:43:41,556 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/lib/libfb303-0.9.3.jar to DistributedCache through /tmp/temp530065216/tmp500406558/libfb303-0.9.3.jar
2017-03-19 20:43:41,634 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/lib/jdo-api-3.0.1.jar to DistributedCache through /tmp/temp530065216/tmp385514294/jdo-api-3.0.1.jar
2017-03-19 20:43:41,689 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/lib/hive-hbase-handler-1.2.1000.2.5.3.0-37.jar to DistributedCache through /tmp/temp530065216/tmp-1334509076/hive-hbase-handler-1.2.1000.2.5.3.0-37.jar
2017-03-19 20:43:41,744 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/hcatalog/share/hcatalog/hive-hcatalog-core-1.2.1000.2.5.3.0-37.jar to DistributedCache through /tmp/temp530065216/tmp916321254/hive-hcatalog-core-1.2.1000.2.5.3.0-37.jar
2017-03-19 20:43:41,844 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/hcatalog/share/hcatalog/hive-hcatalog-pig-adapter-1.2.1000.2.5.3.0-37.jar to DistributedCache through /tmp/temp530065216/tmp2062004886/hive-hcatalog-pig-adapter-1.2.1000.2.5.3.0-37.jar
2017-03-19 20:43:41,921 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/filecache/11/pig.tar.gz/pig/pig-0.16.0.2.5.3.0-37-core-h2.jar to DistributedCache through /tmp/temp530065216/tmp-1676844329/pig-0.16.0.2.5.3.0-37-core-h2.jar
2017-03-19 20:43:42,011 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/filecache/11/pig.tar.gz/pig/lib/automaton-1.11-8.jar to DistributedCache through /tmp/temp530065216/tmp1903502153/automaton-1.11-8.jar
2017-03-19 20:43:42,065 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/filecache/11/pig.tar.gz/pig/lib/antlr-runtime-3.4.jar to DistributedCache through /tmp/temp530065216/tmp3306926/antlr-runtime-3.4.jar
2017-03-19 20:43:42,074 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2017-03-19 20:43:42,082 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2017-03-19 20:43:42,082 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2017-03-19 20:43:42,082 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2017-03-19 20:43:42,163 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2017-03-19 20:43:42,270 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:43:42,270 [JobControl] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:43:42,271 [JobControl] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:43:42,512 [JobControl] WARN org.apache.hadoop.mapreduce.JobResourceUploader - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2017-03-19 20:43:42,722 [JobControl] INFO org.apache.hadoop.hive.ql.log.PerfLogger -
2017-03-19 20:43:42,733 [JobControl] INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat - ORC pushdown predicate: null
2017-03-19 20:43:42,785 [JobControl] INFO org.apache.hadoop.hive.ql.io.orc.OrcUtils - Using schema evolution configuration variables schema.evolution.columns [truckid, driverid, event, latitude, longitude, city, state, velocity, event_ind, idling_ind] / schema.evolution.columns.types [string, string, string, string, string, string, string, string, string, string] (isAcid false)
2017-03-19 20:43:43,268 [JobControl] INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat - FooterCacheHitRatio: 0/2
2017-03-19 20:43:43,269 [JobControl] INFO org.apache.hadoop.hive.ql.log.PerfLogger -
2017-03-19 20:43:43,272 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2017-03-19 20:43:43,440 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2017-03-19 20:43:43,589 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1489933015061_0004
2017-03-19 20:43:43,589 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - Kind: mapreduce.job, Service: job_1489933015061_0003, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@5ed20019)
2017-03-19 20:43:43,812 [JobControl] INFO org.apache.hadoop.mapred.YARNRunner - Job jar is not present. Not adding any jar to the list of resources.
2017-03-19 20:43:44,164 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1489933015061_0004
2017-03-19 20:43:44,210 [JobControl] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://pc-1.thenet.edu:8088/proxy/application_1489933015061_0004/
2017-03-19 20:43:44,214 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1489933015061_0004
2017-03-19 20:43:44,214 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases a,b,c,d,e
2017-03-19 20:43:44,214 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: a[1,4],b[2,4],c[3,4],e[5,4],d[4,4] C: e[5,4],d[4,4] R: e[5,4]
2017-03-19 20:43:44,226 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2017-03-19 20:43:44,226 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1489933015061_0004]
2017-03-19 20:43:58,299 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 25% complete
2017-03-19 20:43:58,299 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1489933015061_0004]
2017-03-19 20:44:03,304 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2017-03-19 20:44:03,304 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1489933015061_0004]
2017-03-19 20:44:04,402 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:44:04,403 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:44:04,403 [main] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:44:04,410 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-03-19 20:44:04,713 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:44:04,713 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:44:04,714 [main] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:44:04,718 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-03-19 20:44:04,842 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:44:04,842 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:44:04,842 [main] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:44:04,846 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-03-19 20:44:04,922 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job
2017-03-19 20:44:04,923 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2017-03-19 20:44:04,956 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2017-03-19 20:44:04,961 [main] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083
2017-03-19 20:44:04,962 [main] INFO hive.metastore - Connected to metastore.
2017-03-19 20:44:04,990 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers.
2017-03-19 20:44:04,991 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2017-03-19 20:44:05,025 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=1173
2017-03-19 20:44:05,025 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2017-03-19 20:44:05,026 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - This job cannot be converted run in-process
2017-03-19 20:44:05,122 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/lib/hive-metastore-1.2.1000.2.5.3.0-37.jar to DistributedCache through /tmp/temp530065216/tmp1032592565/hive-metastore-1.2.1000.2.5.3.0-37.jar
2017-03-19 20:44:05,166 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/lib/libthrift-0.9.3.jar to DistributedCache through /tmp/temp530065216/tmp-1884636127/libthrift-0.9.3.jar
2017-03-19 20:44:05,275 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/lib/hive-exec-1.2.1000.2.5.3.0-37.jar to DistributedCache through /tmp/temp530065216/tmp1405477484/hive-exec-1.2.1000.2.5.3.0-37.jar
2017-03-19 20:44:05,322 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/lib/libfb303-0.9.3.jar to DistributedCache through /tmp/temp530065216/tmp-221134062/libfb303-0.9.3.jar
2017-03-19 20:44:05,366 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/lib/jdo-api-3.0.1.jar to DistributedCache through /tmp/temp530065216/tmp1295558880/jdo-api-3.0.1.jar
2017-03-19 20:44:05,410 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/lib/hive-hbase-handler-1.2.1000.2.5.3.0-37.jar to DistributedCache through /tmp/temp530065216/tmp1536231592/hive-hbase-handler-1.2.1000.2.5.3.0-37.jar
2017-03-19 20:44:05,455 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/hcatalog/share/hcatalog/hive-hcatalog-core-1.2.1000.2.5.3.0-37.jar to DistributedCache through /tmp/temp530065216/tmp1498516008/hive-hcatalog-core-1.2.1000.2.5.3.0-37.jar
2017-03-19 20:44:05,499 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/usercache/subrata/appcache/application_1489933015061_0003/container_e25_1489933015061_0003_01_000002/hive.tar.gz/hive/hcatalog/share/hcatalog/hive-hcatalog-pig-adapter-1.2.1000.2.5.3.0-37.jar to DistributedCache through /tmp/temp530065216/tmp1664110372/hive-hcatalog-pig-adapter-1.2.1000.2.5.3.0-37.jar
2017-03-19 20:44:05,555 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/filecache/11/pig.tar.gz/pig/pig-0.16.0.2.5.3.0-37-core-h2.jar to DistributedCache through /tmp/temp530065216/tmp1578488361/pig-0.16.0.2.5.3.0-37-core-h2.jar
2017-03-19 20:44:05,599 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/filecache/11/pig.tar.gz/pig/lib/automaton-1.11-8.jar to DistributedCache through /tmp/temp530065216/tmp845131845/automaton-1.11-8.jar
2017-03-19 20:44:05,644 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Added jar file:/hadoop/yarn/local/filecache/11/pig.tar.gz/pig/lib/antlr-runtime-3.4.jar to DistributedCache through /tmp/temp530065216/tmp-787430943/antlr-runtime-3.4.jar
2017-03-19 20:44:05,645 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2017-03-19 20:44:05,646 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2017-03-19 20:44:05,646 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2017-03-19 20:44:05,646 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2017-03-19 20:44:05,691 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2017-03-19 20:44:05,763 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:44:05,763 [JobControl] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:44:05,764 [JobControl] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:44:05,765 [JobControl] WARN org.apache.hadoop.hdfs.ClientContext - Existing client context 'default' does not match requested configuration. Existing: shortCircuitStreamsCacheSize = 4096, shortCircuitStreamsCacheExpiryMs = 300000, shortCircuitMmapCacheSize = 256, shortCircuitMmapCacheExpiryMs = 3600000, shortCircuitMmapCacheRetryTimeout = 300000, shortCircuitCacheStaleThresholdMs = 1800000, socketCacheCapacity = 16, socketCacheExpiry = 3000, shortCircuitLocalReads = true, useLegacyBlockReaderLocal = false, domainSocketDataTraffic = false, shortCircuitSharedMemoryWatcherInterruptCheckMs = 60000, keyProviderCacheExpiryMs = 864000000, Requested: shortCircuitStreamsCacheSize = 4096, shortCircuitStreamsCacheExpiryMs = 300000, shortCircuitMmapCacheSize = 256, shortCircuitMmapCacheExpiryMs = 3600000, shortCircuitMmapCacheRetryTimeout = 300000, shortCircuitCacheStaleThresholdMs = 1800000, socketCacheCapacity = 16, socketCacheExpiry = 3000, shortCircuitLocalReads = false, useLegacyBlockReaderLocal = false, domainSocketDataTraffic = false, shortCircuitSharedMemoryWatcherInterruptCheckMs = 60000, keyProviderCacheExpiryMs = 864000000
2017-03-19 20:44:05,806 [JobControl] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.metastore.local does not exist
2017-03-19 20:44:05,806 [JobControl] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.io.file.read.all.columns does not exist
2017-03-19 20:44:05,807 [JobControl] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.io.file.readcolumn.ids does not exist
2017-03-19 20:44:05,813 [JobControl] INFO hive.metastore - Trying to connect to metastore with URI thrift://pc-1.thenet.edu:9083
2017-03-19 20:44:05,814 [JobControl] INFO hive.metastore - Connected to metastore.
2017-03-19 20:44:05,855 [JobControl] WARN org.apache.hadoop.mapreduce.JobResourceUploader - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2017-03-19 20:44:05,893 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2017-03-19 20:44:05,893 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2017-03-19 20:44:05,894 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2017-03-19 20:44:05,898 [JobControl] INFO org.apache.hadoop.hive.ql.log.PerfLogger -
2017-03-19 20:44:05,898 [JobControl] INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat - ORC pushdown predicate: null
2017-03-19 20:44:05,899 [JobControl] INFO org.apache.hadoop.hive.ql.io.orc.OrcUtils - Using schema evolution configuration variables schema.evolution.columns [driverid, totmiles] / schema.evolution.columns.types [string, bigint] (isAcid false)
2017-03-19 20:44:05,906 [JobControl] INFO org.apache.hadoop.hive.ql.io.orc.OrcInputFormat - FooterCacheHitRatio: 0/2
2017-03-19 20:44:05,906 [JobControl] INFO org.apache.hadoop.hive.ql.log.PerfLogger -
2017-03-19 20:44:05,906 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2017-03-19 20:44:05,999 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - number of splits:2
2017-03-19 20:44:06,066 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1489933015061_0005
2017-03-19 20:44:06,066 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - Kind: mapreduce.job, Service: job_1489933015061_0003, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@24479399)
2017-03-19 20:44:06,069 [JobControl] INFO org.apache.hadoop.mapred.YARNRunner - Job jar is not present. Not adding any jar to the list of resources.
2017-03-19 20:44:06,336 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1489933015061_0005
2017-03-19 20:44:06,340 [JobControl] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://pc-1.thenet.edu:8088/proxy/application_1489933015061_0005/
2017-03-19 20:44:06,341 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1489933015061_0005
2017-03-19 20:44:06,341 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases final_data,g,h
2017-03-19 20:44:06,341 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: h[7,4],g[6,4],h[7,4] C: R: final_data[8,13]
2017-03-19 20:44:23,413 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 75% complete
2017-03-19 20:44:23,413 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1489933015061_0005]
2017-03-19 20:44:28,419 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1489933015061_0005]
2017-03-19 20:44:31,487 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:44:31,487 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:44:31,488 [main] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:44:31,492 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-03-19 20:44:31,640 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:44:31,640 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:44:31,640 [main] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:44:31,644 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-03-19 20:44:31,736 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:44:31,736 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:44:31,736 [main] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:44:31,740 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-03-19 20:44:31,768 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2017-03-19 20:44:31,802 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.7.3.2.5.3.0-37 0.16.0.2.5.3.0-37 yarn 2017-03-19 20:43:40 2017-03-19 20:44:31 HASH_JOIN,GROUP_BY,FILTER
Success!
Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTime AvgMapTime MedianMapTime MaxReduceTime MinReduceTime AvgReduceTime MedianReducetime Alias Feature Outputs
job_1489933015061_0004 1 1 3 3 3 3 2 2 2 2 a,b,c,d,e GROUP_BY,COMBINER
job_1489933015061_0005 2 1 4 4 4 4 3 3 3 3 final_data,g,h HASH_JOIN riskfactor,
Input(s):
Successfully read 8000 records (26675 bytes) from: "geolocation"
Successfully read 100 records from: "driver_mileage"
Output(s):
Successfully stored 99 records (1462 bytes) in: "riskfactor"
Counters:
Total records written : 99
Total bytes written : 1462
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1489933015061_0004 -> job_1489933015061_0005,
job_1489933015061_0005
2017-03-19 20:44:31,858 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:44:31,858 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:44:31,858 [main] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:44:31,862 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-03-19 20:44:31,947 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:44:31,947 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:44:31,947 [main] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:44:31,951 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-03-19 20:44:32,049 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:44:32,049 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:44:32,050 [main] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:44:32,053 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-03-19 20:44:32,153 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:44:32,153 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:44:32,153 [main] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:44:32,157 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-03-19 20:44:32,285 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:44:32,285 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:44:32,285 [main] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:44:32,294 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-03-19 20:44:32,426 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://pc-1.thenet.edu:8188/ws/v1/timeline/
2017-03-19 20:44:32,426 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at pc-1.thenet.edu/127.0.1.1:8050
2017-03-19 20:44:32,427 [main] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at pc-1.thenet.edu/127.0.1.1:10200
2017-03-19 20:44:32,432 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-03-19 20:44:32,471 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2017-03-19 20:44:32,486 [main] INFO org.apache.pig.Main - Pig script completed in 58 seconds and 757 milliseconds (58757 ms)