Thanks!! Looks like you haven't seen my previous email. I was able to resolve the issue as it was due to lack of RAM on my system after closing some few unwanted applications. The Heartbeat issue was over and the pig script worked in hue. Although, the entire process as I have mentioned is extremely slow again due to lack of RAM 4GB : Hopefully, In the pursuit of happiness,  in the coming future. I will also climb the expensive 16GB ladder 🙂 But, I once again appreciate your effort! Cheers,vibhor ... View more

Hi, Physical RAM on my system is max 4GB and I am not using cloudera Manager. Interesting thing is that if I try to execute the script from grunt shell it works well and if I try to execute same via pig editor in hue then it freezes with heart beat. Please let me know if I need to change or specify some additional properties in yarn-site.xml. Presently I am running the system with all the default properties. Thanks,vibhor  ... View more

Hi,   I am also facing similar problem in CDH 5.3 in psuedo distributed mode. Please let me know if you were able to find any solution?   Apache Pig version 0.12.0-cdh5.3.0 (rexported) compiled Dec 16 2014, 19:05:55 Run pig script using PigRunner.run() for Pig version 0.8+ 2015-05-23 09:28:23,905 [main] INFO org.apache.pig.Main - Apache Pig version 0.12.0-cdh5.3.0 (rexported) compiled Dec 16 2014, 19:05:55 2015-05-23 09:28:23,909 [main] INFO org.apache.pig.Main - Logging error messages to: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appcache/application_1432393487863_0003/container_1432393487863_0003_01_000003/pig-job_1432393487863_0003.log 2015-05-23 09:28:24,043 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /var/lib/hadoop-yarn/.pigbootup not found 2015-05-23 09:28:24,199 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2015-05-23 09:28:24,200 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS 2015-05-23 09:28:24,200 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://quickstart.cloudera:8020 2015-05-23 09:28:24,207 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: localhost:8032 2015-05-23 09:28:25,668 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN 2015-05-23 09:28:25,877 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, DuplicateForEachColumnRewrite, GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]} 2015-05-23 09:28:26,198 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false 2015-05-23 09:28:26,255 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1 2015-05-23 09:28:26,255 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1 2015-05-23 09:28:26,623 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at localhost/127.0.0.1:8032 2015-05-23 09:28:26,972 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job 2015-05-23 09:28:27,071 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent 2015-05-23 09:28:27,071 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2015-05-23 09:28:27,071 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress 2015-05-23 09:28:27,079 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job2635559100793052600.jar 2015-05-23 09:28:34,692 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job2635559100793052600.jar created 2015-05-23 09:28:34,692 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jar is deprecated. Instead, use mapreduce.job.jar 2015-05-23 09:28:34,737 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2015-05-23 09:28:34,826 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code. 2015-05-23 09:28:34,874 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cache 2015-05-23 09:28:34,920 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize [] 2015-05-23 09:28:34,997 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2015-05-23 09:28:35,086 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.http.address is deprecated. Instead, use mapreduce.jobtracker.http.address 2015-05-23 09:28:35,086 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2015-05-23 09:28:35,114 [JobControl] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at localhost/127.0.0.1:8032 2015-05-23 09:28:35,359 [JobControl] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS 2015-05-23 09:28:44,197 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 2015-05-23 09:28:44,211 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1 2015-05-23 09:28:45,222 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1 2015-05-23 09:28:47,256 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1 Heart beat 2015-05-23 09:28:52,256 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1432393487863_0004 2015-05-23 09:28:52,256 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - Kind: mapreduce.job, Service: job_1432393487863_0003, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@3a335d31) 2015-05-23 09:28:52,674 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - Kind: RM_DELEGATION_TOKEN, Service: 127.0.0.1:8032, Ident: (owner=cloudera, renewer=oozie mr token, realUser=oozie, issueDate=1432397362327, maxDate=1433002162327, sequenceNumber=10, masterKeyId=2) 2015-05-23 09:29:19,549 [JobControl] WARN org.apache.hadoop.mapreduce.v2.util.MRApps - cache file (mapreduce.job.cache.files) hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20141218070949/pig/json-simple-1.1.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://quickstart.cloudera:8020/user/oozie/share/lib/lib_20141218070949/oozie/json-simple-1.1.jar This will be an error in Hadoop 2.0 Heart beat 2015-05-23 09:29:27,684 [JobControl] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1432393487863_0004 2015-05-23 09:29:29,035 [JobControl] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://quickstart.cloudera:8088/proxy/application_1432393487863_0004/ 2015-05-23 09:29:29,036 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1432393487863_0004 2015-05-23 09:29:29,036 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases mydata 2015-05-23 09:29:29,036 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: mydata[1,9],mydata[-1,-1] C: R: 2015-05-23 09:29:29,036 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://localhost:50030/jobdetails.jsp?jobid=job_1432393487863_0004 2015-05-23 09:29:31,194 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat ... View more