question Re: The Pig Script Hangs While Running in Support Questions

The Pig Script Hangs While Running

myetginbal — Mon, 02 May 2016 14:56:37 GMT

Hi all,

I have created a 2 node cluster (2 Core, 8 GB Ram on each) and started to follow the tutorial on this page. Whenever I try to execute the example pig script (Step 3.4: Execute Pig Script on Tez), script hangs although the status stays in "Running". As can be seen from the dashboard screenshot, all my services are seem to be up & running.

I have also attached the screenshots of simple yarn config screen and info regarding to specific MapReduce job.

Why might be the problem in my case?

Thank you.

Re: The Pig Script Hangs While Running

pvillard — Mon, 02 May 2016 14:59:38 GMT

Hi,

Could you have a look at YARN logs? Just to be sure that there is enough memory tu run containers.

Re: The Pig Script Hangs While Running

myetginbal — Mon, 19 Aug 2019 10:08:03 GMT

Hi Pierre,

You mean the logs in here, right?

I have also attached the YARN Dashboard.

Re: The Pig Script Hangs While Running

pvillard — Mon, 02 May 2016 16:11:03 GMT

Yes. What do you see in stderr for example?

You could also directly access the logs inside your nodes in /var/log/...

Re: The Pig Script Hangs While Running

myetginbal — Mon, 02 May 2016 17:13:52 GMT

Thank you. I have shared the logs (stderr) below:

May 02, 2016 9:58:54 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class
May 02, 2016 9:58:54 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class
May 02, 2016 9:58:54 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class
May 02, 2016 9:58:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM'
May 02, 2016 9:58:54 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton"
May 02, 2016 9:58:55 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"
May 02, 2016 9:58:55 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest"

Re: The Pig Script Hangs While Running

myetginbal — Mon, 02 May 2016 18:12:35 GMT

I have also added the "syslog":

2016-05-02 13:50:06,593 INFO [IPC Server handler 13 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:51:06,684 INFO [IPC Server handler 24 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:52:06,742 INFO [IPC Server handler 4 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:53:06,820 INFO [IPC Server handler 13 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:54:06,883 INFO [IPC Server handler 4 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:55:07,209 INFO [IPC Server handler 14 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:56:07,291 INFO [IPC Server handler 5 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:57:07,405 INFO [IPC Server handler 9 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:58:07,468 INFO [IPC Server handler 14 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:59:07,550 INFO [IPC Server handler 19 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:00:07,611 INFO [IPC Server handler 28 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:01:07,689 INFO [IPC Server handler 16 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:02:07,766 INFO [IPC Server handler 9 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:03:07,863 INFO [IPC Server handler 1 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:04:07,937 INFO [IPC Server handler 14 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:05:08,009 INFO [IPC Server handler 23 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:06:08,079 INFO [IPC Server handler 3 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:07:08,160 INFO [IPC Server handler 8 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:08:08,223 INFO [IPC Server handler 5 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:09:08,282 INFO [IPC Server handler 2 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:10:08,397 INFO [IPC Server handler 20 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:11:08,458 INFO [IPC Server handler 12 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0

Re: The Pig Script Hangs While Running

myetginbal — Wed, 04 May 2016 01:14:38 GMT

Do you have any suggestion on this?

Re: The Pig Script Hangs While Running

pvillard — Wed, 04 May 2016 02:59:33 GMT

@Muhammed Yetginbal Logs look OK. It is difficult to help. As suggested, I'd recommend you to have a look into /var/log/... directories/files to look for error messages at the time your Pig job was running.

Re: The Pig Script Hangs While Running

myetginbal — Wed, 04 May 2016 14:17:04 GMT

Unfortunately I did not see something specific in the logs in /var/log/.. but I realized that whenever I remove the last line of the sample pig script which is:

store final_data into 'riskfactor' using org.apache.hive.hcatalog.pig.HCatStorer();

everything worked fine and the script execution is completed. Does that provide you some clue on this?

Re: The Pig Script Hangs While Running

pvillard — Wed, 04 May 2016 14:27:05 GMT

@Muhammed Yetginbal

Considering your last comment and information provided, I had a look at:

https://cwiki.apache.org/confluence/display/Hive/HCatalog+LoadStore#HCatalogLoadStore-HCatStorer

Can you confirm that the pre-requisites are met in your case? The table 'riskFactor' exists (with the correct schema)?

Besides is Hive up and running?

Are you running your script where Hive is installed? are you in a clustered env?

Re: The Pig Script Hangs While Running

myetginbal — Wed, 04 May 2016 14:48:53 GMT

Thank you for your quick feedback. I can confirm that the table 'riskfactor' exists with the correct scheme (I used the HiveQL statement in the tutorial). Hive is up & running. I am in a 2 node cluster environment.

Only not sure about the question whether I am running my script where Hive installed or not. I can confirm that services like Hive Metastore, HiveServer2 are installed in the second node of my cluster and the map reduce job that I see in the ResourceManager UI says that Node is the second one (also attached the screenshot).

Re: The Pig Script Hangs While Running

pvillard — Wed, 04 May 2016 15:48:00 GMT

You are running the pig script from the pig view in ambari as described in the tutorial? If yes, do you have a chance to get the logs from the pig view?

Re: The Pig Script Hangs While Running

myetginbal — Wed, 04 May 2016 18:26:29 GMT

Yes, I do run the pig script from the pig view in Ambari, however since it constantly stays on the "Running" status no further log is displayed on the "Logs" panel in the view.

Btw, in order to observe the scenario you have mentioned, I changed the table name in the store operation to a non-existing table and it directly threw an error saying that the related table is not found. So we can assume that the script sees the "riskfactor" table but somehow stuck in performing the store operation.

Re: The Pig Script Hangs While Running

myetginbal — Wed, 04 May 2016 18:30:50 GMT

I am also adding the screenshot regarding to the scenario where I do not use the "Store" operation.

Re: The Pig Script Hangs While Running

pvillard — Wed, 04 May 2016 18:38:35 GMT

@Muhammed Yetginbal One option would be to test launching the Pig script from the console (https://wiki.apache.org/pig/RunPig). Using the client (grunt) you could try to execute the same commands. Maybe it will give you more details about what is happening when "running".

Re: The Pig Script Hangs While Running

myetginbal — Wed, 04 May 2016 19:51:53 GMT

Finally! The script has worked via grunt, I have added the whole grunt execution logs in the attachment. First of all thank you Pierre, happy to see that the script has worked 🙂 But in this case, what might be the problem with the Pig View?

Re: The Pig Script Hangs While Running

pvillard — Thu, 05 May 2016 00:04:56 GMT

Honestly... I don't know 🙂 I'd suggest digging into the logs (ambari, mapreduce, yarn, hive, etc) to find an explanation. But maybe someone else on HCC will have an idea 🙂

Re: The Pig Script Hangs While Running

myetginbal — Thu, 05 May 2016 01:07:39 GMT

Thank you Pierre 🙂 You've helped me a lot, appreciated 🙂 I'll accept your answer since it was the only option for me to be able to execute the script.

Re: The Pig Script Hangs While Running

kabadou_rawia — Wed, 06 Dec 2017 21:52:24 GMT

@Muhammed Yetginbal

Hello, I'm facing the same issue by following the same tutorial mentioned in:

https://hortonworks.com/tutorial/hadoop-tutorial-getting-started-with-hdp/section/4/.

Did you find a solution for this issue? I'll be grateful if you could help me.