The Pig Script Hangs While Running

[#1]

Hi all,

I have created a 2 node cluster (2 Core, 8 GB Ram on each) and started to follow the tutorial on this page. Whenever I try to execute the example pig script (Step 3.4: Execute Pig Script on Tez), script hangs although the status stays in "Running". As can be seen from the dashboard screenshot, all my services are seem to be up & running.

I have also attached the screenshots of simple yarn config screen and info regarding to specific MapReduce job.

Why might be the problem in my case?

Thank you.

[#2]

Hi,

Could you have a look at YARN logs? Just to be sure that there is enough memory tu run containers.

[#3]

Hi Pierre,

You mean the logs in here, right?

I have also attached the YARN Dashboard.

[#4]

Yes. What do you see in stderr for example?

You could also directly access the logs inside your nodes in /var/log/...

[#5]

Thank you. I have shared the logs (stderr) below:

May 02, 2016 9:58:54 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class
May 02, 2016 9:58:54 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class
May 02, 2016 9:58:54 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class
May 02, 2016 9:58:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM'
May 02, 2016 9:58:54 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton"
May 02, 2016 9:58:55 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"
May 02, 2016 9:58:55 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest"

[#6]

I have also added the "syslog":

2016-05-02 13:50:06,593 INFO [IPC Server handler 13 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:51:06,684 INFO [IPC Server handler 24 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:52:06,742 INFO [IPC Server handler 4 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:53:06,820 INFO [IPC Server handler 13 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:54:06,883 INFO [IPC Server handler 4 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:55:07,209 INFO [IPC Server handler 14 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:56:07,291 INFO [IPC Server handler 5 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:57:07,405 INFO [IPC Server handler 9 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:58:07,468 INFO [IPC Server handler 14 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:59:07,550 INFO [IPC Server handler 19 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:00:07,611 INFO [IPC Server handler 28 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:01:07,689 INFO [IPC Server handler 16 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:02:07,766 INFO [IPC Server handler 9 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:03:07,863 INFO [IPC Server handler 1 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:04:07,937 INFO [IPC Server handler 14 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:05:08,009 INFO [IPC Server handler 23 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:06:08,079 INFO [IPC Server handler 3 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:07:08,160 INFO [IPC Server handler 8 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:08:08,223 INFO [IPC Server handler 5 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:09:08,282 INFO [IPC Server handler 2 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:10:08,397 INFO [IPC Server handler 20 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:11:08,458 INFO [IPC Server handler 12 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0

[#7]

Do you have any suggestion on this?

[#8]

@Muhammed Yetginbal Logs look OK. It is difficult to help. As suggested, I'd recommend you to have a look into /var/log/... directories/files to look for error messages at the time your Pig job was running.

[#9]

Unfortunately I did not see something specific in the logs in /var/log/.. but I realized that whenever I remove the last line of the sample pig script which is:

store final_data into 'riskfactor' using org.apache.hive.hcatalog.pig.HCatStorer();

everything worked fine and the script execution is completed. Does that provide you some clue on this?