The Pig Script Hangs While Running

[#1]

Hi all,

I have created a 2 node cluster (2 Core, 8 GB Ram on each) and started to follow the tutorial on this page. Whenever I try to execute the example pig script (Step 3.4: Execute Pig Script on Tez), script hangs although the status stays in "Running". As can be seen from the dashboard screenshot, all my services are seem to be up & running.

I have also attached the screenshots of simple yarn config screen and info regarding to specific MapReduce job.

Why might be the problem in my case?

Thank you.

[#2]

Hi,

Could you have a look at YARN logs? Just to be sure that there is enough memory tu run containers.

[#3]

Hi Pierre,

You mean the logs in here, right?

I have also attached the YARN Dashboard.

[#4]

Yes. What do you see in stderr for example?

You could also directly access the logs inside your nodes in /var/log/...

[#5]

Thank you. I have shared the logs (stderr) below:

May 02, 2016 9:58:54 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class
May 02, 2016 9:58:54 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class
May 02, 2016 9:58:54 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class
May 02, 2016 9:58:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM'
May 02, 2016 9:58:54 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton"
May 02, 2016 9:58:55 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"
May 02, 2016 9:58:55 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest"

[#6]

I have also added the "syslog":

2016-05-02 13:50:06,593 INFO [IPC Server handler 13 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:51:06,684 INFO [IPC Server handler 24 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:52:06,742 INFO [IPC Server handler 4 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:53:06,820 INFO [IPC Server handler 13 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:54:06,883 INFO [IPC Server handler 4 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:55:07,209 INFO [IPC Server handler 14 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:56:07,291 INFO [IPC Server handler 5 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:57:07,405 INFO [IPC Server handler 9 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:58:07,468 INFO [IPC Server handler 14 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 13:59:07,550 INFO [IPC Server handler 19 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:00:07,611 INFO [IPC Server handler 28 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:01:07,689 INFO [IPC Server handler 16 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:02:07,766 INFO [IPC Server handler 9 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:03:07,863 INFO [IPC Server handler 1 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:04:07,937 INFO [IPC Server handler 14 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:05:08,009 INFO [IPC Server handler 23 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:06:08,079 INFO [IPC Server handler 3 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:07:08,160 INFO [IPC Server handler 8 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:08:08,223 INFO [IPC Server handler 5 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:09:08,282 INFO [IPC Server handler 2 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:10:08,397 INFO [IPC Server handler 20 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0
2016-05-02 14:11:08,458 INFO [IPC Server handler 12 on 51273] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1462171966114_0001_m_000000_0 is : 0.0

[#7]

Do you have any suggestion on this?

[#8]

@Muhammed Yetginbal Logs look OK. It is difficult to help. As suggested, I'd recommend you to have a look into /var/log/... directories/files to look for error messages at the time your Pig job was running.

[#9]

Unfortunately I did not see something specific in the logs in /var/log/.. but I realized that whenever I remove the last line of the sample pig script which is:

store final_data into 'riskfactor' using org.apache.hive.hcatalog.pig.HCatStorer();

everything worked fine and the script execution is completed. Does that provide you some clue on this?

[#10 · Accepted solution]

@Muhammed Yetginbal

Considering your last comment and information provided, I had a look at:

https://cwiki.apache.org/confluence/display/Hive/HCatalog+LoadStore#HCatalogLoadStore-HCatStorer

Can you confirm that the pre-requisites are met in your case? The table 'riskFactor' exists (with the correct schema)?

Besides is Hive up and running?

Are you running your script where Hive is installed? are you in a clustered env?