question Re: INSERT COMMAND HANGING in Archives of Support Questions (Read Only)

INSERT COMMAND HANGING

aliyesami — Thu, 12 May 2016 00:42:56 GMT

Iam using HDC2.4 with ambari 2.2.1 , If I insert a record into a table it just hangs , how can I debug ?

"0: jdbc:hive2://MasterNode:10000> INSERT INTO customers (id, firstname, lastname) VALUES (1, 'John', 'Smith'); INFO : Tez session hasn't been created yet. Opening session

Re: INSERT COMMAND HANGING

aliyesami — Thu, 12 May 2016 01:39:00 GMT

btw I have started the resourcemanager on the masternode and the nodemanager on each slavenodes.

Re: INSERT COMMAND HANGING

aliyesami — Thu, 12 May 2016 02:40:33 GMT

also HIVE command is hanging ...

[hdfs@ip-172-31-8-77 ~]$ hive WARNING: Use "yarn jar" to launch YARN applications.

Logging initialized using configuration in file:/etc/hive/2.4.2.0-258/0/hive-log4j.properties

Re: INSERT COMMAND HANGING

nismaily — Thu, 12 May 2016 05:30:22 GMT

Can you attach your logs for hive ?

Also change the log level to debug ($HIVE_HOME/bin/hive --hiveconf hive.root.logger=DEBUG,DRFA)

Re: INSERT COMMAND HANGING

grajagopal — Thu, 12 May 2016 06:58:42 GMT

Please provide the hiveserver2.log. Also check the RM to see the job has launched and it is in an accepted state. If yes, check what is it waiting on.

Re: INSERT COMMAND HANGING

aliyesami — Thu, 12 May 2016 08:01:55 GMT

sorry iam just learning hadoop so please tell me where I can find this hiveserver2.log file ?

also whats RM and how can I check if the job has launced, I use hortonworks and I don't see anywhere on the dashboard such info.

Re: INSERT COMMAND HANGING

aliyesami — Thu, 12 May 2016 08:04:34 GMT

don't I have to kill the hive session first and then run it in background as :

nohup

($HIVE_HOME/bin/hive --hiveconf hive.root.logger=DEBUG,DRFA

how can I find out what is $HIVE_HOME ?

Re: INSERT COMMAND HANGING

aliyesami — Thu, 12 May 2016 11:29:25 GMT

ok I found two hiveserver log files , one as hiveserver2.log and one as hive-server2.log , I will upload both

Re: INSERT COMMAND HANGING

aliyesami — Thu, 12 May 2016 11:35:16 GMT

hiveserver-logs.zip

hive server logs

Re: INSERT COMMAND HANGING

aliyesami — Thu, 12 May 2016 20:39:59 GMT

also found the RM UI but if I try to start it from Ambari as YARN-QUICKLINKS-Resourcemanager UI , I get the following error:

Network Error (tcp_error)

A communication error occurred: "Operation timed out"

The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.

Re: INSERT COMMAND HANGING

aliyesami — Thu, 12 May 2016 20:58:06 GMT

ok I could grab the RM log from the server and its not able to start the job , due to error shown below, how can I fix?

2016-05-12 09:56:51,114 INFO resourcemanager.ClientRMService (ClientRMService.java:getNewApplicationId(290)) - Allocated new applicationId: 11 2016-05-12 09:56:51,225 INFO resourcemanager.ClientRMService (ClientRMService.java:submitApplication(585)) - Application with id 11 submitted by user hadoop 2016-05-12 09:56:51,226 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(170)) - USER=hadoop IP=172.31.8.77 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1463059172548_0011 CALLERCONTEXT=HIVE_QUERY_ID:hive_20160512095650_21a047fd-e7c6-427e-9b8d-5242621e0b60 2016-05-12 09:56:51,226 INFO rmapp.RMAppImpl (RMAppImpl.java:transition(1058)) - Storing application with id application_1463059172548_0011 2016-05-12 09:56:51,226 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(767)) - application_1463059172548_0011 State change from NEW to NEW_SAVING 2016-05-12 09:56:51,227 INFO recovery.RMStateStore (RMStateStore.java:transition(199)) - Storing info for app: application_1463059172548_0011 2016-05-12 09:56:51,234 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(767)) - application_1463059172548_0011 State change from NEW_SAVING to SUBMITTED 2016-05-12 09:56:51,234 INFO capacity.ParentQueue (ParentQueue.java:addApplication(347)) - Application added - appId: application_1463059172548_0011 user: hadoop leaf-queue of parent: root #applications: 17 2016-05-12 09:56:51,234 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplication(828)) - Accepted application application_1463059172548_0011 from user: hadoop, in queue: default 2016-05-12 09:56:51,234 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(767)) - application_1463059172548_0011 State change from SUBMITTED to ACCEPTED 2016-05-12 09:56:51,234 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:registerAppAttempt(678)) - Registering app attempt : appattempt_1463059172548_0011_000001 2016-05-12 09:56:51,235 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(807)) - appattempt_1463059172548_0011_000001 State change from NEW to SUBMITTED 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,235 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,236 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,236 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,236 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,236 INFO capacity.LeafQueue (LeafQueue.java:activateApplications(629)) - not starting application as amIfStarted exceeds amLimit 2016-05-12 09:56:51,236 INFO capacity.LeafQueue (LeafQueue.java:addApplicationAttempt(678)) - Application added - appId: application_1463059172548_0011 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@67b7a456, leaf-queue: default #user-pending-applications: 10 #user-active-applications: 2 #queue-pending-applications: 15 #queue-active-applications: 2 2016-05-12 09:56:51,237 INFO capacity.CapacityScheduler (CapacityScheduler.java:addApplicationAttempt(857)) - Added Application Attempt appattempt_1463059172548_0011_000001 to scheduler from user hadoop in queue default 2016-05-12 09:56:51,238 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:handle(807)) - appattempt_1463059172548_0011_000001 State change from SUBMITTED to SCHEDULED

Re: INSERT COMMAND HANGING

aliyesami — Thu, 12 May 2016 22:15:20 GMT

I found the answer on google, changing the below parameter from 0.2 to 0.7 fixed the issue.

yarn.scheduler.capacity.maximum-am-resource-percent=0.7

thanks all for your help n feedback