Reply
Contributor
Posts: 28
Registered: ‎03-04-2014

Unable to bring up Impala daemon

hi,

 

 impalad version 2.1.2-cdh5

 

When i try to bring up Impala service , statestore , catalog servers are up but the daemons are not coming up , totally  9 nodes are configured for daemons.

 

I do not see any error/log nothing in /var/log/impalad

 

 

I see the logs given below from scm-agent location


==> cloudera-scm-agent.log <==

[10/Apr/2015 17:56:55 +0000] 40453 CP Server Thread-7 _cplogging INFO 169.35.91.254 - - [10/Apr/2015:17:56:55] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0"
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Deleting process 1604-impala-IMPALAD
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Triggering supervisord update.
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Retiring process 1604-impala-IMPALAD
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Activating Process 1614-impala-IMPALAD
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Created /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Chowning /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD to impala (5004832) impala (5003869)
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD to 0751
[10/Apr/2015 17:56:55 +0000] 40453 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.3.2-1.cdh5.3.2.p0.10'}, [u'cdh', u'impala'], [u'impala-plugin']
[10/Apr/2015 17:56:55 +0000] 40453 MainThread parcel INFO The following requested parcels are not available: {}
[10/Apr/2015 17:56:55 +0000] 40453 MainThread parcel INFO Obtained tags ['cdh', 'impala', 'sentry', 'solr', 'spark'] for parcel CDH
[10/Apr/2015 17:56:55 +0000] 40453 MainThread parcel INFO prepare_environment end: {'CDH': '5.3.2-1.cdh5.3.2.p0.10'}
[10/Apr/2015 17:56:55 +0000] 40453 MainThread util INFO Extracted 15 files and 0 dirs to /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD.
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Created /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD/logs
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Chowning /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD/logs to impala (5004832) impala (5003869)
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD/logs to 0751
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO reading limits: {u'limit_memlock': None, u'limit_fds': None}
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Triggering supervisord update.

==> supervisord.log <==
2015-04-10 17:56:55,499 INFO spawned: '1614-impala-IMPALAD' with pid 6566
2015-04-10 17:56:55,502 INFO exited: 1614-impala-IMPALAD (exit status 127; not expected)

 

==> cloudera-scm-agent.log <==
[10/Apr/2015 17:56:55 +0000] 40453 MainThread abstract_monitor INFO Refreshing ImpaladMonitor for None
[10/Apr/2015 17:56:55 +0000] 40453 MainThread __init__ INFO New monitor: (<cmf.monitor.impalad.ImpaladMonitor object at 0x2f8cd10>,)
[10/Apr/2015 17:56:55 +0000] 40453 MainThread __init__ INFO Starting query monitoring thread.

 

==> supervisord.log <==
2015-04-10 17:56:56,539 INFO spawned: '1614-impala-IMPALAD' with pid 6749
2015-04-10 17:56:56,543 INFO exited: 1614-impala-IMPALAD (exit status 127; not expected)
2015-04-10 17:56:58,546 INFO spawned: '1614-impala-IMPALAD' with pid 6750
2015-04-10 17:56:58,549 INFO exited: 1614-impala-IMPALAD (exit status 127; not expected)

 

==> cloudera-scm-agent.log <==
[10/Apr/2015 17:56:58 +0000] 40453 Audit-Plugin audit_plugin INFO stopping Audit Plugin with pipelines
[10/Apr/2015 17:56:58 +0000] 40453 Audit-Plugin audit_plugin_pipeline INFO stopping (None)

 

==> supervisord.log <==
2015-04-10 17:57:01,554 INFO spawned: '1614-impala-IMPALAD' with pid 6751
2015-04-10 17:57:01,558 INFO exited: 1614-impala-IMPALAD (exit status 127; not expected)
2015-04-10 17:57:02,559 INFO gave up: 1614-impala-IMPALAD entered FATAL state, too many start retries too quickly

 

==> cloudera-scm-agent.log <==
[10/Apr/2015 17:57:55 +0000] 40453 MonitorDaemon-Scheduler __init__ INFO Monitor expired: ('ImpaladMonitor for impala-IMPALAD-47ca717fdad7c75f75c9555c264423dc',)

 

 

Role Log /Stdout : has written Nothing  

 

stderr: 

supervisor: couldn't setuid to <UID>: Can't find uid <UID>
supervisor: child process was not spawned

 

Can you please help.

 

 

Regards

Suresh

Highlighted
Contributor
Posts: 28
Registered: ‎03-04-2014

Re: Unable to bring up Impala daemon

Hi,

Can someone help on this
Explorer
Posts: 10
Registered: ‎11-11-2014

Re: Unable to bring up Impala daemon

This is a very late update, but still thought it would help someone facing similar issue.

 

This issue was fixed after hard restarting agent.

 

service cloudera-scm-agent hard_restart_confirmed

 

Reason being the uids from AD were cached by agent, when centrifyDC agent gets restarted the uids couldn't relfect in the server , so a hard restart of the agent flushes all cached ids.