Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to bring up Impala daemon

Highlighted

Unable to bring up Impala daemon

Explorer

hi,

 

 impalad version 2.1.2-cdh5

 

When i try to bring up Impala service , statestore , catalog servers are up but the daemons are not coming up , totally  9 nodes are configured for daemons.

 

I do not see any error/log nothing in /var/log/impalad

 

 

I see the logs given below from scm-agent location


==> cloudera-scm-agent.log <==

[10/Apr/2015 17:56:55 +0000] 40453 CP Server Thread-7 _cplogging INFO 169.35.91.254 - - [10/Apr/2015:17:56:55] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0"
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Deleting process 1604-impala-IMPALAD
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Triggering supervisord update.
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Retiring process 1604-impala-IMPALAD
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Activating Process 1614-impala-IMPALAD
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Created /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Chowning /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD to impala (5004832) impala (5003869)
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD to 0751
[10/Apr/2015 17:56:55 +0000] 40453 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.3.2-1.cdh5.3.2.p0.10'}, [u'cdh', u'impala'], [u'impala-plugin']
[10/Apr/2015 17:56:55 +0000] 40453 MainThread parcel INFO The following requested parcels are not available: {}
[10/Apr/2015 17:56:55 +0000] 40453 MainThread parcel INFO Obtained tags ['cdh', 'impala', 'sentry', 'solr', 'spark'] for parcel CDH
[10/Apr/2015 17:56:55 +0000] 40453 MainThread parcel INFO prepare_environment end: {'CDH': '5.3.2-1.cdh5.3.2.p0.10'}
[10/Apr/2015 17:56:55 +0000] 40453 MainThread util INFO Extracted 15 files and 0 dirs to /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD.
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Created /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD/logs
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Chowning /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD/logs to impala (5004832) impala (5003869)
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/1614-impala-IMPALAD/logs to 0751
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO reading limits: {u'limit_memlock': None, u'limit_fds': None}
[10/Apr/2015 17:56:55 +0000] 40453 MainThread agent INFO Triggering supervisord update.

==> supervisord.log <==
2015-04-10 17:56:55,499 INFO spawned: '1614-impala-IMPALAD' with pid 6566
2015-04-10 17:56:55,502 INFO exited: 1614-impala-IMPALAD (exit status 127; not expected)

 

==> cloudera-scm-agent.log <==
[10/Apr/2015 17:56:55 +0000] 40453 MainThread abstract_monitor INFO Refreshing ImpaladMonitor for None
[10/Apr/2015 17:56:55 +0000] 40453 MainThread __init__ INFO New monitor: (<cmf.monitor.impalad.ImpaladMonitor object at 0x2f8cd10>,)
[10/Apr/2015 17:56:55 +0000] 40453 MainThread __init__ INFO Starting query monitoring thread.

 

==> supervisord.log <==
2015-04-10 17:56:56,539 INFO spawned: '1614-impala-IMPALAD' with pid 6749
2015-04-10 17:56:56,543 INFO exited: 1614-impala-IMPALAD (exit status 127; not expected)
2015-04-10 17:56:58,546 INFO spawned: '1614-impala-IMPALAD' with pid 6750
2015-04-10 17:56:58,549 INFO exited: 1614-impala-IMPALAD (exit status 127; not expected)

 

==> cloudera-scm-agent.log <==
[10/Apr/2015 17:56:58 +0000] 40453 Audit-Plugin audit_plugin INFO stopping Audit Plugin with pipelines
[10/Apr/2015 17:56:58 +0000] 40453 Audit-Plugin audit_plugin_pipeline INFO stopping (None)

 

==> supervisord.log <==
2015-04-10 17:57:01,554 INFO spawned: '1614-impala-IMPALAD' with pid 6751
2015-04-10 17:57:01,558 INFO exited: 1614-impala-IMPALAD (exit status 127; not expected)
2015-04-10 17:57:02,559 INFO gave up: 1614-impala-IMPALAD entered FATAL state, too many start retries too quickly

 

==> cloudera-scm-agent.log <==
[10/Apr/2015 17:57:55 +0000] 40453 MonitorDaemon-Scheduler __init__ INFO Monitor expired: ('ImpaladMonitor for impala-IMPALAD-47ca717fdad7c75f75c9555c264423dc',)

 

 

Role Log /Stdout : has written Nothing  

 

stderr: 

supervisor: couldn't setuid to <UID>: Can't find uid <UID>
supervisor: child process was not spawned

 

Can you please help.

 

 

Regards

Suresh

2 REPLIES 2

Re: Unable to bring up Impala daemon

Explorer
Hi,

Can someone help on this

Re: Unable to bring up Impala daemon

This is a very late update, but still thought it would help someone facing similar issue.

 

This issue was fixed after hard restarting agent.

 

service cloudera-scm-agent hard_restart_confirmed

 

Reason being the uids from AD were cached by agent, when centrifyDC agent gets restarted the uids couldn't relfect in the server , so a hard restart of the agent flushes all cached ids.