Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

All service can't start, and not have any service log.

avatar
author-rank Sartner
Rising Star

Created on ‎10-19-2015 12:01 AM - edited ‎09-16-2022 02:44 AM

I have a cluster with 5 servers (hostname hadoop-215 to hadoop-219);

 

Today, I changed Hue configuration, but it can't startup.

Then I tried out other services(Impala/HBase thrift server),  both of them can't be started, and have no logs in /var/logs/{service_name}.

 

 

 

 

cdh.png

 

$ tail /var/log/cloudera-scm-server/cloudera-scm-server.log

2015-10-19 13:46:39,857 INFO StaleEntityEviction:com.cloudera.cmf.model.HeartbeatStore: Reaped 1 process heartbeats
2015-10-19 13:46:39,861 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
2015-10-19 13:46:39,862 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2013-10-19T05:46:39.861Z to reap.
2015-10-19 13:46:39,863 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests.
2015-10-19 13:46:39,863 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests.
2015-10-19 13:46:48,220 INFO 1810859969@scm-web-125735:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command Start SvcCmdArgs{targetRoles=[DbRole{id=107, name=hue-HUE_SERVER-b672e8f973cd8ddace77def263871fb0, hostName=hadoop-215}], args=[]}. Service: DbService{id=23, name=hue}
2015-10-19 13:46:48,243 INFO 1810859969@scm-web-125735:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=23, name=hue} Role: DbRole{id=107, name=hue-HUE_SERVER-b672e8f973cd8ddace77def263871fb0, hostName=hadoop-215}
2015-10-19 13:46:48,307 INFO 1810859969@scm-web-125735:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: Added BringUp command to service DbService{id=23, name=hue}.
2015-10-19 13:46:48,340 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:46:53,345 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:46:58,350 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:03,354 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:03,434 INFO 1338356818@agentServer-14005:com.cloudera.cmf.command.components.StalenessChecker: No staleness check scheduled, scheduling one in 30 seconds
2015-10-19 13:47:03,456 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:08,460 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:13,465 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:18,471 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:23,476 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:28,481 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:33,434 INFO ScheduledStalenessChecker:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ProcessStalenessCheckCommand BasicCmdArgs{args=[First reason why: Process (id=940) has a brand new heartbeat]}.
2015-10-19 13:47:33,440 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:33,949 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.AbstractRoleHandler: (26 skipped) Client configs of DbService{id=22, name=zookeeper} wanted but not available.
2015-10-19 13:47:33,977 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check done. Duration: PT0.535S 
2015-10-19 13:47:33,982 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:38,989 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:43,993 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:48,998 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:54,003 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:47:59,008 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:48:04,012 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:48:09,017 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:48:14,022 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:48:19,027 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:48:24,032 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:48:29,038 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:48:34,042 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:48:39,054 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:48:44,059 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:48:49,063 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:48:54,068 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:48:59,072 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:49:04,077 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:49:09,081 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:49:14,085 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) is still active on service DbService{id=23, name=hue}.
2015-10-19 13:49:19,090 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: Aborting BringUp command (3097) on service DbService{id=23, name=hue} role DbRole{id=107, name=hue-HUE_SERVER-b672e8f973cd8ddace77def263871fb0, hostName=hadoop-215}.
2015-10-19 13:49:19,098 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (3096) has finished on service DbService{id=23, name=hue}.
2015-10-19 13:49:33,589 WARN 770526468@agentServer-14007:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=940 name=null host=1db4c618-eee6-49cc-9d8f-475b1fe7fd15/hadoop-215
2015-10-19 13:56:39,865 INFO StaleEntityEviction:com.cloudera.cmf.model.HeartbeatStore: Reaped 1 process heartbeats
2015-10-19 13:56:39,872 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
2015-10-19 13:56:39,874 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2013-10-19T05:56:39.872Z to reap.
2015-10-19 13:56:39,875 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests.
2015-10-19 13:56:39,875 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests.
2015-10-19 14:00:00,003 INFO com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command GlobalPoolsRefresh BasicCmdArgs{scheduleId=1, scheduledTime=2015-10-19T06:00:00.000Z}.
2015-10-19 14:00:00,140 INFO com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.scheduler.CommandDispatcherJob: Skipping scheduled command 'GlobalPoolsRefresh' since it is a noop.

$ tail /var/log/cloudera-scm-agent/cloudera-scm-agent.log

[19/Oct/2015 13:46:48 +0000] 16355 CP Server Thread-6 _cplogging   INFO     192.168.4.215 - - [19/Oct/2015:13:46:48] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0"
[19/Oct/2015 13:46:48 +0000] 16355 MainThread util         INFO     Using generic audit plugin for process hue-HUE_SERVER
[19/Oct/2015 13:46:48 +0000] 16355 MainThread util         INFO     Creating metadata plugin for process hue-HUE_SERVER
[19/Oct/2015 13:46:48 +0000] 16355 MainThread util         INFO     Using specific metadata plugin for process hue-HUE_SERVER
[19/Oct/2015 13:46:48 +0000] 16355 MainThread util         INFO     Using generic metadata plugin for process hue-HUE_SERVER
[19/Oct/2015 13:46:48 +0000] 16355 MainThread __init__     INFO     Instantiating generic monitor for service HUE and role HUE_SERVER
[19/Oct/2015 13:46:48 +0000] 16355 MainThread agent        INFO     Activating Process 940-hue-HUE_SERVER
[19/Oct/2015 13:46:48 +0000] 16355 MainThread agent        INFO     Created /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER
[19/Oct/2015 13:46:48 +0000] 16355 MainThread agent        INFO     Chowning /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER to hue (480) hue (479)
[19/Oct/2015 13:46:48 +0000] 16355 MainThread agent        INFO     Chmod'ing /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER to 0751
[19/Oct/2015 13:46:48 +0000] 16355 MainThread parcel       INFO     prepare_environment begin: {u'GPLEXTRAS': u'5.4.3-1.cdh5.4.3.p0.5', u'CDH': u'5.4.3-1.cdh5.4.3.p0.6', u'KAFKA': u'0.8.2.0-1.kafka1.3.0.p0.29'}, [u'cdh'], [u'cdh-plugin', u'hue-plugin']
[19/Oct/2015 13:46:48 +0000] 16355 MainThread parcel       INFO     The following requested parcels are not available: {}
[19/Oct/2015 13:46:48 +0000] 16355 MainThread parcel       INFO     Obtained tags ['kafka'] for parcel KAFKA
[19/Oct/2015 13:46:48 +0000] 16355 MainThread parcel       INFO     Obtained tags ['cdh', 'impala', 'sentry', 'solr', 'spark'] for parcel CDH
[19/Oct/2015 13:46:48 +0000] 16355 MainThread parcel       INFO     Obtained tags ['cdh-plugin', 'impala-plugin', 'solr-plugin', 'spark-plugin', 'hadoop_lzo'] for parcel GPLEXTRAS
[19/Oct/2015 13:46:48 +0000] 16355 MainThread parcel       INFO     prepare_environment end: {'CDH': '5.4.3-1.cdh5.4.3.p0.6', 'GPLEXTRAS': '5.4.3-1.cdh5.4.3.p0.5'}
[19/Oct/2015 13:46:48 +0000] 16355 MainThread util         INFO     Extracted 35 files and 0 dirs to /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER.
[19/Oct/2015 13:46:48 +0000] 16355 MainThread agent        INFO     Created /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER/logs
[19/Oct/2015 13:46:48 +0000] 16355 MainThread agent        INFO     Chowning /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER/logs to hue (480) hue (479)
[19/Oct/2015 13:46:48 +0000] 16355 MainThread agent        INFO     Chmod'ing /var/run/cloudera-scm-agent/process/940-hue-HUE_SERVER/logs to 0751
[19/Oct/2015 13:46:48 +0000] 16355 MainThread cgroups      INFO     Creating cgroup /var/run/cloudera-scm-agent/cgroups/blkio/940-hue-HUE_SERVER
[19/Oct/2015 13:46:48 +0000] 16355 MainThread cgroups      INFO     Creating cgroup /var/run/cloudera-scm-agent/cgroups/cpuacct/940-hue-HUE_SERVER
[19/Oct/2015 13:46:48 +0000] 16355 MainThread cgroups      INFO     Creating cgroup /var/run/cloudera-scm-agent/cgroups/cpu/940-hue-HUE_SERVER
[19/Oct/2015 13:46:48 +0000] 16355 MainThread cgroups      INFO     Reconfiguring cgroup pseudofile /var/run/cloudera-scm-agent/cgroups/cpu/940-hue-HUE_SERVER/cpu.rt_runtime_us with value 1000
[19/Oct/2015 13:46:48 +0000] 16355 MainThread agent        INFO     reading limits: {u'limit_memlock': None, u'limit_fds': None}
[19/Oct/2015 13:46:48 +0000] 16355 MainThread agent        INFO     Triggering supervisord update.
[19/Oct/2015 13:46:48 +0000] 16355 MainThread navigator_plugin INFO     Scheduling a refresh for Audit Plugin for hue-HUE_SERVER with pipelines []
[19/Oct/2015 13:46:48 +0000] 16355 MainThread navigator_plugin INFO     Scheduling a refresh for Metadata Plugin for hue-HUE_SERVER with pipelines []
[19/Oct/2015 13:46:48 +0000] 16355 MainThread abstract_monitor INFO     Refreshing GenericMonitor HUE-HUE_SERVER for None
[19/Oct/2015 13:46:48 +0000] 16355 MainThread __init__     INFO     New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x416f550>,)
[19/Oct/2015 13:46:51 +0000] 16355 Audit-Plugin navigator_plugin INFO     Refreshing Audit Plugin for hue-HUE_SERVER with pipelines []
[19/Oct/2015 13:46:51 +0000] 16355 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for hue-HUE_SERVER (log dir: None)
[19/Oct/2015 13:46:51 +0000] 16355 Metadata-Plugin navigator_plugin INFO     Refreshing Metadata Plugin for hue-HUE_SERVER with pipelines []
[19/Oct/2015 13:46:51 +0000] 16355 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for hue-HUE_SERVER (log dir: None)
[19/Oct/2015 13:47:48 +0000] 16355 MonitorDaemon-Scheduler __init__     INFO     Monitor expired: ('GenericMonitor HUE-HUE_SERVER for hue-HUE_SERVER-b672e8f973cd8ddace77def263871fb0',)
[19/Oct/2015 13:49:33 +0000] 16355 MainThread agent        INFO     Deactivating process 940-hue-HUE_SERVER
[19/Oct/2015 13:49:33 +0000] 16355 MainThread agent        INFO     Supervisord doesn't know about process 940-hue-HUE_SERVER. It cannot be deactivated by this Agent, assuming it even exists
[19/Oct/2015 13:49:33 +0000] 16355 MainThread agent        INFO     Deleting process 940-hue-HUE_SERVER
[19/Oct/2015 13:49:33 +0000] 16355 MainThread cgroups      INFO     Destroying cgroup /var/run/cloudera-scm-agent/cgroups/blkio/940-hue-HUE_SERVER
[19/Oct/2015 13:49:33 +0000] 16355 MainThread cgroups      INFO     Destroying cgroup /var/run/cloudera-scm-agent/cgroups/cpuacct/940-hue-HUE_SERVER
[19/Oct/2015 13:49:33 +0000] 16355 MainThread cgroups      INFO     Destroying cgroup /var/run/cloudera-scm-agent/cgroups/cpu/940-hue-HUE_SERVER
[19/Oct/2015 13:49:33 +0000] 16355 MainThread agent        INFO     Triggering supervisord update.
[19/Oct/2015 13:49:33 +0000] 16355 MainThread agent        INFO     Retiring process 940-hue-HUE_SERVER
[19/Oct/2015 13:49:36 +0000] 16355 Audit-Plugin navigator_plugin INFO     stopping Audit Plugin for hue-HUE_SERVER with pipelines []
[19/Oct/2015 13:49:36 +0000] 16355 Audit-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for hue-HUE_SERVER (log dir: None)
[19/Oct/2015 13:49:36 +0000] 16355 Metadata-Plugin navigator_plugin INFO     stopping Metadata Plugin for hue-HUE_SERVER with pipelines []
[19/Oct/2015 13:49:36 +0000] 16355 Metadata-Plugin navigator_plugin_pipeline INFO     Stopping Navigator Plugin Pipeline '' for hue-HUE_SERVER (log dir: None)

 

Now, I moved the Hue server to another server, It's start successed.

But the server hadoop-215 is my master server(have namenode, resource manager, hbase master on it), can't be stoped.

 

 

Any body can help me ? 

 

 

 

 

 

 

 

 

 

 

 

 

4,698 Views
0 Kudos
1
1 ACCEPTED SOLUTION

avatar
author-rank Sartner
Rising Star

Created ‎10-19-2015 06:31 PM

I restarted the cm-agent on the error server. The problem solved..

View solution in original post

4,563 Views
0 Kudos
1 REPLY 1

avatar
author-rank Sartner
Rising Star

Created ‎10-19-2015 06:31 PM

I restarted the cm-agent on the error server. The problem solved..
4,564 Views
0 Kudos
Announcements
What's New @ Cloudera
[RELEASED] Cloudera Streams Messaging - Kubernetes Operator ...
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics 1.14 for Cloudera Pu...
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
View More Announcements