Reply
Explorer
Posts: 8
Registered: ‎01-11-2017

Cloudera Install - Cluster Setup - Failed to execute command Start on service HBase

Hi,

   Hope someone could advise in an error I have encountered on the installation of Cloudera Enterprise 5.9.0 on Red Hat Enterprise Linux Server release 6.5 (Santiago).  The error I am getting at the Cluster Setup screen for the First Run Command is Completed 5 of 10 step(s) at the Start HBase, Solr:  "Failed to execute command Start on service HBase".   Your help would be greatly appreciated.

 

Regards,

Peter

Cloudera Employee
Posts: 44
Registered: ‎08-13-2014

Re: Cloudera Install - Cluster Setup - Failed to execute command Start on service HBase

Hi,
What do the role logs tell you about the failure? Check the HBase logs to see if the cause is indicated there. If you can post any errors you see at the end of the logs to the forum it might shed some light on the issue.

Regards,
Jim
Explorer
Posts: 8
Registered: ‎01-11-2017

Re: Cloudera Install - Cluster Setup - Failed to execute command Start on service HBase

I restarted the AWS cloudera cluster and reran the Cluster Setup.  It got passed HBase but no it's stuck on starting YARN.  

 

Start YARN (MR2 Included), Key-Value Store Indexer
Completed only 1/2 steps. First failure: Failed to execute command Start on service YARN (MR2 Included)

 

The Events Logs show:

 

ATEGORY
AUDIT_EVENT
COMMAND
First Run
COMMAND_ARGS
BasicCmdArgs{args=[zookeeper, hdfs, hbase, solr, yarn, ks_indexer, spark_on_yarn, hive, impala, oozie, hue, mgmt]}
COMMAND_ID
181
COMMAND_RESULT
Completed only 1/2 steps. First failure: Failed to execute command Start on service YARN (MR2 Included)
COMMAND_STATUS
FAILED
EVENTCODE
EV_FIRST_RUN
MESSAGE_CODES
COMMAND_FAILED
SERVICE
Cloudera Manager
SERVICE_TYPE
ManagerServer
SEVERITY
INFORMATIONAL
URL
http://ip-172-31-3-59.us-west-2.compute.internal:7180/cmf/eventRedirect/0e221f30-6fdd-4959-9bb3-edd0...
USER
clouderaManager
__persist_timestamp
1484280404172
__uuid
0e221f30-6fdd-4959-9bb3-edd0d11a7810
January 12, 2017 11:06 PM
View
IMPORTANT caught end of stream exception
Service: ZooKeeper Role: Server (ip-172-31-3-59) Hosts: ip-172-31-3-59.us-west-2.compute.internal Category: LOG_MESSAGE Collapse
CATEGORY
LOG_MESSAGE
CLUSTER
cluster
CLUSTER_DISPLAY_NAME
Cluster 1
CLUSTER_ID
1
EVENTCODE
EV_LOG_EVENT
EXCEPTION_TYPES
org.apache.zookeeper.server.ServerCnxn$EndOfStreamException
HOSTS
ip-172-31-3-59.us-west-2.compute.internal
HOST_IDS
i-00e6d8c7ac612fe00
LOG_LEVEL
WARN
ROLE
Server (ip-172-31-3-59)
ROLE_TYPE
SERVER
SERVICE
ZooKeeper
SERVICE_TYPE
ZOOKEEPER
SEVERITY
IMPORTANT
STACKTRACE
EndOfStreamException: Unable to read additional data from client sessionid 0x15995f59c850027, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745)

URL
http://ip-172-31-3-59.us-west-2.compute.internal:7180/cmf/eventRedirect/991b3e96-59c0-49dd-a403-2d05...
__persist_timestamp
1484280396985
__uuid
991b3e96-59c0-49dd-a403-2d053e2c93be

 

CRITICAL
The health test result for HBASE_REGION_SERVERS_HEALTHY has become bad: Healthy RegionServer: 0. Concerning RegionServer: 0. Total RegionServer: 3. Percent healthy: 0.00%. Percent healthy or concerning: 0.00%. Critical threshold: 90.00%.
Service: HBase Category: HEALTH_CHECK Collapse
ALERT_SUMMARY
The health of service hbase has become bad.
BAD_TEST_RESULTS
1
CATEGORY
HEALTH_CHECK
CLUSTER
cluster
CLUSTER_DISPLAY_NAME
Cluster 1
CLUSTER_ID
1
CURRENT_COMPLETE_HEALTH_TEST_RESULTS
{"content":"The health test result for HBASE_MASTER_HEALTH has become bad: Master summary: ip-172-31-3-59.us-west-2.compute.internal (Availability: Unknown, Health: Bad). This health test is bad because the Service Monitor did not find an active Master.","testName":"HBASE_MASTER_HEALTH","eventCode":"EV_SERVICE_HEALTH_CHECK_BAD","severity":"CRITICAL","suppressed":false},{"content":"The health test result for HBASE_REGION_SERVERS_HEALTHY has become bad: Healthy RegionServer: 0. Concerning RegionServer: 0. Total RegionServer: 3. Percent healthy: 0.00%. Percent healthy or concerning: 0.00%. Critical threshold: 90.00%.","testName":"HBASE_REGION_SERVERS_HEALTHY","eventCode":"EV_SERVICE_HEALTH_CHECK_BAD","severity":"CRITICAL","suppressed":false}
CURRENT_HEALTH_SUMMARY
RED
EVENTCODE
EV_SERVICE_HEALTH_CHECK_BAD
HEALTH_TEST_NAME
HBASE_REGION_SERVERS_HEALTHY
HEALTH_TEST_RESULTS
HBASE_REGION_SERVERS_HEALTHY
Event Code: EV_SERVICE_HEALTH_CHECK_BAD
MONITOR_STARTUP
false
PREVIOUS_COMPLETE_HEALTH_TEST_RESULTS
{"content":"The health test result for HBASE_MASTER_HEALTH has become bad: Master summary: ip-172-31-3-59.us-west-2.compute.internal (Availability: Unknown, Health: Bad). This health test is bad because the Service Monitor did not find an active Master.","testName":"HBASE_MASTER_HEALTH","eventCode":"EV_SERVICE_HEALTH_CHECK_BAD","severity":"CRITICAL","suppressed":false},{"content":"The health test result for HBASE_REGION_SERVERS_HEALTHY has become good: Healthy RegionServer: 3. Concerning RegionServer: 0. Total RegionServer: 3. Percent healthy: 100.00%. Percent healthy or concerning: 100.00%.","testName":"HBASE_REGION_SERVERS_HEALTHY","eventCode":"EV_SERVICE_HEALTH_CHECK_GOOD","severity":"INFORMATIONAL","suppressed":false}
PREVIOUS_HEALTH_SUMMARY
RED
SERVICE
HBase
SERVICE_TYPE
HBASE
SEVERITY
CRITICAL
URL
http://ip-172-31-3-59.us-west-2.compute.internal:7180/cmf/eventRedirect/0318e54a-49a4-4b70-91b3-0775...
__persist_timestamp
1484280185498
__uuid
0318e54a-49a4-4b70-91b3-0775b2d2fe41

Explorer
Posts: 8
Registered: ‎01-11-2017

Re: Cloudera Install - Cluster Setup - Failed to execute command Start on service HBase

I'm creating this as a POC environment.  I'm speculating that this is related to storage space when I setup the ControlNode.  The AWS storage for the control node is set to 10 GB which could be the issue.  I can increase this to 100 GB but not really sure if this is the issue.  I have it setup as 1 Control Node and 3 data Nodes each with 10GB. The control node is set to an EC2 m3.large and the data nodes are set as m3.medium.

Explorer
Posts: 8
Registered: ‎01-11-2017

Re: Cloudera Install - Cluster Setup - Failed to execute command Start on service HBase

I increased the size of the nodes, in the configuration assigned more memory to a couple configuration items that were listed, restarted and this appeared to resolve my issue.
Highlighted
Cloudera Employee
Posts: 44
Registered: ‎08-13-2014

Re: Cloudera Install - Cluster Setup - Failed to execute command Start on service HBase

Hi,
I'm glad to hear you managed to resolve the issue. Choosing too small an instance, particularly in regards to memory, will often give you a cluster that installs correctly but have one or more services with insufficient resources to start correctly.

Another common issue related to this is having insufficient memory on your worker nodes so that the maximum number of YARN containers is very small. This can lead to jobs not launching correctly as they are blocked waiting for resources.

Regards,
Jim
Announcements