Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to start Haddop services on Sandbox

Unable to start Haddop services on Sandbox

I am running a sandbox with HDP 2.2.4.2. on VMware. after starting when I start all hadoop services, the services do not start. The error reported is:

Fail: Execution of 'ls /var/run/hadoop-yarn/yarn/yarn-yarn-timelineserver.pid >/dev/null 2>&1 && ps -p `cat /var/run/hadoop-yarn/yarn/yarn-yarn-timelineserver.pid` >/dev/null 2>&1' returned 1. 
2015-12-09 23:13:07,046 - Command: /usr/bin/hdp-select status hadoop-yarn-timelineserver > /tmp/tmp6_S5Dp
Output: hadoop-yarn-timelineserver - 2.2.4.2-2
When single hdfs service is started it shows success but within few minutes the service is stopped again. but no error is reported. the log file in /var/log/hadoop/hdfs/hadoop-hdfs-namenode-sandbox.hortonworks.com.log shows following entries:
2015-12-09 23:43:13,572 INFO  namenode.FSNamesystem (FSNamesystem.java:listCorruptFileBlocks(7509)) - there are no corrupt file blocks.
2015-12-09 23:43:14,014 INFO  namenode.FSNamesystem (FSNamesystem.java:listCorruptFileBlocks(7509)) - there are no corrupt file blocks.
2015-12-09 23:43:17,678 INFO  provider.AsyncAuditProvider (AsyncAuditProvider.java:logSummaryIfRequired(215)) - AsyncAuditProvider-stats:HdfsAuditProvider: past 01:00.008 minutes: inLogs=36, outLogs=36, dropped=0, currentQueueSize=0
2015-12-09 23:43:17,678 INFO  provider.AsyncAuditProvider (AsyncAuditProvider.java:logSummaryIfRequired(221)) - AsyncAuditProvider-stats:HdfsAuditProvider: process lifetime: inLogs=630, outLogs=630, dropped=0
2015-12-09 23:43:17,742 INFO  provider.AsyncAuditProvider (AsyncAuditProvider.java:logSummaryIfRequired(215)) - AsyncAuditProvider-stats:DbAuditProvider: past 01:00.015 minutes: inLogs=36, outLogs=36, dropped=0, currentQueueSize=0
2015-12-09 23:43:17,742 INFO  provider.AsyncAuditProvider (AsyncAuditProvider.java:logSummaryIfRequired(221)) - AsyncAuditProvider-stats:DbAuditProvider: process lifetime: inLogs=631, outLogs=631, dropped=0
2015-12-09 23:43:19,145 INFO  blockmanagement.CacheReplicationMonitor (CacheReplicationMonitor.java:run(178)) - Rescanning after 30001 milliseconds
2015-12-09 23:43:19,145 INFO  blockmanagement.CacheReplicationMonitor (CacheReplicationMonitor.java:run(201)) - Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s).
2015-12-09 23:43:20,079 INFO  namenode.FSNamesystem (FSNamesystem.java:listCorruptFileBlocks(7509)) - there are no corrupt file blocks.
2015-12-09 23:43:20,524 INFO  namenode.FSNamesystem (FSNamesystem.java:listCorruptFileBlocks(7509)) - there are no corrupt file blocks.
2015-12-09 23:43:21,384 INFO  httpclient.HttpMethodDirector (HttpMethodDirector.java:executeWithRetry(439)) - I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2015-12-09 23:43:21,385 INFO  httpclient.HttpMethodDirector (HttpMethodDirector.java:executeWithRetry(445)) - Retrying request
2015-12-09 23:43:21,385 INFO  httpclient.HttpMethodDirector (HttpMethodDirector.java:executeWithRetry(439)) - I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2015-12-09 23:43:21,385 INFO  httpclient.HttpMethodDirector (HttpMethodDirector.java:executeWithRetry(445)) - Retrying request
2015-12-09 23:43:21,386 INFO  httpclient.HttpMethodDirector (HttpMethodDirector.java:executeWithRetry(439)) - I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2015-12-09 23:43:21,386 INFO  httpclient.HttpMethodDirector (HttpMethodDirector.java:executeWithRetry(445)) - Retrying request
2015-12-09 23:43:21,386 WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(206)) - Unable to send metrics to collector by address:http://sandbox.hortonworks.com:6188/ws/v1/timeline/metrics
[root@sandbox hdfs]#
9 REPLIES 9
Highlighted

Re: Unable to start Haddop services on Sandbox

When starting the VM I also see this error:

safemode: Call from sandbox.hortonworks.com/192.168.171.136 to sandbox.hortonworks.com.8020 failed on connection exception: connection refused

Is there any settings in the VMware Player which is causing this issue?

Highlighted

Re: Unable to start Haddop services on Sandbox

@Sanjay Sharma

Connection refused

Looks like services are not up. Please login to ambari console and try to start the services one by one

Start with hdfs then mapreduce and yarn

Highlighted

Re: Unable to start Haddop services on Sandbox

@Neeraj Sabharwal

The connection refused message is displayed when VM is being initialized. After VM is initialized, services are not initialized. All attempts to start services using Ambari is failing with the error:

Execution of 'ls /var/run/hadoop-yarn/yarn/yarn-yarn-timelineserver.pid >/dev/null 2>&1 && ps -p `cat /var/run/hadoop-yarn/yarn/yarn-yarn-timelineserver.pid` >/dev/null 2>&1' returned 1

The single service initialization for hdfs is also failing.

Highlighted

Re: Unable to start Haddop services on Sandbox

@Sanjay Sharma Did you change anything in the /etc/hosts?

Highlighted

Re: Unable to start Haddop services on Sandbox

@Sanjay Sharma This is related to networking issue if I am not wrong.

Would you mind reimporting the image and then follow the install guide?

Highlighted

Re: Unable to start Haddop services on Sandbox

I did the re-import. The start-up still displyed the connection refused error. The services started correctly but now the spring jobs for loading data in hdfs are failing due to connection problem:

2015-12-12T01:00:07+0000 1.2.0.RELEASE ERROR task-scheduler-1 step.AbstractStep - Encountered an error executing step reconcile in job **** java.net.ConnectException: Call From sandbox.hortonworks.com/192.168.171.137 to localhost.localdomain:8020 failed on connection exception: java.net.ConnectException:

Highlighted

Re: Unable to start Haddop services on Sandbox

Highlighted

Re: Unable to start Haddop services on Sandbox

If the HDFS is in safemode, then some of the operations may be prevented from completing. You can try to exit safemode and restart services by following these steps

1. Login to the sandbox as root

2. Switch user to hdfs 

3. hdfs dfsadmin -safemode leave

4. Try to restart the services with HDFS first, then YARN, MapReduce, etc. 
Highlighted

Re: Unable to start Haddop services on Sandbox

Followed the above steps with no success. All services show not running. I tried starting name node but it shuts down immediately. The name node log shows following errors:

2015-12-10 23:03:15,265 ERROR datanode.DataNode (DataXceiver.java:run(253)) - sandbox.hortonworks.com:50010:DataXceiver error processing unknown operation src: /127.0.0.1:57983 dst: /127.0.0.1:50010

java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)

Don't have an account?
Coming from Hortonworks? Activate your account here