History Server can not start

[#1]

After installing fresh HDP 2.5.3 cluster (ambari 2.4.1.0), all sevices (default selection) installed successfully without any warning. When Starting services, History Server fail to start and make mapreduce fail also.

curl: (52) Empty reply from server
100

[#2]

@Wael Horchani

Anything obvious you see in history server log.

[#3]

where is located this log file?

thanks in advance.

[#4]

please find below the error and log filesstderr.txtstdout.txt

[#5]

@Wael Horchani

Can you please check the "History Server" log to see if there is any error?

Does it has sufficient memory ? And Warning/Error in the log?

You can find the log in the following location on History Server host:

/var/log/hadoop-mapreduce/mapred 

- Based on the error looks like it might be having some connection issue with NameNode. So can you try running the same curl command from the HistoryServer Host to see if it works fine?

curl -sS -L  -X PUT --data-binary @/usr/hdp/2.5.3.0-37/hadoop/mapreduce.tar.gz 'http://vds002.databridge.tn:50070/webhdfs/v1/hdp/apps/2.5.3.0-37/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'

.

[#6]

Hi Jay,

Thanks for the help.

1. no files found under /var/log/hadoop-mapreduce/mapred
2. the command doesn't works : curl: (7) Failed connect to vds002.databridge.tn:50070; Connection refused

The cluster is one machine. The firewalld is disabled.

any idea plz?

[#7]

@Wael Horchani

Once the History Server will start it should write in the mentioned log location. However it should at lease have written the out file there.

Anyway now we know that the cause is "Failed connect to vds002.databridge.tn:50070; Connection refused"

Can you please make sure that the "vds002.databridge.tn:50070" host and portare accessible from the History Server Host. Are you able to do from History Server (for remote testing)

telnet vds002.databridge.tn  50070

However as you mentioned that the cluster is on a single host hence can you please share what is the output of the following command:

1) FQDN

hostname -f

2). Port "50070" is opened or not ?

netstat -tnlpa | grep 50070

Also can you also please share the value for the property "dfs.namenode.http-address" from "Custom hdfs-site". You can get that value form ambari. Please check the value of that mentioned property uses the correct FQDN

.

[#8]

@Jay SenSharma

1. hostname -f : vds002.databridge.tn
2. netstat -tnlpa | grep 50070 : nothing

root@vds002:~# netstat -tnlpa | grep 50070

root@vds002:~#

how to enable communication on this port. trying : firewall-cmd --add-port 50070/tcp --permanent

but no effect.

[#9]

I had the same issue, increasing the heap space for HDFS helped. Also apparently a reboot was required to disable selinux - as the command sestatus was showing enabled.

[#10]

@Wael Horchani

As you mentioned that you have a single host cluster. So I guess NameNode also will be running on the same host. So in that case the port "50070" should be opened. As the netstat output is showing that no such port is opened which means your NameNode is down. Please bring it On.

Also please check NameNode log to see if it ever started successfully or not? and why it did not open the port.