question Re: History Server can not start in Archives of Support Questions (Read Only)

History Server can not start

wael_horchani1 — Wed, 21 Dec 2016 18:04:06 GMT

After installing fresh HDP 2.5.3 cluster (ambari 2.4.1.0), all sevices (default selection) installed successfully without any warning. When Starting services, History Server fail to start and make mapreduce fail also.

curl: (52) Empty reply from server
100

Re: History Server can not start

rpathak — Wed, 21 Dec 2016 18:06:26 GMT

@Wael Horchani

Anything obvious you see in history server log.

Re: History Server can not start

wael_horchani1 — Wed, 21 Dec 2016 18:29:38 GMT

where is located this log file?

thanks in advance.

Re: History Server can not start

wael_horchani1 — Wed, 21 Dec 2016 21:49:00 GMT

please find below the error and log filesstderr.txt stdout.txt

Re: History Server can not start

jsensharma — Wed, 21 Dec 2016 22:00:26 GMT

@Wael Horchani

Can you please check the "History Server" log to see if there is any error?

Does it has sufficient memory ? And Warning/Error in the log?

You can find the log in the following location on History Server host:

/var/log/hadoop-mapreduce/mapred

- Based on the error looks like it might be having some connection issue with NameNode. So can you try running the same curl command from the HistoryServer Host to see if it works fine?

curl -sS -L  -X PUT --data-binary @/usr/hdp/2.5.3.0-37/hadoop/mapreduce.tar.gz 'http://vds002.databridge.tn:50070/webhdfs/v1/hdp/apps/2.5.3.0-37/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'

Re: History Server can not start

wael_horchani1 — Wed, 21 Dec 2016 23:03:29 GMT

Hi Jay,

Thanks for the help.

no files found under /var/log/hadoop-mapreduce/mapred
the command doesn't works : curl: (7) Failed connect to vds002.databridge.tn:50070; Connection refused

The cluster is one machine. The firewalld is disabled.

any idea plz?

Re: History Server can not start

jsensharma — Wed, 21 Dec 2016 23:07:26 GMT

@Wael Horchani

Once the History Server will start it should write in the mentioned log location. However it should at lease have written the out file there.

Anyway now we know that the cause is "Failed connect to vds002.databridge.tn:50070; Connection refused"

Can you please make sure that the "vds002.databridge.tn:50070" host and portare accessible from the History Server Host. Are you able to do from History Server (for remote testing)

telnet vds002.databridge.tn  50070

However as you mentioned that the cluster is on a single host hence can you please share what is the output of the following command:

1) FQDN

hostname -f

2). Port "50070" is opened or not ?

netstat -tnlpa | grep 50070

Also can you also please share the value for the property "dfs.namenode.http-address" from "Custom hdfs-site". You can get that value form ambari. Please check the value of that mentioned property uses the correct FQDN

Re: History Server can not start

wael_horchani1 — Thu, 22 Dec 2016 03:10:38 GMT

@Jay SenSharma

hostname -f : vds002.databridge.tn
netstat -tnlpa | grep 50070 : nothing

root@vds002:~# netstat -tnlpa | grep 50070

root@vds002:~#

how to enable communication on this port. trying : firewall-cmd --add-port 50070/tcp --permanent

but no effect.

Re: History Server can not start

jsensharma — Thu, 22 Dec 2016 03:44:59 GMT

@Wael Horchani

As you mentioned that you have a single host cluster. So I guess NameNode also will be running on the same host. So in that case the port "50070" should be opened. As the netstat output is showing that no such port is opened which means your NameNode is down. Please bring it On.

Also please check NameNode log to see if it ever started successfully or not? and why it did not open the port.

Re: History Server can not start

wael_horchani1 — Fri, 23 Dec 2016 02:34:10 GMT

@Jay SenSharma

the real problem is the namenode heap of memory. When History Server try to start, The memory usage of the NameNode climbs quickly to exceed the limit of 1 Gega byte (default configuration) and causes the service to fall. When changing max memory heap to 3 Gb it works fine. I installed previously ambari 2.4.0.1 and i don't seen this behaviour (2.4.2.0 same behaviour as 2.4.1.0). Do you know why?

Re: History Server can not start

bugs0522 — Fri, 20 Jan 2017 09:18:18 GMT

Hi, in which secction do I increase the memory? HDFS or MapReduce? thanks a lot

Re: History Server can not start

wael_horchani1 — Sat, 21 Jan 2017 05:23:02 GMT

Hi Aldo, in HDFS. The parameter is called "NameNode Java heap size".

Re: History Server can not start

avijeetd — Wed, 05 Apr 2017 12:44:53 GMT

I had the same issue, increasing the heap space for HDFS helped. Also apparently a reboot was required to disable selinux - as the command sestatus was showing enabled.

Re: History Server can not start

masahiro_hirama — Mon, 30 Oct 2017 18:08:55 GMT

Hi, thank you for each of you by contributing with insights and experiences.

Just my contribution:

I could install my 1st cluster recently and I have got these behavior from it; a dumb distraction was the cause: I thought /etc/hosts were fine and they were not (missing entries) - fixing them properly and restarting services, they got up promptly - reds became green at end in ambari dashboard.

cheers and good luck

Massa