Installation of HDP2.4 with Ambari got stuck

[#1]

I am setting up a 3 node HDP cluster using Ambari. I have installed ambari server and continuing step wise installation through web UI. I had completed setup till step 9, install,start and test but it was not moving ahead of 32%(see screenshot 2017-08-19.png). So, I restarted the ambari server and now the UI has started from first step again.

Now, I have reached step 3, confirm hosts. The hosts are registered and status is success but The UI is not moving ahead and the message being show is 'Please wait while the hosts are being checked for potential problems...' (see screenshot 2017-08-19-2.png) I tried waiting for a long time but still it doesn't progresses further.

I have also tried restarting ambari agents, restarting ambari server but still same result. These steps I am performing now are already passed once successfully and I haven't changed anything major on the nodes.

Please suggest a solution.

[#2]

@Anup Shirolkar

Can you copy the stuck URL of step 3

http://xxxxx:8080/step3 

and open a new window paste it and change the number to 4 , it will continue.

http://xxxxx:8080/step4

and hit continue

Please let me now

[#3]

Thank you for the workaround. But i did reset the ambari server and have started from step 1.

I have progressed till step 9 i.e. 'Install, Start and Test'.

However, My installation is seems to stuck at 18%(spent more than couple of hours 2017-08-19-4.png) with no specific errors on any node. The status is waiting to install xxx service. It will be great help if you can tell anything to make it progress? (see screenshot)

[#4]

Hi @Geoffrey Shelton Okot

I'm facing the same problem we had before, but I'm using ambari-2.6.0.0 to install HDP2.6.3. The workaround doen't work for me because in step 5, when I have to decide on which node I have to install which service, there is no server listed, althought it was green in step 3. I worked through all suggestions and did everything twice in minimum but I'm totally not able to install Hortonworks and this is really frustrating, because I did it so many times before. Maybe you can give me a final hint to proceed here.

Attaches the Screenshot:

ambari.jpg

Thanks in advance and best regards,

Christian

[#5]

@Christian Nunez

Can you check in the ambari database whether the hosts have been registered below is from mysql

mysql> select host_id,host_name,last_registration_time, public_host_name from hosts;

Please let me know, an advice always open a new thread because this is closed thread and members usually ignore.

[#6]

@Geoffrey Shelton Okot

Thank you very much. Next time I will open a new Thread.

Here is the output of the query:

ambaridatabase.jpg

ambari=> select host_id, host_name, last_registration_time, public_host_name from hosts; host_id | host_name | last_registration_time | public_host_name ---------+---------------------------+------------------------+--------------------------- 1 | master1.XXXXXX.de | 1512415944737 | master1.XXXXXX.de 2 | slave1.XXXXXX.de | 1512415945031 | slave1.XXXXXX.de 4 | slave3.XXXXXX.de | 1512415948038 | slave3.XXXXXX.de 3 | slave2.XXXXXX.de | 1512415948901 | slave2.XXXXXX.de 5 | master2.XXXXXX.de | 1512415944037 | master2.XXXXXX.de

Thank you so much,

Christian

[#7]

Hello everyone,

It seems like there is a bug in the newest Ambari version. I took this link as stated in the installation guide of Hortonworks: "wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.0.0/ambari.repo -O /etc/yum.repos.d/ambari.repo", but when I changed to "wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.0.3/ambari.repo -O /etc/yum.repos.d/ambari.repo" the installation went through also with HDP2.6.3. Does anyone know why?

But anyways now it is working :-).

Best regards, Christian

[#8]

@Anup Shirolkar

There are a couple of things I would like you to clarify

check the host entry on all the servers they should be identical

cat  /etc/hosts

Your yum repos, these 2 files should be point to either public or internal repo and these files should be available on all the nodes in the cluster

cat /etc/yum.repos.d/ambari.repo
cat /etc/yum.repos.d/hdp.repo

Make sure that the firewall is disabled on all the hosts

Passwordless connect is working !

Zip and upload here your ambari-server logs found in

cat /var/log/ambari-server/*

[#9]

Thank you for the info. @Geoffrey Shelton Okot

I checked that the /etc/hosts file on all nodes with proper FQDNs.

The /etc/yum.repos.d/ambari.repo and /etc/yum.repos.d/HDP.repo also contain urls for hdp on internet and there is internet connectivity on nodes.

Just the difference is HDP.repo, 'HDP' is ambari-server-logs.zipcaps I think this should not be a problem.

The pausing behavior varies every time, it gets stuck at any point in installation and ends after a long time with timeout sign for uninstalled services.

Attaching files of /var/log/ambari-server/* ambari-server-logs.zip

[#10 · Accepted solution]

@Anup Shirolkar

Ambari server log

2017-08-18 11:17:39,720 [CRITICAL] [HIVE] [hive_server_process] (HiveServer2 Process) Connection failed on host hdp25-node2.wulme4ci31tu3lwdofvykqwgkh.bx.internal.cloudapp.net:10000 (Traceback (most recent call last): 

Ensure ambari agent is running and the port is is free

(Ambari Agent Heartbeat) hdp25-node1.wulme4ci31tu3lwdofvykqwgkh.bx.internal.cloudapp.net

copy the Ambari,HDP* .repo to /etc/yum.repos.d/ to all other hosts

Confirm the repos are accessible by

# yum repolist

You should see something like this

HDP-2.3.2.0                                       | 2.9 kB     00:00
HDP-UTILS-1.1.0.20                                | 2.9 kB     00:00
Updates-ambari-2.1.2.1                            | 2.9 kB     00:00

Check the ambari-agents on these nodes are running if not restart them ensure the value hostname points to your ambari server in the /etc/ambari-agent/conf/ambari-agent.ini

[server] 
hostname={your-ambari-server} 
url_port=8440 
secured_url_port=8441 
hdp25-node1.wulme4ci31tu3lwdofvykqwgkh.bx.internal.cloudapp.net is not sending heartbeats 
hdp25-node2.wulme4ci31tu3lwdofvykqwgkh.bx.internal.cloudapp.net is not sending heartbeats 
hdp25-node3.wulme4ci31tu3lwdofvykqwgkh.bx.internal.cloudapp.net is not sending heartbeats 

Error

Caused by: org.apache.ambari.server.HostNotFoundException: Host not found, hostname= 

Double check your DNS.

$hostname -f 

The output should be FQDN I see a lot of connection refused in the log can you ensure the ambari server can access the other hosts in the cluster