Ambari Blueprints stuck.

[#1]

Hello community! I have been trying to deploy a cluster with high availability for namenode with Ambari Blueprints following this tutorial and this blueprints, however even if I am using the same blueprints and they are accepted by Ambari Server (After deactivating topology check), once the process starts and I take a look at Ambari Server, all hosts groups are in "PENDING HOST ASSIGNMENT" status, as can be seen in this screenshot:

At first I thought there might be a connection problem between ambari agent and ambari server, like hostnames not being resolved properly, then I could see how from Ambari Server I can ssh to any other host using the hostname instead of the IP.

Then I thought maybe the agents didn't have Ambari Server FQDN properly set in the config file, sadly it was correctly written and it is resolved since I can ssh form the agent machine to the server one.

So unless there is something missing I believe this has to be related with using virtual machines and how the network adapter is working for them. Since I am running a Chef cookbook to install and deploy the software with Kitchen, I am using Vagrant with VirtualBox as a provider, due to this the first adapter must be NAT and then I added a second adapter as Bridged, so machines can connect and communicate between them, also the domain name can be resolved. Can it be a problem with Ambari and the need to have the bridged network card on the second adapter slot instead of the first one, which is NAT and can't communicate with other machines? Any idea to workaround this? Anything I am missing in the blueprint? Thanks a lot!

[#2]

@Jose Luis Navarro Vicente

Couple of things to check

1. Can you please make sure that you have referred example blueprint and configured your blueprint accordingly?

2. Is it possible to attach your blueprint to this thread?

Common issues

1. Ambari agent not installed on Ambari server - Can you please confirm if this is not the case.

2. Please see this bug - https://issues.apache.org/jira/browse/AMBARI-18179

[#3]

Hello Kuldeep,

1. Can you please make sure that you have referred example blueprint and configured your blueprint accordingly? 2. Is it possible to attach your blueprint to this thread?

The blueprint I am using is the same I linked in my question, and I am using the same cluster.json that is being used in the guide also linked in my question. Obviously changing the hostnames to match my own cluster ones.

1. Ambari agent not installed on Ambari server - Can you please confirm if this is not the case.

Ambari Agent is installed in Ambari Server machine.

[vagrant@node1-centos-72 ~]$ ps -ef | grep ambari
root      2863  2137  0 Oct11 tty1     00:00:00 sudo ambari-server start
root      2864  2863  0 Oct11 tty1     00:00:00 bash /sbin/ambari-server start
root      2883  2864  0 Oct11 tty1     00:00:00 /usr/bin/python /usr/sbin/ambari-server.py start
root      2895  2883  0 Oct11 tty1     00:00:33 /usr/jdk64/jdk1.8.0_77/bin/java -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.checks.DatabaseConsistencyChecker
root      3429     1  0 Oct11 ?        00:00:00 /bin/sh -c ulimit -n 10000 ; /usr/jdk64/jdk1.8.0_77/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true   -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar' org.apache.ambari.server.controller.AmbariServer > /var/log/ambari-server/ambari-server.out 2>&1 || echo $? > /var/run/ambari-server/ambari-server.exitcode &
root      3430  3429  1 Oct11 ?        00:13:13 /usr/jdk64/jdk1.8.0_77/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -Dsun.zip.disableMemoryMapping=true -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer
root      3639     1  0 Oct11 ?        00:00:00 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/AmbariAgent.py start
root     30722  3639  0 07:17 ?        00:00:03 /usr/bin/python /usr/lib/python2.6/site-packages/ambari_agent/main.py start
vagrant  31430  4910  0 07:41 pts/0    00:00:00 grep --color=auto ambari
[vagrant@node1-centos-72 ~]$

2. Please see this bug - https://issues.apache.org/jira/browse/AMBARI-18179

ZKFC is added to the namenodes, as it can be seen in the blueprint.

[#4]

I forgot to mention I am running this on Centos7, Ambari is version 2.4 and I am trying to install HDP stack 2.5.

[#5]

@Jose Luis Navarro Vicente

Is your host registered?

Try :

api/v1/clusters/$clustername/hosts

If it hasn't been registered, you need to update /etc/ambari-agent/conf/ambari-agent.ini and restart ambari agent

[#6]

Hello sbhat,

Thanks for your answer, this seems to be closer the answer.

They are not registered, indeed. However the ambari-agent.ini contains the correct fqdn of ambari server, and it can be resolved. In fact, even the ambari-agent in the same machine where ambari-server is is not shown as registered. Any idea? The agent is running on all hosts, double checked.