question Re: Ambari setup stuck at step 3 in Support Questions

Ambari setup stuck at step 3

rahul_narayanan — Mon, 13 Nov 2017 18:37:39 GMT

Hi,

While installing Ambari 2.6 to setup a 2 node HDP cluster i was stuck at step 3 saying "Please wait while the hosts are being checked for potential problems..". i have checked the ambari-server.log and it says "Unable to lookup the cluster by ID; assuming that there is no cluster and therefore no configs for this execution command: Cluster not found, clusterName=clusterID=-1"

Here is the excerpt from the ambari-server.log

13 Nov 2017 13:09:24,422 WARN [ambari-action-scheduler] ExecutionCommandWrapper:225 - Unable to lookup the cluster by ID; assuming that there is no cluster and therefore no configs for this execution command: Cluster not found, clusterName=clusterID=-1 13 Nov 2017 13:09:25,427 WARN [ambari-action-scheduler] ExecutionCommandWrapper:225 - Unable to lookup the cluster by ID; assuming that there is no cluster and therefore no configs for this execution command: Cluster not found, clusterName=clusterID=-1 13 Nov 2017 13:11:11,433 INFO [pool-18-thread-1] MetricsServiceImpl:65 - Attempting to initialize metrics sink 13 Nov 2017 13:11:11,434 INFO [pool-18-thread-1] MetricsServiceImpl:81 - ********* Configuring Metric Sink ********** 13 Nov 2017 13:11:11,434 INFO [pool-18-thread-1] AmbariMetricSinkImpl:95 - No clusters configured. 13 Nov 2017 13:16:11,434 INFO [pool-18-thread-1] MetricsServiceImpl:65 - Attempting to initialize metrics sink 13 Nov 2017 13:16:11,435 INFO [pool-18-thread-1] MetricsServiceImpl:81 - ********* Configuring Metric Sink ********** 13 Nov 2017 13:16:11,435 INFO [pool-18-thread-1] AmbariMetricSinkImpl:95 - No clusters configured.

pls help

i have followed the steps in hortonworks youtube channel on setting up the cluster but dont know what i am missing.

Thanks

Rahul

Re: Ambari setup stuck at step 3

jsensharma — Mon, 13 Nov 2017 18:44:43 GMT

As we see that the folllowing message is appearing in your logs/UI. Which indicates that the hosts are not yet successfully registered.

 Cluster not found, clusterName=clusterID=-1

In this step ambari server basically performs some of the checks like "transparent huge pages", "ntpd service", "SELinux" ...etc So please check if you have setup the passwordless ssh properly from ambari server host to the agent host.

Also please check if the "iptables" is disabled on the agent machine (sothat it can communicate with ambari server on default ports 8440, 8441). So please check the following from ambari agent machine. Please check the ambari-agent "/etc/hosts" file if it is able to resolving the ambari server hostname.

# nc -v $AMBARI_HOSTNAME 8440
# nc -v $AMBARI_HOSTNAME 8441
(OR)
# telnet $AMBARI_HOSTNAME 8440
# telnet $AMBARI_HOSTNAME 8441

As this is a fresh cluster setup hence i will suggest if above does not solve the problem then try to reset the ambari DB by issuing the following command and then retry the ambari-server (cluster creation wizard)

# ambari-server stop
# ambari-server reset           ----->   (this will clean up ambari DB to default state)
# ambari-server start

Re: Ambari setup stuck at step 3

rahul_narayanan — Tue, 14 Nov 2017 17:23:16 GMT

Thanks @Jay Kumar SenSharma for your quick response

As u said all the host entries , ntpd , selinux all set up properly but still the same issue persists.

i am able to reach the ambari server host from agent at 8440 and 8441

i have also done the below steps several times but still same "Unable to lookup the cluster by ID" persists.

# ambari-server stop
# ambari-server reset -----> (this will clean up ambari DB to default state)
# ambari-server start

i am attaching the ambari-server.log ambari-server.txt

Also attaching the Install wizard screenshot ambari.png

Thanks

Rahul

Re: Ambari setup stuck at step 3

jsensharma — Tue, 14 Nov 2017 17:38:40 GMT

@Rahul Narayanan

In your ambari-server.log i see the following message:

ERROR [main] StackContext:217 - Could not load repo results
java.net.UnknownHostException: public-repo-1.hortonworks.com

Which makes me feel that your ambari-server host does not have internet connectivity Or there is a firewall rule Or proxy that is restricting Ambari Server host to pull repo data from the Hortonworks public repo. I am suspecting that the same might be the case from Ambari Agent hosts as well and hence after validation during host registration when ambari is tryign to add the hosts it is getting stuck.

Please check if your ambari agent hosts have internet connectivity or access to the public repo?

Additionally, Please check that all the hosts are having "lowercase" hostnames, Mixed or Upper case hostnames sometimes causes the issue during installation as well as with HDFS service checks.

Re: Ambari setup stuck at step 3

jsensharma — Tue, 14 Nov 2017 17:57:13 GMT

Additionally, If you are using Local Repositories Or using Http Proxy Host then please do not forget to specify the following properties as described in the following doc:

-Dhttp.proxyHost=<yourProxyHost> -Dhttp.proxyPort=<yourProxyPort><br>

Also in the "/etc/yum.conf" of every host:

proxy=http://<yourProxyHost>:<yourProxyPort>

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-administration/content/ch_setting_up_an_internet_proxy_server_for_ambari.html

Re: Ambari setup stuck at step 3

rahul_narayanan — Tue, 14 Nov 2017 18:11:03 GMT

@Jay Kumar SenSharma

I am using local repository as i don't have internet connection on my nodes. I am giving the local repo details during step 1 (select versions) for hdp and hdputils.

i have my host file will nodes fqdn in lowecase

Thanks

Rahul

Re: Ambari setup stuck at step 3

jsensharma — Tue, 14 Nov 2017 18:45:26 GMT

Can you please check , From your ambari-agent hosts are you able to access the Local Repository?

Re: Ambari setup stuck at step 3

rahul_narayanan — Tue, 14 Nov 2017 19:45:52 GMT

yes, i am able to do wget on to my local repository from ambari agent as well also i tried yum install to check and it was able to connect to local repository.

Re: Ambari setup stuck at step 3

rahul_narayanan — Wed, 15 Nov 2017 17:51:45 GMT

@Jay Kumar SenSharma yes am bale to access the local repository from ambari-agent as well

Also the error "15 Nov 2017 15:00:56,033 ERROR [main] StackContext:217 - Could not load repo results java.net.UnknownHostException: public-repo-1.hortonworks.com"

comes immediately once i start my ambari-server. where will i configure ambari to look in to my local repo and not the public repo

Re: Ambari setup stuck at step 3

Shelton — Wed, 15 Nov 2017 23:49:41 GMT

@Rahul Narayanan

These are the 2 files (baseurl)you should update to point to the local repo

/etc/yum.repos.d/ambari.repo
/etc/yum.repos.d/hdp.repo

Hope that helps

Re: Ambari setup stuck at step 3

rahul_narayanan — Thu, 16 Nov 2017 01:27:36 GMT

@Geoffrey Shelton Okot its already updated to point my local repository

i thought that the repository url that i define in step 1 of install wizard will be used. But anyways my ambari.repo and hdp.repo in all my nodes are updated to point to local repo .

should i create the directory as mentioned in the documents for the baseurl as the tar that i downloaded when extracted was not in that pattern. does it really matters.

Re: Ambari setup stuck at step 3

Shelton — Thu, 16 Nov 2017 05:31:07 GMT

@Rahul Narayanan

It's very important to stick to the official documentation see link

Defining a repository during the install step wont force Ambari to pick up that location it will always read from the /etc/yum.repos.d/*

Yes you should create the /var/www/html/ directory and unzip as stated in the document. Remember to copy the repos to all the nodes in the cluster!

Good luck

Re: Ambari setup stuck at step 3

rahul_narayanan — Thu, 16 Nov 2017 13:30:51 GMT

@Geoffrey Shelton Okot Can you please clarify on this

The tar that i downloaded

ambari-2.6.0.0-centos6.tar.gz when extracted the directory looks like "/<webserver>/ambari/centos6/2.6.0.0-267"

HDP-2.6.3.0-centos6-rpm.tar.gz when extracted looks like "/<webserver>/HDP/centos6/2.6.3.0-235"

HDP-UTILS-1.1.0.21-centos6.tar.gz when extracted does not create structure its just extracts in the same directory

whereas in the doc it says

Ambari Base URL

http://<web.server>/Ambari-2.6.0.0/<OS>;

HDP Base URL

http://<web.server>/hdp/HDP/<OS>/2.x/updates/<latest.version>;

HDP-UTILS Base URL

http://<web.server>/hdp/HDP-UTILS-<version>/repos/<OS>;

so should i manually go ahead and create these directories to look like above.

Re: Ambari setup stuck at step 3

rahul_narayanan — Thu, 16 Nov 2017 15:38:21 GMT

@Geoffrey Shelton Okot

The main issue that i am facing is "Unable to lookup the cluster by ID; assuming that there is no cluster and therefore no configs for this execution command: Cluster not found, clusterName=clusterID=-1"

The only error that i see in ambari-server.log is during ambari-server start which is

ERROR [main]StackContext:217-Couldnot load repo results
java.net.UnknownHostException:public-repo-1.hortonworks.co

even though i have set up my local yum repo as mentioned in the doc.

i get success at step 3 of confirm hosts but the checking for additional warning is looping for infinite and when i see the log it gives unable to find cluster by iD

Re: Ambari setup stuck at step 3

Shelton — Thu, 16 Nov 2017 16:34:30 GMT

@Rahul Narayanan

Can you share your ambari/hdp.repos in /etc/yum.repos.d/*, because the error just confirms that installation is pointing to an public repo

Clean out all packages and metadata from cache

yum clean all

Ensure your ambari & hdp repos point to the internal repos above

Check enabled repos

yum repolist

Retry the installation

Re: Ambari setup stuck at step 3

rahul_narayanan — Thu, 16 Nov 2017 16:47:04 GMT

@Geoffrey Shelton Okot

I have attached the hdp.repo and ambari.repo that i have in all my nodes.

i have also done yum clean all and yum repolist before installing.

The urls in the hdp.repo and ambari.repo is accesible from both the host as well

hdprepo.txt

ambarirepo.txt

Re: Ambari setup stuck at step 3

Shelton — Thu, 16 Nov 2017 17:18:48 GMT

@Rahul Narayanan

When you hit the below URL's can you see the bits you downloaded and unzipped earlier?

http://xxxxmaapl685.ad.infosys.com/Ambari-2.6.0.0/centos6 
http://xxxxmaapl685.ad.infosys.com/hdp/HDP-UTILS-1.1.0.21/repos/centos6 
http://xxxxmaapl685.ad.infosys.com/hdp/HDP/centos6/2.x/updates/2.6.3.0

Please see this 2 videos

localrepo1

localrepo2

Ensure your setup matches this short demo

Re: Ambari setup stuck at step 3

rahul_narayanan — Thu, 16 Nov 2017 18:15:10 GMT

@Geoffrey Shelton Okot

Yes My local repository setup matches with the one that was shown in both the videos

i am able to browse the files in the browser when i hit the following urls

Also i am using the default postgres that ambari installed. will that be a problem or should i configure Mysql

Re: Ambari setup stuck at step 3

Shelton — Thu, 16 Nov 2017 18:58:31 GMT

@Rahul Narayanan

Good news, you can now proceed using postgres doesn't cause any problems at all, but if you are a fan of MySQL then see link.

if you intend to use the embedded postgres then choose option 1 when running

ambari-server setup

Using Postgres is straightforward and easy

Re: Ambari setup stuck at step 3

rahul_narayanan — Thu, 16 Nov 2017 19:43:57 GMT

@Geoffrey Shelton Okot

So far my configuration is proper as per all of yours suggestion I don’t know what I am missing or what is going wrong how can I debug this to solve so that I can create a successful cluster and does not have to see the “unable to find cluster by ID” again.