Ambari Install, Start, and Test Errors

[#1]

Hello,

So I'm quite lost in the errors in my cluster at this point. I'm hoping someone would be kind enough to take a look at the problems below and perhaps give me a couple of pointers or somewhere to start looking.

Background: I used the hortonworks documentation guidelines to setup ambari server and agents, along with a local repository. I am able to access the local repository through my browser as well as a wget from each node. The registration step goes smoothly and no warnings remain in that step. I am using a proxy and added the appropriate -DhttpProxyHost and -DhttpProxyPort in the way recommended to each host.

I have gone through and tested the network configurations given by both of the following links:

https://community.hortonworks.com/storage/attachments/2326-network-and-prereq-setup.pdf

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Installing_HDP_AMB/content/_prepare_the_e...

1. The "Select Stack" step from the install wizard returns a 404 for my local repository base URL, which wasn't happening before but I don't recall exactly when it stopped working (initially the green checkmarks appeared and it failed later on)

2. The 404 repository base url problem seems to be in the logs for the installation errors in the final step as well, after successfully installing accumulo client and server as well as the data node for the appropriate nodes

3. The first failure is the Falcon server, which is the only "Failure encountered" and seems to trigger the warnings for all of the preceding steps.

4. wget to the baseurl works from each node, however wget to the repomd.xml times out and returns the 404 in the end.

Is there anything obvious that I could be missing or an important step that is not included in the documentation? Any advice or ideas are appreciated.

Thanks in advance,

Savanna

[#2]

Can you please verify the BASE URL present in repo files is accessible from all the nodes?.Cross-verify the same in ambari server configuration file.

[#3]

I double checked that the BASE URL in each repo file is the same as the browser URL. I also did wgets to the repomd.xml files from the nodes and this works (200 OK).

I looked at the ambari server configuration file, however I am not sure exactly what I am supposed to change here as the repo links in that file seem to only be for the JDK, which I do not have a repository for and was under the impression I didn't need to have a local repo for. Is this incorrect? What specifically am I supposed to verify in the server configuration file?

[#4]

Pleas can you past the Image where exactly you getting error

[#5]

Thank you for the response @zkfs. Here are the errors:

falcon-err-failure.png install-start-test-main-page.png select-stack-err.png

Relevant/indicative error log from the UI for the falcon error (the only error that actually causes a failure and not just a warning in the Install step:

2016-08-05 07:59:03,052 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2016-08-05 07:59:03,053 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'}
2016-08-05 07:59:03,054 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-08-05 07:59:03,055 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2016-08-05 07:59:03,059 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2016-08-05 07:59:03,059 - Group['hdfs'] {}
2016-08-05 07:59:03,060 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']}
2016-08-05 07:59:03,060 - FS Type: 
2016-08-05 07:59:03,060 - Directory['/etc/hadoop'] {'mode': 0755}
2016-08-05 07:59:03,075 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2016-08-05 07:59:03,075 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777}
2016-08-05 07:59:03,090 - Repository['HDP-2.3'] {'base_url': 'http://xxxxxx:8012/hdp/centos6/HDP-2.3.4.0/', 'action': ['create'], 'components': ['HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP', 'mirror_list': None}
2016-08-05 07:59:03,098 - File['/etc/yum.repos.d/HDP.repo'] {'content': '[HDP-2.3]\nname=HDP-2.3\nbaseurl=http://mo-23f02a5a3.mo.sap.corp:8012/hdp/centos6/HDP-2.3.4.0/\n\npath=/\nenabled=1\ngpgcheck=0'}
2016-08-05 07:59:03,099 - Repository['HDP-UTILS-1.1.0.20'] {'base_url': 'http://xxxxxxx:8012/hdp/centos6/HDP-UTILS-1.1.0.20/', 'action': ['create'], 'components': ['HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}
2016-08-05 07:59:03,102 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': '[HDP-UTILS-1.1.0.20]\nname=HDP-UTILS-1.1.0.20\nbaseurl=http://xxxxxxxxxxxx/hdp/centos6/HDP-UTILS-1.1.0.20/\n\npath=/\nenabled=1\ngpgcheck=0'}
2016-08-05 07:59:03,103 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2016-08-05 07:59:03,219 - Skipping installation of existing package unzip
2016-08-05 07:59:03,219 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2016-08-05 07:59:03,227 - Skipping installation of existing package curl
2016-08-05 07:59:03,228 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2016-08-05 07:59:03,236 - Skipping installation of existing package hdp-select
2016-08-05 07:59:03,376 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-08-05 07:59:03,384 - Package['falcon_2_3_*'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2016-08-05 07:59:03,499 - Installing package falcon_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install 'falcon_2_3_*'')

[#6]

Check your Linux box and see if it has connectivity to your repo . yum is failing to get the package from the repo. Confirm your proxy is working as you intended it. -DhttpProxyHost and -DhttpProxyPort

The Ambari url failure might be related to your desktop is not proxied either - the url check is initiated at the browser.

Please post screenshots of errors .

grep -i baseurl /etc/yum.repos.d/*.repos each result will be a url that all of your linux boxes will need access to via the network.

yum clean all

yum repolist -v - this will show failures if you have a network issue.

yum install unzip

[#7]

So yum repolist -v showed a failure in resolving the proxy. Thank you for your response @mthiele, at least I now know what the problem most likely is.

I am new to working with proxies like this, so please forgive my lack of knowledge. Is the only configuration necessary with my proxy to add those lines -DhttpProxyHost and -DhttpProxyPort to /var/lib/ambari-server/ambari-env.sh? Or is there something else I missed in configuring this? I have played around with this and don't know how else I could input them.

Here is the syntax of how I edited the file if my proxy was 111.111.1.111:8080:

AMBARI_PASSHPHRASE="DEV"
export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dhttp.proxyHost=111.111.1.111 -Dhttp.proxyPort=8080'

Here is the exact response to "yum repolist -v"

[root@xxxxx ~]# yum repolist -v
Not loading "rhnplugin" plugin, as it is disabled
Loading "product-id" plugin
Loading "subscription-manager" plugin
Updating Subscription Management repositories.
Unable to read consumer identity
Config time: 0.118
Yum Version: 3.2.29
<a href="http://MY_SERVER/ambari/centos6/Updates-ambari-2.2.2.0/repodata/repomd.xml:">http://MY_SERVER/ambari/centos6/Updates-ambari-2.2.2.0/repodata/repomd.xml:</a> [Errno 14] PYCURL ERROR 5 - "Couldn't resolve proxy '|111.111.1.111'"
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: Updates-ambari-2.2.2.0. Please verify its path and try again

Here are screenshots of the errors seen with the Ambari UI:

falcon-err-failure.png install-start-test-main-page.png select-stack-err.png

[#8 · Accepted solution]

Hi @Savanna Endicott : Like most people suggested in the post, this looks like a network issue.

I faced the exact same issue today and it looked like my VM had problems connecting to the proxy. It had nothing to do with the ambari repo (I disabled ambari repo and still faced the same issue).

What helped was to disable proxy and try running yum clean all; yum update.

To bypass your proxy, follow the post :

https://community.hortonworks.com/questions/26872/forbidden-403-error-on-hdp-24-installation.html

At the same time, you can talk to your IT folks to check why proxy settings are not working on your node. (May be iptables, selinux services etc are misconfigured)

Hope this helps!

[#9]

Hi @sbhat thank you for your response, I tried following that guide but now my epel.repo is broken because it is provided through my network ( I think that makes sense). This is the result. I guess I could get this repository elsewhere but I have a feeling this will cause other large scale issues as I am using a product for my VMs that is provided only through my company's network.

Is the only way to bypass the proxy completely? I am hoping there is something else I can do.

[root@mo-1184a7ee4 ~]# yum update Loaded plugins: product-id, subscription-manager Setting up Update Process http://my-proxy-addr:8080/mrepo/redhat/6/rhel6epel-x86_64/RPMS.all/repodata/repomd.xml: [Errno 12] Timeout on http://my-proxy-addr:8080/mrepo/redhat/6/rhel6epel-x86_64/RPMS.all/repodata/repomd.xml: (28, 'Operation too slow. Less than 1 bytes/sec transfered the last 30 seconds') Trying other mirror. Error: Cannot retrieve repository metadata (repomd.xml) for repository: epel.repo. Please verify its path and try again

[#10]

I am facing "Failure encountered" error.pngat the final step of installing Ambari as shown below. Could you please help me to resolve this issue