Ambari host registration with server failed

[#1]

I am trying to build a three node HDP cluster but getting below error during host registration. I am able to ssh from node1.example.com to other two nodes without password.screenshot-from-2016-10-23-11-08-52.png

• node1.example.com - ambari-server running
• node2.example.com - ambari-agent running
  • vi /etc/ambari-agent/conf/ambari-agent.ini
    • hostname = node1.example.com
    • url_port = 8440
    • secured_url_port = 8441
    • ip_address = 192.168.0.1

• node3.example.com - ambari-agent running
• node4.example.com - local repository where Apache web server is running.

Ambari - 2.4.2.0, HDP - 2.4.0.0 and cent os 7.

What is the missing part here? Please help!

Log: node2.example.com

========================== Creating target directory... ==========================

Command start time 2016-12-25 00:20:46

Connection to node2.example.com closed. SSH command execution finished host=node2.example.com, exitcode=0 Command end time 2016-12-25 00:20:47

========================== Copying ambari sudo script... ==========================

Command start time 2016-12-25 00:20:47

scp /var/lib/ambari-server/ambari-sudo.sh host=node2.example.com, exitcode=0 Command end time 2016-12-25 00:20:50

========================== Copying common functions script... ==========================

Command start time 2016-12-25 00:20:50

scp /usr/lib/python2.6/site-packages/ambari_commons host=node2.example.com, exitcode=0 Command end time 2016-12-25 00:20:55

========================== Copying OS type check script... ==========================

Command start time 2016-12-25 00:20:55

scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py host=node2.example.com, exitcode=0 Command end time 2016-12-25 00:21:09

========================== Running OS type check... ==========================

Command start time 2016-12-25 00:21:09 Cluster primary/cluster OS family is redhat7 and local/current OS family is redhat7

Connection to node2.example.com closed. SSH command execution finished host=node2.example.com, exitcode=0 Command end time 2016-12-25 00:21:16

========================== Checking 'sudo' package on remote host... ==========================

Command start time 2016-12-25 00:21:16

Connection to node2.example.com closed. SSH command execution finished host=node2.example.com, exitcode=0 Command end time 2016-12-25 00:21:17

========================== Copying setup script file... ==========================

Command start time 2016-12-25 00:21:24

scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py host=node2.example.com, exitcode=0 Command end time 2016-12-25 00:21:26

========================== Running setup agent script... ==========================

Command start time 2016-12-25 00:21:26 (' File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 355, in main (retries, connected, stopped) = netutil.try_to_connect(server_url, MAX_RETRIES, logger) UnboundLocalError: local variable \'server_url\' referenced before assignment INFO 2016-12-24 23:47:25,915 ExitHelper.py:53 - Performing cleanup before exiting... INFO 2016-12-25 00:08:33,242 main.py:90 - loglevel=logging.INFO INFO 2016-12-25 00:08:33,242 main.py:90 - loglevel=logging.INFO INFO 2016-12-25 00:08:33,242 main.py:90 - loglevel=logging.INFO INFO 2016-12-25 00:08:33,246 DataCleaner.py:39 - Data cleanup thread started INFO 2016-12-25 00:08:33,248 DataCleaner.py:120 - Data cleanup started INFO 2016-12-25 00:08:33,248 DataCleaner.py:122 - Data cleanup finished INFO 2016-12-25 00:08:33,328 PingPortListener.py:50 - Ping port listener started on port: 8670 WARNING 2016-12-25 00:08:33,331 main.py:351 - Unable to determine the IP address of the Ambari server \'node1.example.com:8080\' ERROR 2016-12-25 00:08:33,331 main.py:392 - Exiting with exception: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 387, in <module> main(heartbeat_stop_callback) File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 355, in main (retries, connected, stopped) = netutil.try_to_connect(server_url, MAX_RETRIES, logger) UnboundLocalError: local variable \'server_url\' referenced before assignment INFO 2016-12-25 00:08:33,527 ExitHelper.py:53 - Performing cleanup before exiting... ', None) (' File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 355, in main (retries, connected, stopped) = netutil.try_to_connect(server_url, MAX_RETRIES, logger) UnboundLocalError: local variable \'server_url\' referenced before assignment INFO 2016-12-24 23:47:25,915 ExitHelper.py:53 - Performing cleanup before exiting... INFO 2016-12-25 00:08:33,242 main.py:90 - loglevel=logging.INFO INFO 2016-12-25 00:08:33,242 main.py:90 - loglevel=logging.INFO INFO 2016-12-25 00:08:33,242 main.py:90 - loglevel=logging.INFO INFO 2016-12-25 00:08:33,246 DataCleaner.py:39 - Data cleanup thread started INFO 2016-12-25 00:08:33,248 DataCleaner.py:120 - Data cleanup started INFO 2016-12-25 00:08:33,248 DataCleaner.py:122 - Data cleanup finished INFO 2016-12-25 00:08:33,328 PingPortListener.py:50 - Ping port listener started on port: 8670 WARNING 2016-12-25 00:08:33,331 main.py:351 - Unable to determine the IP address of the Ambari server \'node1.example.com:8080\' ERROR 2016-12-25 00:08:33,331 main.py:392 - Exiting with exception: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 387, in <module> main(heartbeat_stop_callback) File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 355, in main (retries, connected, stopped) = netutil.try_to_connect(server_url, MAX_RETRIES, logger) UnboundLocalError: local variable \'server_url\' referenced before assignment INFO 2016-12-25 00:08:33,527 ExitHelper.py:53 - Performing cleanup before exiting... ', None)Connection to node2.example.com closed. SSH command execution finished host=node2.example.com, exitcode=255 Command end time 2016-12-25 00:21:47ERROR: Bootstrap of host node2.example.com fails because previous action finished with non-zero exit code (255) ERROR MESSAGE: Connection to node2.example.com closed.STDOUT: (' File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 355, in main (retries, connected, stopped) = netutil.try_to_connect(server_url, MAX_RETRIES, logger) UnboundLocalError: local variable \'server_url\' referenced before assignment INFO 2016-12-24 23:47:25,915 ExitHelper.py:53 - Performing cleanup before exiting... INFO 2016-12-25 00:08:33,242 main.py:90 - loglevel=logging.INFO INFO 2016-12-25 00:08:33,242 main.py:90 - loglevel=logging.INFO INFO 2016-12-25 00:08:33,242 main.py:90 - loglevel=logging.INFO INFO 2016-12-25 00:08:33,246 DataCleaner.py:39 - Data cleanup thread started INFO 2016-12-25 00:08:33,248 DataCleaner.py:120 - Data cleanup started INFO 2016-12-25 00:08:33,248 DataCleaner.py:122 - Data cleanup finished INFO 2016-12-25 00:08:33,328 PingPortListener.py:50 - Ping port listener started on port: 8670 WARNING 2016-12-25 00:08:33,331 main.py:351 - Unable to determine the IP address of the Ambari server \'node1.example.com:8080\' ERROR 2016-12-25 00:08:33,331 main.py:392 - Exiting with exception: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 387, in <module> main(heartbeat_stop_callback) File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 355, in main (retries, connected, stopped) = netutil.try_to_connect(server_url, MAX_RETRIES, logger) UnboundLocalError: local variable \'server_url\' referenced before assignment INFO 2016-12-25 00:08:33,527 ExitHelper.py:53 - Performing cleanup before exiting... ', None) (' File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 355, in main (retries, connected, stopped) = netutil.try_to_connect(server_url, MAX_RETRIES, logger) UnboundLocalError: local variable \'server_url\' referenced before assignment INFO 2016-12-24 23:47:25,915 ExitHelper.py:53 - Performing cleanup before exiting... INFO 2016-12-25 00:08:33,242 main.py:90 - loglevel=logging.INFO INFO 2016-12-25 00:08:33,242 main.py:90 - loglevel=logging.INFO INFO 2016-12-25 00:08:33,242 main.py:90 - loglevel=logging.INFO INFO 2016-12-25 00:08:33,246 DataCleaner.py:39 - Data cleanup thread started INFO 2016-12-25 00:08:33,248 DataCleaner.py:120 - Data cleanup started INFO 2016-12-25 00:08:33,248 DataCleaner.py:122 - Data cleanup finished INFO 2016-12-25 00:08:33,328 PingPortListener.py:50 - Ping port listener started on port: 8670 WARNING 2016-12-25 00:08:33,331 main.py:351 - Unable to determine the IP address of the Ambari server \'node1.example.com:8080\' ERROR 2016-12-25 00:08:33,331 main.py:392 - Exiting with exception: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 387, in <module> main(heartbeat_stop_callback) File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 355, in main (retries, connected, stopped) = netutil.try_to_connect(server_url, MAX_RETRIES, logger) UnboundLocalError: local variable \'server_url\' referenced before assignment INFO 2016-12-25 00:08:33,527 ExitHelper.py:53 - Performing cleanup before exiting... ', None)

Connection to node2.example.com closed.

Thank you.

BR - Sampath

[#2 · Accepted solution]

@Sampath Kumar

Have you followed the pre-requisites to prepare all the nodes?

The following "Unable to determine the IP address of the Ambari server \'node1.example.com:8080\'" indicates that you did not resolve the FDQN requirement. Read this: https://community.hortonworks.com/questions/42911/fqdns-are-they-necessary.html.

Let me know.

[#3]

deployment-hdp.png yumreposd-directory.png

Thank you so much. I rebuilt all the nodes and followed the pre-requisites to prepare the same. Now Ambari host registration with server is successful. But during deployment, I am getting below error message. Can you please help on this.

Log:

line 140, in _call_wrapper

result = _call(command, **kwargs_copy)

File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 293, in _call

raise ExecutionFailed(err_msg, code, out, err)

resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install hdp-select' returned 1. One of the configured repositories failed (HDP-2.4),

and yum doesn't have enough cached data to continue. At this point the only

safe thing yum can do is fail. There are a few ways to work "fix" this:

1. Contact the upstream for the repository and get them to fix the problem.

2. Reconfigure the baseurl/etc. for the repository, to point to a working

upstream. This is most often useful if you are using a newer

distribution release than is supported by the repository (and the

packages for the previous distribution release still work).

3. Run the command with the repository temporarily disabled

yum --disablerepo=HDP-2.4 ...

4. Disable the repository permanently, so yum won't use it by default. Yum

will then just ignore the repository until you permanently enable it

again or use --enablerepo for temporary usage:

yum-config-manager --disable HDP-2.4

or

subscription-manager repos --disable=HDP-2.4

5. Configure the failing repository to be skipped, if it is unavailable.

Note that yum will try to contact the repo. when it runs most commands,

so will have to try and fail each time (and thus. yum will be be much

slower). If it is a very temporary problem though, this is often a nice

compromise:

yum-config-manager --save --setopt=HDP-2.4.skip_if_unavailable=true

failure: repodata/repomd.xml from HDP-2.4: [Errno 256] No more mirrors to try.

http://192.168.0.12/repo/HDP/centos7/2.x/updates/2.4.0.0/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found

stdout: /var/lib/ambari-agent/data/output-200.txt

2016-12-26 18:21:41,206 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf

2016-12-26 18:21:41,208 - Group['spark'] {}

2016-12-26 18:21:41,319 - Initializing 2 repositories

2016-12-26 18:21:41,320 - Repository['HDP-2.4'] {'base_url': 'http://192.168.0.12/repo/HDP/centos7/2.x/updates/2.4.0.0/', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP', 'mirror_list': None}

2016-12-26 18:21:41,339 - File['/etc/yum.repos.d/HDP.repo'] {'content': '[HDP-2.4]\nname=HDP-2.4\nbaseurl=http://192.168.0.12/repo/HDP/centos7/2.x/updates/2.4.0.0/\n\npath=/\nenabled=1\ngpgcheck=0'}

2016-12-26 18:21:41,341 - Repository['HDP-UTILS-1.1.0.20'] {'base_url': 'http://192.168.0.12/repo/HDP-UTILS-1.1.0.20/repos/centos7/', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}

2016-12-26 18:21:41,347 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': '[HDP-UTILS-1.1.0.20]\nname=HDP-UTILS-1.1.0.20\nbaseurl=http://192.168.0.12/repo/HDP-UTILS-1.1.0.20/repos/centos7/\n\npath=/\nenabled=1\ngpgcheck=0'}

2016-12-26 18:21:41,348 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5}

2016-12-26 18:21:41,528 - Skipping installation of existing package unzip

2016-12-26 18:21:41,528 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5}

2016-12-26 18:21:41,558 - Skipping installation of existing package curl

2016-12-26 18:21:41,558 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5}

2016-12-26 18:21:41,593 - Installing package hdp-select ('/usr/bin/yum -d 0 -e 0 -y install hdp-select')

2016-12-26 18:21:41,932 - Execution of '/usr/bin/yum -d 0 -e 0 -y install hdp-select' returned 1. One of the configured repositories failed (HDP-2.4),

and yum doesn't have enough cached data to continue. At this point the only

safe thing yum can do is fail. There are a few ways to work "fix" this:

1. Contact the upstream for the repository and get them to fix the problem.

2. Reconfigure the baseurl/etc. for the repository, to point to a working

upstream. This is most often useful if you are using a newer

distribution release than is supported by the repository (and the

packages for the previous distribution release still work).

3. Run the command with the repository temporarily disabled

yum --disablerepo=HDP-2.4 ...

4. Disable the repository permanently, so yum won't use it by default. Yum

will then just ignore the repository until you permanently enable it

again or use --enablerepo for temporary usage:

yum-config-manager --disable HDP-2.4

or

subscription-manager repos --disable=HDP-2.4

5. Configure the failing repository to be skipped, if it is unavailable.

Note that yum will try to contact the repo. when it runs most commands,

so will have to try and fail each time (and thus. yum will be be much

slower). If it is a very temporary problem though, this is often a nice

compromise:

yum-config-manager --save --setopt=HDP-2.4.skip_if_unavailable=true

failure: repodata/repomd.xml from HDP-2.4: [Errno 256] No more mirrors to try.

http://192.168.0.12/repo/HDP/centos7/2.x/updates/2.4.0.0/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found

2016-12-26 18:21:41,932 - Failed to install package hdp-select. Executing '/usr/bin/yum clean metadata'

2016-12-26 18:21:42,267 - Retrying to install package hdp-select after 30 seconds

Command failed after 1 tries

BR-Sampath