Ambari not using available repos

[#1]

RHEL 7.2 on Power8

ambari-server-2.5.0.0-1096.ppc64le

Trying to install HDP 2.6.

For some reason the install (Install, Start, and Test part of the installation) is failing. It appears that it is having problems using the repos that are available to it. For example:

2017-08-09 18:35:03,706 - Skipping installation of existing package curl
2017-08-09 18:35:03,706 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2017-08-09 18:35:03,728 - Skipping installation of existing package hdp-select
2017-08-09 18:35:04,016 - Package['hadoop_2_6_0_0_598-yarn'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2017-08-09 18:35:04,177 - Skipping installation of existing package hadoop_2_6_0_0_598-yarn
2017-08-09 18:35:04,179 - Package['hadoop_2_6_0_0_598-mapreduce'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2017-08-09 18:35:04,202 - Installing package hadoop_2_6_0_0_598-mapreduce ('/usr/bin/yum -d 0 -e 0 -y install hadoop_2_6_0_0_598-mapreduce')

So it cannot install hadoop_2_6_0_0_598-mapreduce. But if I go to the command line, I can install this:

==============================================================================================================
Package Arch Version Repository Size
==============================================================================================================
Installing:
hadoop_2_6_0_0_598-mapreduce ppc64le 2.7.3.2.6.0.0-598 HDP-2.6 32 M

Transaction Summary
==============================================================================================================
Install 1 Package

Total download size: 32 M
Installed size: 36 M
Is this ok [y/d/N]: y
Downloading packages:
hadoop_2_6_0_0_598-mapreduce-2.7.3.2.6.0.0-598.ppc64le.rpm | 32 MB 00:00:03
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : hadoop_2_6_0_0_598-mapreduce-2.7.3.2.6.0.0-598.ppc64le 1/1
Verifying : hadoop_2_6_0_0_598-mapreduce-2.7.3.2.6.0.0-598.ppc64le 1/1

Installed:
hadoop_2_6_0_0_598-mapreduce.ppc64le 0:2.7.3.2.6.0.0-598

Complete!

I don't understand why Ambari is having such a problem using the HDP and HDP-Utils repositories. They are configured like this:

[root@hdplab80 yum.repos.d]# cat HDP.repo
[HDP-2.6]
name=HDP-2.6
baseurl=http://private-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.0.0-598

path=/
enabled=1
gpgcheck=0[root@hdplab80 yum.repos.d]# cat HDP-UTILS.repo
[HDP-UTILS-1.1.0.21]
name=HDP-UTILS-1.1.0.21
baseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ppc64le

path=/
enabled=1

Can anyone spot what is wrong here? Essentially, Ambari is not using the repos available to it, I think.

Thanks in advance.

John

[#2]

@John Wright

I notice in your error log its the SSL on Ambari server causing the problem.

INFO 2017-08-22 13:33:55,016 NetUtil.py:67 - Connecting to https://F.Q.D.N:8440/ca
ERROR 2017-08-22 13:33:55,093 NetUtil.py:93 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2017-08-22 13:33:55,093 NetUtil.py:94 - SSLError: Failed to connect. Please check openssl library versions. Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.WARNING 2017-08-22 13:33:55,095 NetUtil.py:121 - Server at https://F.Q.D.N:8440 is not reachable, sleeping for 10 seconds...

See attached RHEL Bugzilla

Can you temporarily disable it

Option 1. Set Up Two-Way SSL Between Ambari Server and Ambari Agents

1. On the Ambari Server host, open /etc/ambari-server/conf/ambari.properties with a text editor.
2. Add the following property:

   security.server.two_way_ssl = true

3. Start or restart the Ambari Server.

   ambari-server restart

The Agent certificates are downloaded automatically during Agent Registration.

Option 2. Disable HTTPS

If SSL is enabled you will get the option to disable when you run the below tool

# ambari-server setup-security
Using python  /usr/bin/python2.6
Security setup options...
===========================================================================
Choose one of the following options:
  [1] Enable HTTPS for Ambari server.
  [2] Encrypt passwords stored in ambari.properties file.
  [3] Setup Ambari kerberos JAAS configuration.
  [4] Setup truststore.
  [5] Import certificate to truststore.
===========================================================================

After disable and default the port number to 8080. Let me know

[#3]

Hi @Geoffrey Shelton Okot @Jay SenSharma

I am back looking at this.

Ambari now at 2.5.2.0.

There is now a redhat-ppc7 public repo suggested by Ambari

Which configures HDP.repo and HDP-UTILS.repo in /etc/yum.repos.d.. The contents of which are:

[root@hdplab80 yum.repos.d]# cat HDP.repo
[HDP-2.6]
name=HDP-2.6
baseurl=http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0

path=/
enabled=1
gpgcheck=0

[root@hdplab80 yum.repos.d]# cat HDP-UTILS.repo
[HDP-UTILS-1.1.0.21]
name=HDP-UTILS-1.1.0.21
baseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ppc64le

path=/
enabled=1

However, when Ambari is attempt to perform the HDP software install, we are back to seeing problems with it being able to use the repos.

2017-09-04 15:35:05,569 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2017-09-04 15:35:05,590 - Installing package hdp-select ('/usr/bin/yum -d 0 -e 0 -y install hdp-select')
2017-09-04 15:42:11,982 - Execution of '/usr/bin/yum -d 0 -e 0 -y install hdp-select' returned 1.  One of the configured repositories failed (HDP-2.6),
 and yum doesn't have enough cached data to continue.## output cut ##failure: repodata/repomd.xml from HDP-2.6: [Errno 256] No more mirrors to try.
http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to public-repo-1.hortonworks.com:80; Operation now in progress"
http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to public-repo-1.hortonworks.com:80; Operation now in progress"
http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to public-repo-1.hortonworks.com:80; Operation now in progress"
http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to public-repo-1.hortonworks.com:80; Operation now in progress"
http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to public-repo-1.hortonworks.com:80; Operation now in progress"
http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to public-repo-1.hortonworks.com:80; Operation now in progress"
http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to public-repo-1.hortonworks.com:80; Operation now in progress"
http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to public-repo-1.hortonworks.com:80; Operation now in progress"
http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to public-repo-1.hortonworks.com:80; Operation now in progress"
http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to public-repo-1.hortonworks.com:80; Operation now in progress"
2017-09-04 15:42:11,982 - Failed to install package hdp-select. Executing '/usr/bin/yum clean metadata'
2017-09-04 15:44:19,568 - Retrying to install package hdp-select after 30 seconds
Command aborted. Reason: 'Server considered task failed and automatically aborted it'

But if I drop to the CLI and try to manually instally hdp-select, it works:

[root@hdplab81 yum.repos.d]# yum install hdp-select
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
rhel-7-for-power-le-rpms                                                                    | 2.3 kB  00:00:00
Resolving Dependencies
--> Running transaction check
---> Package hdp-select.noarch 0:2.6.2.0-205 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================
 Package                     Arch                    Version                        Repository                Size
===================================================================================================================
Installing:
 hdp-select                  noarch                  2.6.2.0-205                    HDP-2.6                   11 k

Transaction Summary
===================================================================================================================
Install  1 Package

Total download size: 11 k
Installed size: 29 k
Is this ok [y/d/N]:

I don't see why Ambari is failing to use a repo which is clearly working.

Any help much appreciated (again). I am out of ideas.

Thanks,

John

[#4]

I Have checked I can use wget to get the repomd.xml files for both repos. It works.

[root@hdplab80 tmp]# wget http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml
--2017-09-04 16:20:31--  http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml
Connecting to x.x.x.x:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2989 (2.9K) [text/xml]
Saving to: ‘repomd.xml’

100%[=========================================================================>] 2,989       --.-K/s   in 0s

2017-09-04 16:20:31 (898 MB/s) - ‘repomd.xml’ saved [2989/2989]


[root@hdplab80 tmp]# rm repomd.xml
rm: remove regular file ‘repomd.xml’? y

[root@hdplab80 tmp]# wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ppc64le/repodata/repomd.xml
--2017-09-04 16:20:53--  http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ppc64le/repodata/repomd.xml
Connecting to x.x.x.x:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2583 (2.5K) [text/xml]
Saving to: ‘repomd.xml’

100%[=========================================================================>] 2,583       --.-K/s   in 0s

2017-09-04 16:20:53 (822 MB/s) - ‘repomd.xml’ saved [2583/2583]

[#5]

@John Wright

If only ambari is not able to get the repomd.xml means it might be sending request via some proxy.

Better to check if ambari is mistakenly using any proxy?

Usually we define the proxyHost port information for ambari inside the file. SO it will be worth checking this file if someone has added a proxy setting there by mistake.

# grep 'proxy'  /var/lib/ambari-server/ambari-env.sh

.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-administration/content/ch_setting...

Also

# grep 'proxy'   /etc/yum.conf

[#6]

@John Wright

Looks like from ambari server the HWX repo is not accessible:

http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to public-repo-1.hortonworks.com:80; Operation now in progress"

.

That can happen if the ambari server host is configured to pass the requests to external resources via some proxy server.

So can you please check if you have any proxy configured of if the request need to pass through a proxy on your network?

Please check the following files for proxy configurations:

# grep 'proxy'   /etc/yum.conf
# grep 'proxy'  ~/.bash_profile
# grep 'proxy'  ~/.profile
# grep 'proxy'  /var/lib/ambari-server/ambari-env.sh

.

export http_proxy=http://localhost:80

.

Are you able to do "wget" or "curl" from ambari server host to the repo?

wget http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml

.

[#7]

@Jay SenSharma

I am convinced this is something to do with Ambari not going via the proxy.

I can use wget to retrieve files from both repo locations (I have blanked IP address with x.x.x.x):

[root@hdplab80 tmp]# wget http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml
--2017-09-04 16:20:31--  http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.2.0/repodata/repomd.xml
Connecting to x.x.x.x:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2989 (2.9K) [text/xml]
Saving to: ‘repomd.xml’

100%[=========================================================================>] 2,989       --.-K/s   in 0s

2017-09-04 16:20:31 (898 MB/s) - ‘repomd.xml’ saved [2989/2989]


[root@hdplab80 tmp]# rm repomd.xml
rm: remove regular file ‘repomd.xml’? y

[root@hdplab80 tmp]# wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ppc64le/repodata/repomd.xml
--2017-09-04 16:20:53--  http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ppc64le/repodata/repomd.xml
Connecting to x.x.x.x:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2583 (2.5K) [text/xml]
Saving to: ‘repomd.xml’

100%[=========================================================================>] 2,583       --.-K/s   in 0s

2017-09-04 16:20:53 (822 MB/s) - ‘repomd.xml’ saved [2583/2583]

Here's the output to the greps:

[root@hdplab80 ~]# grep 'proxy'   /etc/yum.conf
[root@hdplab80 ~]# grep 'proxy'  ~/.bash_profile
[root@hdplab80 ~]# grep 'proxy'  ~/.profile
grep: /root/.profile: No such file or directory
[root@hdplab80 ~]#  grep 'proxy'  /var/lib/ambari-server/ambari-env.sh
export AMBARI_JVM_ARGS="$AMBARI_JVM_ARGS -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dhttp.proxyHost=x.x.x.x -Dhttp.proxyPort=3128"

I have my /etc/environment file configured as such:

[root@hdplab80 ~]# cat /etc/environment
export ftp_proxy=http://x.x.x.x:3128/
export http_proxy=http://x.x.x.x:3128/
export ntp_proxy=http://x.x.x.x:3128/
export https_proxy=http://x.x.x.x:3128/

I have also run all those exports from the CLI.

The export http_proxy=http://localhost:80

Should I run that? Won't it break my proxy connection out?

[#8 · Accepted solution]

@John Wright

Based on your output it looks like you have configured proxy.

So i guess in that case you should also set proxy inside your "/etc/yum.conf" file something like following so that the yum will also start using the same proxy.

# cat /etc/yum.conf 
[main] 
proxy=http://x.x.x.x:3128

cachedir=/var/cache/yum/$basearch/$releasever 
keepcache=0 
debuglevel=2 
logfile=/var/log/yum.log 
exactarch=1 
obsoletes=1 
gpgcheck=1 
plugins=1 
installonly_limit=3 

.

The perform a yum clean and then try again.

# yum clean all

.

[#9]

@Jay SenSharma

We are getting closer...

It now appears to be getting further BUT there's a problem with:

2017-09-04 17:21:49,354 - Execution of '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_6_2_0_205-hdfs' returned 1. Error: Package: hadoop_2_6_2_0_205-hdfs-2.7.3.2.6.2.0-205.ppc64le (HDP-2.6)
           Requires: libtirpc-devel

I have this error on most of the nodes. Crucially, I also see this error from the CLI.

Looks like this has been seen before (although this is an x86 reference):

https://community.hortonworks.com/questions/96763/hdp-26-ambari-install-fails-on-rhel-7-on-libtirpc....

So trying to figure out how I either get this package or which ppc64 repo I am supposed to add from RHEL.

[#10]

I can see you have already answered similar questions to this on other threads. I am reading them now.

[#11]

@John Wright

As the "Requires: libtirpc-devel" issue is slightly different from the proxy issue.

So i will suggest closing this thread by marking the thread as "Accepted" , So that other users will not get confused with other issues and can directly browse the answer quickly.