Adding HDFS service in HDP 2.1, asking more dependent packages as below

[#1]

Is there any other way to come out of it, these rpm around 100+, need to install all nodes it consume time.

Hence any other alternate way for this.

result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install hadoop' returned 1. Error Downloading Packages: mesa-private-llvm-3.6.2-1.el6.x86_64: failure: Packages/mesa-private-llvm-3.6.2-1.el6.x86_64.rpm from base: [Errno 256] No more mirrors to try. mesa-libGL-11.0.7-4.el6.x86_64: failure: Packages/mesa-libGL-11.0.7-4.el6.x86_64.rpm from base: [Errno 256] No more mirrors to try. libgudev1-147-2.73.el6.x86_64: failure: Packages/libgudev1-147-2.73.el6.x86_64.rpm from base: [Errno 256] No more mirrors to try. 1:phonon-backend-gstreamer-4.6.2-28.el6_5.x86_64: failure: Packages/phonon-backend-gstreamer-4.6.2-28.el6_5.x86_64.rpm from base: [Errno 256] No more mirrors to try. cvs-1.11.23-16.el6.x86_64: failure: Packages/cvs-1.11.23-16.el6.x86_64.rpm from base: [Errno 256] No more mirrors to try. redhat-lsb-compat-4.0-7.el6.centos.x86_64: failure: Packages/redhat-lsb-compat-4.0-7.el6.centos.x86_64.rpm from base: [Errno 256] No more mirrors to try. perl-ExtUtils-MakeMaker-6.55-141.el6_7.1.x86_64: failure: Packages/perl-ExtUtils-MakeMaker-6.55-141.el6_7.1.x86_64.rpm from base: [Errno 256] No more mirrors to try. gettext-0.17-18.el6.x86_64: failure: Packages/gettext-0.17-18.el6.x86_64.rpm from base: [Errno 256] No more mirrors to try. mailx-12.4-8.el6_6.x86_64: failure: Packages/mailx-12.4-8.el6_6.x86_64.rpm from base: [Errno 256] No more mirrors to try.

[#2]

You can install a local repo on one of your nodes (Ambari node, or another master node), and configure Ambari to use your local repo instead of downloading all packages from Internet. You can find details here

[#3]

Mine Concern is Why these rpm packages dependency it asking while installing

1) During Installation of Ambari WI it has asked for packages

2) Again when i Adding HDFS server only it asking again ,

If i go to next level of installation of YARN,Quoram ..etc i may ask dependent packages. they could have been included in HDP ?

[#4]

If you install a local repo, as per my answer above, you will have all required packages to install Ambari and all HDP components there, and you won't be asked again for any other ones.

[#5]

@zkfs

In addition to answers given.

1) During Installation of Ambari WI it has asked for packages

Ambari is completely different RPM in different repo.

If you want to check dependency for any rpm then you can run below commands:

example:

[root@kknode3 ~]# rpm -qR hadoop_2_4_2_0_258-hdfs-2.7.1.2.4.2.0-258.el6.x86_64
hadoop_2_4_2_0_258 = 2.7.1.2.4.2.0-258.el6
bigtop-jsvc
/bin/sh
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(CompressedFileNames) <= 3.0.4-1
/bin/bash
/bin/sh
/usr/bin/env
rpmlib(PayloadIsXz) <= 5.2-1

example 2:

[root@kknode3 ~]# rpm -qR hadoop_2_4_2_0_258-client-2.7.1.2.4.2.0-258.el6.x86_64
hadoop_2_4_2_0_258 = 2.7.1.2.4.2.0-258.el6
hadoop_2_4_2_0_258-hdfs = 2.7.1.2.4.2.0-258.el6
hadoop_2_4_2_0_258-yarn = 2.7.1.2.4.2.0-258.el6
hadoop_2_4_2_0_258-mapreduce = 2.7.1.2.4.2.0-258.el6
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadIsXz) <= 5.2-1 

Same thing using yum command:

[root@kknode3 ~]# yum deplist hadoop_2_4_2_0_258-client-2.7.1.2.4.2.0-258.el6.x86_64
Loaded plugins: fastestmirror
Finding dependencies:
Loading mirror speeds from cached hostfile
HDP-2.4                                                                                                                                                                        | 2.9 kB     00:00
HDP-UTILS-1.1.0.20                                                                                                                                                             | 2.9 kB     00:00
Updates-ambari-2.2.2.0                                                                                                                                                         | 2.9 kB     00:00
base                                                                                                                                                                           | 3.7 kB     00:00
epel                                                                                                                                                                           | 4.3 kB     00:00
epel/primary_db                                                                                                                                                                | 5.9 MB     00:00
extras                                                                                                                                                                         | 3.4 kB     00:00
updates                                                                                                                                                                        | 3.4 kB     00:00
package: hadoop_2_4_2_0_258-client.x86_64 2.7.1.2.4.2.0-258.el6
  dependency: hadoop_2_4_2_0_258 = 2.7.1.2.4.2.0-258.el6
   provider: hadoop_2_4_2_0_258.x86_64 2.7.1.2.4.2.0-258.el6
  dependency: hadoop_2_4_2_0_258-hdfs = 2.7.1.2.4.2.0-258.el6
   provider: hadoop_2_4_2_0_258-hdfs.x86_64 2.7.1.2.4.2.0-258.el6
  dependency: hadoop_2_4_2_0_258-yarn = 2.7.1.2.4.2.0-258.el6
   provider: hadoop_2_4_2_0_258-yarn.x86_64 2.7.1.2.4.2.0-258.el6
  dependency: hadoop_2_4_2_0_258-mapreduce = 2.7.1.2.4.2.0-258.el6
   provider: hadoop_2_4_2_0_258-mapreduce.x86_64 2.7.1.2.4.2.0-258.el6
[root@kknode3 ~]#

Hope this information helps.

Please do let us know if anything is un-clear. HCC is always there for you 🙂

[#6]

@zkfs

Anbari is a management service that knows how to install and manage each of the HDP components. Ambari itself does not contain the packages or software for HDP. It uses repositories to fetch those packages as needed.

Each component of HDP (YARN, HDFS, Ranger, Knox, Hive, etc) is comprised of a number of RPM packages. When you use Ambari to install HDFS, it has to get all of the required packages for HDFS. If you were to install HDP without Ambari, you would still have to install all of those packages by hand.

This link provides the packages needed for each HDP component: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_installing_manually_book/content/ch_getti...

[#7]

Thanks for your reply,

Yep i understand the your update, yes i need install the all RPM packages, even if install without using Ambari.

I am correcting my question for this and update it

[#8 · Accepted solution]

Thanks you all for your reply. Now added the HDFS service to cluster

below Error While Deploying the HDFS service

1) Connected the Internet and downloaded, installed the dependent package

2) Proceeded with HDFS Deployment, it went fine.

Steps followed as below.

1)

• #yum install libgudev1-147-2.73.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install 1:phonon-backend-gstreamer-4.6.2-28.el6_5.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install cvs-1.11.23-16.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install redhat-lsb-compat-4.0-7.el6.centos.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install perl-ExtUtils-MakeMaker-6.55-141.el6_7.1.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install gettext-0.17-18.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install mailx-12.4-8.el6_6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install qt3-3.3.8b-30.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install ghostscript-fonts-5.50-23.2.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install gstreamer-0.10.29-1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install 1:qt-sqlite-4.6.2-28.el6_5.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install lcms-libs-1.19-1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install iso-codes-3.16-2.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install redhat-lsb-graphics-4.0-7.el6.centos.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install mesa-dri-drivers-11.0.7-4.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install time-1.7-38.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install xz-4.999.9-0.5.beta.20091007git.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install libXv-1.0.9-2.1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install xz-lzma-compat-4.999.9-0.5.beta.20091007git.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install poppler-utils-0.12.4-10.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install ghostscript-8.70-21.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install redhat-lsb-core-4.0-7.el6.centos.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install nc --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install libmng-1.0.10-4.1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install xml-common-0.6.3-33.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install mesa-dri1-drivers-7.11-8.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install perl-Test-Simple-0.92-141.el6_7.1.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install poppler-data-0.4.0-1.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install man-1.6f-32.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install libvisual-0.4.0-10.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install 1:qt-4.6.2-28.el6_5.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install tmpwatch-2.9.16-6.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install gdbm-devel-1.8.0-39.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install 1:perl-ExtUtils-ParseXS-2.2003.0-141.el6_7.1.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install urw-fonts-2.4-11.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install 1:qt-x11-4.6.2-28.el6_5.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install mesa-libGLU-11.0.7-4.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install redhat-lsb-4.0-7.el6.centos.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install perl-Test-Harness-3.17-141.el6_7.1.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install openjpeg-libs-1.3-11.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install libXxf86vm-1.1.3-2.1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install foomatic-4.0.4-5.el6_7.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install foomatic-db-ppds-4.0-7.20091126.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install 1:libtheora-1.1.0-2.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install gstreamer-plugins-base-0.10.29-2.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install 1:cups-1.4.2-74.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install gstreamer-tools-0.10.29-1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install redhat-lsb-printing-4.0-7.el6.centos.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install bc-1.06.95-1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install perl-CGI-3.51-141.el6_7.1.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install liboil-0.3.16-4.1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install mesa-dri-filesystem-11.0.7-4.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install 4:perl-devel-5.10.1-141.el6_7.1.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install libXt-1.1.4-6.1.el6.x86_64: failure --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install cdparanoia-libs-10.2-5.1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install db4-cxx-4.7.25-20.el6_7.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install at-3.1.10-48.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install db4-devel-4.7.25-20.el6_7.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install foomatic-db-filesystem-4.0-7.20091126.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install patch-2.6-6.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install pax-3.4-10.1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install poppler-0.12.4-10.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install foomatic-db-4.0-7.20091126.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install ed-1.1-3.3.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install portreserve-0.0.4-11.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install snappy-1.1.0-1.el6.x86_64 --downloaddir=/etc/yum.repos.d/rpm_packag
• #yum install snappy-devel --downloaddir=/etc/yum.repos.d/rpm_packag
• #
• #yum install poppler-data-0.4.0-1.el6.noarch --downloaddir=/etc/yum.repos.d/rpm_packag
• #
• #
• #After above still some are missing . installed one by one
• #
• #
• #yum install urw-fonts-2.4-11.el6.noarch
• #
• #yum install xz-lzma-compat-4.999.9-0.5.beta.20091007git.el6.x86_64
• #
• #yum install redhat-lsb-graphics-4.0-7.el6.centos.x86_64
• #yum install xz-4.999.9-0.5.beta.20091007git.el6.x86_64
• #yum install ghostscript-8.70-21.el6.x86_64
• #yum install redhat-lsb-core-4.0-7.el6.centos.x86_64
• #yum install nc-1.84-24.el6.x86_64
• #yum install man-1.6f-32.el6.x86_64
• #yum install tmpwatch-2.9.16-6.el6.x86_64
• #yum install poppler-utils-0.12.4-10.el6.x86_64
• #yum install pax-3.4-10.1.el6.x86_64
• #yum install openjpeg-libs-1.3-11.el6.x86_64
• #yum install foomatic-4.0.4-5.el6_7.x86_64
• #yum install foomatic-db-ppds-4.0-7.20091126.el6.noarch
• #yum install redhat-lsb-printing-4.0-7.el6.centos.x86_64
• #yum install foomatic-db-4.0-7.20091126.el6.noarch
• #yum install bc-1.06.95-1.el6.x86_64
• #yum install perl-CGI-3.51-141.el6_7.1.x86_64
• #yum install ed-1.1-3.3.el6.x86_64
• #yum install libXt-1.1.4-6.1.el6.x86_64
• #yum install at-3.1.10-48.el6.x86_64
• #yum install time-1.7-38.el6.x86_64
• #yum install foomatic-db-filesystem-4.0-7.20091126.el6.noarch
• #yum install patch-2.6-6.el6.x86_64
• #yum install redhat-lsb-4.0-7.el6.centos.x86_64
• #yum install poppler-0.12.4-10.el6.x86_64
• #yum install 1:cups-1.4.2-74.el6.x86_64
• #yum install redhat-lsb-compat-4.0-7.el6.centos.x86_64
• #yum install portreserve-0.0.4-11.el6.x86_64
• #cd /etc/yum.repos.d/rpm_packag

#rpm -Uvh *.rpm

#cd /etc/yum.repos.d/rpm_packag

#rpm -Uvh *.rpm

One of installed packages example

[root@centos ~]# yum install foomatic-4.0.4-5.el6_7.x86_64 Loaded plugins: fastestmirror Setting up Install Process Loading mirror speeds from cached hostfile * base: buaya.klas.or.id * extras: centos.exabytes.com.my * updates: buaya.klas.or.id Resolving Dependencies --> Running transaction check ---> Package foomatic.x86_64 0:4.0.4-5.el6_7 will be installed --> Processing Dependency: foomatic-db for package: foomatic-4.0.4-5.el6_7.x86_64 --> Processing Dependency: cups for package: foomatic-4.0.4-5.el6_7.x86_64 --> Running transaction check ---> Package cups.x86_64 1:1.4.2-74.el6 will be installed --> Processing Dependency: portreserve for package: 1:cups-1.4.2-74.el6.x86_64 ---> Package foomatic-db.noarch 0:4.0-7.20091126.el6 will be installed --> Processing Dependency: foomatic-db-ppds = 4.0-7.20091126.el6 for package: foomatic-db-4.0-7.20091126.el6.noarch --> Processing Dependency: foomatic-db-filesystem = 4.0-7.20091126.el6 for package: foomatic-db-4.0-7.20091126.el6.noarch --> Running transaction check ---> Package foomatic-db-filesystem.noarch 0:4.0-7.20091126.el6 will be installed ---> Package foomatic-db-ppds.noarch 0:4.0-7.20091126.el6 will be installed ---> Package portreserve.x86_64 0:0.0.4-11.el6 will be installed --> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================================ Package Arch Version Repository Size ================================================================================================================================================================================================ Installing: foomatic x86_64 4.0.4-5.el6_7 base 251 k Installing for dependencies: cups x86_64 1:1.4.2-74.el6 base 2.3 M foomatic-db noarch 4.0-7.20091126.el6 base 980 k foomatic-db-filesystem noarch 4.0-7.20091126.el6 base 4.4 k foomatic-db-ppds noarch 4.0-7.20091126.el6 base 19 M portreserve x86_64 0.0.4-11.el6 base 23 k

Transaction Summary ================================================================================================================================================================================================ Install 6 Package(s)

Total download size: 23 M Installed size: 44 M Is this ok [y/N]: y

Downloading Packages: http://buaya.klas.or.id/centos/6.8/os/x86_64/Packages/cups-1.4.2-74.el6.x86_64.rpm: [Errno 12] Timeout on http://buaya.klas.or.id/centos/6.8/os/x86_64/Packages/cups-1.4.2-74.el6.x86_64.rpm: (28, 'connect() timed out!') Trying other mirror. (1/6): cups-1.4.2-74.el6.x86_64.rpm | 2.3 MB 00:01 (2/6): foomatic-4.0.4-5.el6_7.x86_64.rpm | 251 kB 00:00 (3/6): foomatic-db-4.0-7.20091126.el6.noarch.rpm | 980 kB 00:00 (4/6): foomatic-db-filesystem-4.0-7.20091126.el6.noarch.rpm | 4.4 kB 00:00 (5/6): foomatic-db-ppds-4.0-7.20091126.el6.noarch.rpm | 19 MB 00:09 (6/6): portreserve-0.0.4-11.el6.x86_64.rpm | 23 kB 00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Total 558 kB/s | 23 MB 00:41

Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : foomatic-db-filesystem-4.0-7.20091126.el6.noarch 1/6 Installing : portreserve-0.0.4-11.el6.x86_64 2/6 Installing : 1:cups-1.4.2-74.el6.x86_64 3/6 Installing : foomatic-db-ppds-4.0-7.20091126.el6.noarch 4/6 Installing : foomatic-db-4.0-7.20091126.el6.noarch 5/6 Installing : foomatic-4.0.4-5.el6_7.x86_64 6/6 Verifying : foomatic-db-filesystem-4.0-7.20091126.el6.noarch 1/6 Verifying : foomatic-4.0.4-5.el6_7.x86_64 2/6 Verifying : portreserve-0.0.4-11.el6.x86_64 3/6 Verifying : foomatic-db-4.0-7.20091126.el6.noarch 4/6 Verifying : foomatic-db-ppds-4.0-7.20091126.el6.noarch 5/6 Verifying : 1:cups-1.4.2-74.el6.x86_64 6/6

Installed: foomatic.x86_64 0:4.0.4-5.el6_7

Dependency Installed: cups.x86_64 1:1.4.2-74.el6 foomatic-db.noarch 0:4.0-7.20091126.el6 foomatic-db-filesystem.noarch 0:4.0-7.20091126.el6 foomatic-db-ppds.noarch 0:4.0-7.20091126.el6 portreserve.x86_64 0:0.0.4-11.el6

Complete! [root@centos ~]#

[#9]

@zkfs

create a OS repo so it will automatically install all dependency to install the HDP package or liberary.