Install HDFS client without using yum on HDF?

[#1]

Hello,

I was following this community post to install Hadoop client without yum. But, with latest hdp repo 2.5.3.0 I am getting the below exception. I want to install HDFS client on our HDF cluster to access HDP cluster hdfs.

Any suggestions on approaches or how to do it?

I installed the repo using below url's:

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-hdfs-2...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-2.7.3....

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-libhdf...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-yarn-2...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-mapred...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-client...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/zookeeper/zookeeper_2_5_3_0_37-...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/bigtop-jsvc/bigtop-jsvc-1.0.15-...

Install command -->

rpm -Uvh hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-hdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-client-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-mapreduce-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-libhdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-yarn-2.7.3.2.5.3.0-37.el6.x86_64.rpm zookeeper_2_5_3_0_37-3.4.6.2.5.3.0-37.el6.noarch.rpm bigtop-jsvc-1.0.15-37.el6.x86_64.rpm

error:

Failed dependencies:

ranger_2_5_3_0_37-hdfs-plugin is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64

ranger_2_5_3_0_37-yarn-plugin is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64

hdp-select >= 2.5.3.0-37 is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64

spark_2_5_3_0_37-yarn-shuffle is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64

spark2_2_5_3_0_37-yarn-shuffle is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64

nc is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64

hdp-select >= 2.5.3.0-37 is needed by zookeeper_2_5_3_0_37-3.4.6.2.5.3.0-37.el6.noarch

Thank you in advance!

[#2]

You can download tar file from given link and you will get all rpm dependencies.

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.0.0/HDP-2.5.0.0-centos7-rpm.tar.gz

Download dependancies using from below links: http://public-repo-

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-mapred...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/ranger/ranger_2_5_3_0_37-hdfs-p...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/ranger/ranger_2_5_3_0_37-yarn-p...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hdp-select/hdp-select-2.5.3.0-3...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/spark/spark_2_5_3_0_37-yarn-shu...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/spark2/spark2_2_5_3_0_37-yarn-s...

ftp://bo.mirror.garr.it/1/slc/centos/7.1.1503/os/x86_64/Packages/nmap-ncat-6.40-4.el7.x86_64.rpm

Now you can install using below command:

rpm -Uvh hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-hdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-client-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-mapreduce-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-libhdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-yarn-2.7.3.2.5.3.0-37.el6.x86_64.rpm zookeeper_2_5_3_0_37-3.4.6.2.5.3.0-37.el6.noarch.rpm bigtop-jsvc-1.0.15-37.el6.x86_64.rpm ranger_2_5_3_0_37-hdfs-plugin-0.6.0.2.5.3.0-37.el6.x86_64.rpm ranger_2_5_3_0_37-yarn-plugin-0.6.0.2.5.3.0-37.el6.x86_64.rpmhdp-select-2.5.3.0-37.el6.noarch.rpm spark_2_5_3_0_37-yarn-shuffle-1.6.2.2.5.3.0-37.el6.noarch.rpm spark2_2_5_3_0_37-yarn-shuffle-2.0.0.2.5.3.0-37.el6.noarch.rpm nmap-ncat-6.40-4.el7.x86_64.rpm

OR

Download apache hadoop-2.8.0 tar file and make all configurations as per your Cluster. so that you can access your hadoop cluster from this client.

https://archive.apache.org/dist/hadoop/core/hadoop-2.8.0/hadoop-2.8.0.tar.gz

[#3]

Thank you @arjun more. But, this is not working on a HDF cluster. I took another approach and install Apache client manually.

[#4 · Accepted solution]

The above approach will not work as it requires 'hdp-select'. According to HW, HDP is not allowed on a cluster where HDF is installed and vice-versa. An adequate solution is, to install Apache Hadoop (version same as your HDP).

Steps I followed:

1. Setup Java, A running HDF cluster will not require it but no harm in cross check.
2. Download Hadoop from Apache Mirror, Unpack it and move it to the desired location.
3. Set HADOOP_HOME and HADOOP_CONF_DIR in /etc/profile. By default conf. is set to HADOOP_HOME/etc/hadoop. It is good to keep your configs separate. I created HADOOP_HOME/conf.
4. Important step: Copy existing hdp confs (/etc/hadoop/conf) to HADOOP_HOME/conf.
5. Do not format or start Hadoop as we are connecting to an existing cluster.
6. Last step, set HADOOP_HOME/bin in your user profile file (usually .bash_profile/.profile).
7. That's it, try 'hadoop' or 'hdfs' command.

I hope this will help somebody in future!

[#5]

Yes @Shashant Panwar, but this both were worked for me and i already mention this manual apache hadoop client installation approach in above comment.

[#6]

@arjun more I did not notice the last line. Yes, I took that approach.