Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Install HDFS client without using yum on HDF?

Solved Go to solution

Install HDFS client without using yum on HDF?

Rising Star

Hello,

I was following this community post to install Hadoop client without yum. But, with latest hdp repo 2.5.3.0 I am getting the below exception. I want to install HDFS client on our HDF cluster to access HDP cluster hdfs.

Any suggestions on approaches or how to do it?

I installed the repo using below url's:

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-hdfs-2...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-2.7.3....

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-libhdf...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-yarn-2...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-mapred...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-client...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/zookeeper/zookeeper_2_5_3_0_37-...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/bigtop-jsvc/bigtop-jsvc-1.0.15-...

Install command -->

rpm -Uvh hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-hdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-client-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-mapreduce-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-libhdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-yarn-2.7.3.2.5.3.0-37.el6.x86_64.rpm zookeeper_2_5_3_0_37-3.4.6.2.5.3.0-37.el6.noarch.rpm bigtop-jsvc-1.0.15-37.el6.x86_64.rpm

error:

Failed dependencies:

ranger_2_5_3_0_37-hdfs-plugin is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64

ranger_2_5_3_0_37-yarn-plugin is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64

hdp-select >= 2.5.3.0-37 is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64

spark_2_5_3_0_37-yarn-shuffle is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64

spark2_2_5_3_0_37-yarn-shuffle is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64

nc is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64

hdp-select >= 2.5.3.0-37 is needed by zookeeper_2_5_3_0_37-3.4.6.2.5.3.0-37.el6.noarch

Thank you in advance!

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Install HDFS client without using yum on HDF?

Rising Star

The above approach will not work as it requires 'hdp-select'. According to HW, HDP is not allowed on a cluster where HDF is installed and vice-versa. An adequate solution is, to install Apache Hadoop (version same as your HDP).

Steps I followed:

  1. Setup Java, A running HDF cluster will not require it but no harm in cross check.
  2. Download Hadoop from Apache Mirror, Unpack it and move it to the desired location.
  3. Set HADOOP_HOME and HADOOP_CONF_DIR in /etc/profile. By default conf. is set to HADOOP_HOME/etc/hadoop. It is good to keep your configs separate. I created HADOOP_HOME/conf.
  4. Important step: Copy existing hdp confs (/etc/hadoop/conf) to HADOOP_HOME/conf.
  5. Do not format or start Hadoop as we are connecting to an existing cluster.
  6. Last step, set HADOOP_HOME/bin in your user profile file (usually .bash_profile/.profile).
  7. That's it, try 'hadoop' or 'hdfs' command.

I hope this will help somebody in future!

5 REPLIES 5

Re: Install HDFS client without using yum on HDF?

New Contributor

You can download tar file from given link and you will get all rpm dependencies.

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.0.0/HDP-2.5.0.0-centos7-rpm.tar.gz

Download dependancies using from below links: http://public-repo-

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-mapred...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/ranger/ranger_2_5_3_0_37-hdfs-p...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/ranger/ranger_2_5_3_0_37-yarn-p...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hdp-select/hdp-select-2.5.3.0-3...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/spark/spark_2_5_3_0_37-yarn-shu...

http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/spark2/spark2_2_5_3_0_37-yarn-s...

ftp://bo.mirror.garr.it/1/slc/centos/7.1.1503/os/x86_64/Packages/nmap-ncat-6.40-4.el7.x86_64.rpm

Now you can install using below command:

rpm -Uvh hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-hdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-client-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-mapreduce-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-libhdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-yarn-2.7.3.2.5.3.0-37.el6.x86_64.rpm zookeeper_2_5_3_0_37-3.4.6.2.5.3.0-37.el6.noarch.rpm bigtop-jsvc-1.0.15-37.el6.x86_64.rpm ranger_2_5_3_0_37-hdfs-plugin-0.6.0.2.5.3.0-37.el6.x86_64.rpm ranger_2_5_3_0_37-yarn-plugin-0.6.0.2.5.3.0-37.el6.x86_64.rpmhdp-select-2.5.3.0-37.el6.noarch.rpm spark_2_5_3_0_37-yarn-shuffle-1.6.2.2.5.3.0-37.el6.noarch.rpm spark2_2_5_3_0_37-yarn-shuffle-2.0.0.2.5.3.0-37.el6.noarch.rpm nmap-ncat-6.40-4.el7.x86_64.rpm

OR

Download apache hadoop-2.8.0 tar file and make all configurations as per your Cluster. so that you can access your hadoop cluster from this client.

https://archive.apache.org/dist/hadoop/core/hadoop-2.8.0/hadoop-2.8.0.tar.gz

Re: Install HDFS client without using yum on HDF?

Rising Star

Thank you @arjun more. But, this is not working on a HDF cluster. I took another approach and install Apache client manually.

Re: Install HDFS client without using yum on HDF?

Rising Star

The above approach will not work as it requires 'hdp-select'. According to HW, HDP is not allowed on a cluster where HDF is installed and vice-versa. An adequate solution is, to install Apache Hadoop (version same as your HDP).

Steps I followed:

  1. Setup Java, A running HDF cluster will not require it but no harm in cross check.
  2. Download Hadoop from Apache Mirror, Unpack it and move it to the desired location.
  3. Set HADOOP_HOME and HADOOP_CONF_DIR in /etc/profile. By default conf. is set to HADOOP_HOME/etc/hadoop. It is good to keep your configs separate. I created HADOOP_HOME/conf.
  4. Important step: Copy existing hdp confs (/etc/hadoop/conf) to HADOOP_HOME/conf.
  5. Do not format or start Hadoop as we are connecting to an existing cluster.
  6. Last step, set HADOOP_HOME/bin in your user profile file (usually .bash_profile/.profile).
  7. That's it, try 'hadoop' or 'hdfs' command.

I hope this will help somebody in future!

Re: Install HDFS client without using yum on HDF?

New Contributor

Yes @Shashant Panwar, but this both were worked for me and i already mention this manual apache hadoop client installation approach in above comment.

Re: Install HDFS client without using yum on HDF?

Rising Star

@arjun more I did not notice the last line. Yes, I took that approach.

Don't have an account?
Coming from Hortonworks? Activate your account here