question Re: Configure hadoop-client tools to access hdfs from external computer in Archives of Support Questions (Read Only)

Configure hadoop-client tools to access hdfs from external computer

epowell — Fri, 16 Jun 2017 19:33:50 GMT

I would like to be able to perform hdfs commands from a computer that is NOT actually part of the cloudera cluster.

For example, performing simple put/get operations or

hdfs dfs -ls /my/dir

I have installed the correct binaries, I think. I found from CM that I was using CDH4.7.1. So, I installed (sudo apt-get install hadoop-client) the binaries from here.

If I run:

hdfs dfs -ls /

I get:

Error: JAVA_HOME is not set and could not be found.

I feel that this might just be the beginning of a long tinkering and configuring process and I unfortunately know nothing about java. I do, however, know the IPs of the namenodes on my cluster and have access to all admin rights from beginning to end.

Can someone help me get things configured?

P.S. In case it wasn't clear. I can perform all desired functionality on nodes that are part of the cluster. I just want to do something similar from my development environment.

Re: Configure hadoop-client tools to access hdfs from external computer

epowell — Fri, 16 Jun 2017 21:42:21 GMT

I should refine my question. Part of what prompted this is that I noticed on the nodes of the cluster that $JAVA_HOME is not defined either.

That makes me think that there might be certain configuration files on the cluster nodes that maybe I can copy over to my development environment.

Re: Configure hadoop-client tools to access hdfs from external computer

mbigelow — Fri, 16 Jun 2017 23:24:36 GMT

It is part of the magic that happens when you install a gateway roles on a nodes using CM. It installs the binaries, download the client configs, sets env vars in such a way that no users on that nodes need to do it. It just works. I have not dug into it to determine how. I expected that it is wrapped up in scripts somewhere that get called when using the clients.

Without it, you will need to do it all yourself. And JAVA_HOME isn't the only one, HADOOP_HOME, HADOOP_CONF, etc.

The Hadoop docs should have some information on it.

Or... you can add the machine in question to CM, add the gateway role, and then remove it from the cluster. You shouldn't need to uninstall the gateway role to remove it from the cluster. No license is required for the gateway roles either.

Re: Configure hadoop-client tools to access hdfs from external computer

epowell — Fri, 16 Jun 2017 23:28:53 GMT

Thanks a bunch! That's kind of what I feared. Glad I didn't go down too much of a rabbit hole yet.

For what it's worth, the process is really easy to get `impala-shell` working. I just added the Cloudera repository, installed with apt-get and then put the ip address of an impala node into the commands.

I was hoping to get something similar for hdfs. Oh well. I'll look into what you said.

Re: Configure hadoop-client tools to access hdfs from external computer

mbigelow — Mon, 26 Jun 2017 09:22:55 GMT

For what it is worth, I just did this and it worked.

1. set up the cdh 5 repo
2. installed hadoop-client with my package manager
3. updated the configs manually (scp or cm api)
4. ???
5. profit

Re: Configure hadoop-client tools to access hdfs from external computer

epowell — Mon, 26 Jun 2017 14:14:57 GMT

@mbigelow this is really exciting. Thanks for following up on this thread.

I am way back on CM 4.8.5 and CDH 4. Nevertheless, I downloaded and installed the repos similar to what you mention in step two.

Step three is a little foggy for me. Can you elaborate on "updated the configs manually"? Specifically, which configs should I copy over?

Thanks again. This would be a huge win for me if this works.

Re: Configure hadoop-client tools to access hdfs from external computer

mbigelow — Tue, 27 Jun 2017 01:28:35 GMT

I just copied it over from a cluster node.

Re: Configure hadoop-client tools to access hdfs from external computer

epowell — Tue, 27 Jun 2017 14:26:27 GMT

Sorry. I'm really, really new at Hadoop and Cloudera. I actually have no idea where any of the config files are on the cluster nodes.

Can you give me the paths to the config files or tell me where I can find them? Or maybe you could tell me the filenames and I'll search with `find`. Whatever is easiest...

Re: Configure hadoop-client tools to access hdfs from external computer

mbigelow — Tue, 27 Jun 2017 14:43:54 GMT

They should be under /etc/<service>/conf, where service can be hadoop, hive, yarn, etc.

Re: Configure hadoop-client tools to access hdfs from external computer

csguna — Wed, 28 Jun 2017 04:55:11 GMT

Is you cluster managed by cloduera manager ?

if not it is pretty straight forward you can grab it from

/etc/hadoop/conf/

grab the core-site.xml , hdfs-site.xml , hive-site.xml

Re: Configure hadoop-client tools to access hdfs from external computer

csguna — Wed, 28 Jun 2017 11:58:24 GMT

@mbigelow " It is part of the magic that happens when you install a gateway role "

Are you referring to the NFS gateway role ?

Re: Configure hadoop-client tools to access hdfs from external computer

mbigelow — Thu, 29 Jun 2017 02:53:03 GMT

@csguna No, the YARN gateway, HDFS gateway, Hive gateway, etc. Each of these will install the binaries, libraries, set env vars, and client configuration files for its service.

Re: Configure hadoop-client tools to access hdfs from external computer

epowell — Thu, 29 Jun 2017 03:27:41 GMT

Yes, thank you. That did the trick.

So, basically, my procedure was as follows.

Add the cloudera repository containing the hadoop binaries

sudo vim /etc/apt/sources.list.d/cloudera-manager.list

Install the binaries. I used hadoop-client package and that was enough.

sudo apt-get install hadoop-client

Install Java. This made the error about $JAVA_HOME go away even though I still don't have that env set.

sudo apt-get install openjdk-6-jre-headless

Copy the config files from a functioning cluster node

sudo scp user@cluster:/etc/hadoop/conf/* /etc/hadoop/conf

Re: Configure hadoop-client tools to access hdfs from external computer

csguna — Thu, 29 Jun 2017 04:45:50 GMT

@mbigelow I got confused . THanks for clarifying it .

Re: Configure hadoop-client tools to access hdfs from external computer

user21239873 — Tue, 06 Feb 2018 06:07:00 GMT

Hi, referring to the last step, do you encounter the Permission denied error when doing scp?

sudo scp user@cluster:/etc/hadoop/conf/* /etc/hadoop/conf

I managed to copy all the files inside /conf except for container-executor.cfg which shows the message in terminal below:

scp: /etc/hadoop/conf/container-executor.cfg: Permission denied