Configure hadoop-client tools to access hdfs from external computer

[#1]

I would like to be able to perform hdfs commands from a computer that is NOT actually part of the cloudera cluster.

For example, performing simple put/get operations or

hdfs dfs -ls /my/dir

I have installed the correct binaries, I think.  I found from CM that I was using CDH4.7.1.  So, I installed (sudo apt-get install hadoop-client) the binaries from here.

If I run:

hdfs dfs -ls /

I get:

Error: JAVA_HOME is not set and could not be found.

I feel that this might just be the beginning of a long tinkering and configuring process and I unfortunately know nothing about java.  I do, however, know the IPs of the namenodes on my cluster and have access to all admin rights from beginning to end.  

Can someone help me get things configured?

P.S. In case it wasn't clear.  I can perform all desired functionality on nodes that are part of the cluster.  I just want to do something similar from my development environment.

[#2]

I should refine my question.  Part of what prompted this is that I noticed on the nodes of the cluster that $JAVA_HOME is not defined either.

That makes me think that there might be certain configuration files on the cluster nodes that maybe I can copy over to my development environment.

[#3]

It is part of the magic that happens when you install a gateway roles on a nodes using CM.  It installs the binaries, download the client configs, sets env vars in such a way that no users on that nodes need to do it.  It just works.  I have not dug into it to determine how.  I expected that it is wrapped up in scripts somewhere that get called when using the clients.

Without it, you will need to do it all yourself.  And JAVA_HOME isn't the only one, HADOOP_HOME, HADOOP_CONF, etc.

The Hadoop docs should  have some information on it.

Or... you can add the machine in question to CM, add the gateway role, and then remove it from the cluster.  You shouldn't need to uninstall the gateway role to remove it from the cluster.  No license is required for the gateway roles either.

[#4]

Thanks a bunch!  That's kind of what I feared.  Glad I didn't go down too much of a rabbit hole yet.

For what it's worth, the process is really easy to get `impala-shell` working.  I just added the Cloudera repository, installed with apt-get and then put the ip address of an impala node into the commands.

I was hoping to get something similar for hdfs.  Oh well.  I'll look into what you said.

[#5 · Accepted solution]

For what it is worth, I just did this and it worked.

1. set up the cdh 5 repo
2. installed hadoop-client with my package manager
3. updated the configs manually (scp or cm api)
4. ???
5. profit

[#6]

@mbigelow this is really exciting.  Thanks for following up on this thread.

I am way back on CM 4.8.5 and CDH 4.  Nevertheless, I downloaded and installed the repos similar to what you mention in step two.

Step three is a little foggy for me.  Can you elaborate on "updated the configs manually"?  Specifically, which configs should I copy over?

Thanks again.  This would be a huge win for me if this works.

[#7]

I just copied it over from a cluster node.

[#8]

Sorry.  I'm really, really new at Hadoop and Cloudera.  I actually have no idea where any of the config files are on the cluster nodes.

Can you give me the paths to the config files or tell me where I can find them?  Or maybe you could tell me the filenames and I'll search with `find`.  Whatever is easiest...

[#9]

They should be under /etc/<service>/conf, where service can be hadoop, hive, yarn, etc.

[#10]

Is you cluster managed by cloduera manager ?

if not it is pretty straight forward you can grab it from

/etc/hadoop/conf/

grab the core-site.xml , hdfs-site.xml , hive-site.xml

[#11]

Yes, thank you.  That did the trick.

So, basically, my procedure was as follows.

• Add the cloudera repository containing the hadoop binaries

sudo vim /etc/apt/sources.list.d/cloudera-manager.list

• Install the binaries.  I used hadoop-client package and that was enough. 

sudo apt-get install hadoop-client

• Install Java.  This made the error about $JAVA_HOME go away even though I still don't have that env set.

sudo apt-get install openjdk-6-jre-headless

• Copy the config files from a functioning cluster node

sudo scp user@cluster:/etc/hadoop/conf/* /etc/hadoop/conf   

[#12]

Hi, referring to the last step, do you encounter the Permission denied error when doing scp? 

sudo scp user@cluster:/etc/hadoop/conf/* /etc/hadoop/conf 

I managed to copy all the files inside /conf except for container-executor.cfg which shows the message in terminal below:

scp: /etc/hadoop/conf/container-executor.cfg: Permission denied

[#13]

@mbigelow  " It is part of the magic that happens when you install a gateway role "

Are you referring to the  NFS gateway role ?

[#14]

@csguna No, the YARN gateway, HDFS gateway, Hive gateway, etc. Each of these will install the binaries, libraries, set env vars, and client configuration files for its service.

[#15]

@mbigelow  I got confused . THanks for clarifying it .