MR Job using Hive libraries

[#1]

Hi all,

I am using Cloudera manager

 Cloudera Standard 4.7.2 (#135 built by jenkins on 20130918-2007 git: 72d3f9dfa797fe2c627d00dc6414a1e0151b91d6)

CDH 4.4.0-1.cdh4.4.0.p0.39

I have deployed hive and the hive2 server to one of the boxes on my cluster, along with it being a Journal Node and HDFS Balencer. When I do a hadoop fs -ls /user/, i get a bunch of directories; but if I kick off a MR job via the hive --service jar ... I am getting a

Exception in thread "main" java.io.FileNotFoundException: File /user does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:334)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1462)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1502)
at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:570)

because it looks like it is not picking up the right core-site.xml, etc. I have deployed the client configuration for both hive and hdfs. Anyone have an idea?

thanks

dave

[#2]

Nevermind, figured it out. This job used to run on a box with hdfs and MR installed. Only runs on a box with hdfs config. I was asking for the file system via the JobClient and was getting back the local file system.

[#3]

Is there a way in Cloudera Manager to put MR configs on a box that isn't a TT or Job tracker?

[#4 · Accepted solution]

Click on your MapReduce service, go to the instances tab, and add a Gateway to the host where you want configs. Then deploy client configuration.

In general, you can use gateway roles to deploy client configs to hosts. This works for services like Hive and HBase as well.

Thanks,

Darren