Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

MR Job using Hive libraries

avatar
Explorer

Hi all,

 

I am using Cloudera manager

 

 Cloudera Standard 4.7.2 (#135 built by jenkins on 20130918-2007 git: 72d3f9dfa797fe2c627d00dc6414a1e0151b91d6)

 

CDH 4.4.0-1.cdh4.4.0.p0.39

 

I have deployed hive and the hive2 server to one of the boxes on my cluster, along with it being a Journal Node and HDFS Balencer. When I do a hadoop fs -ls /user/, i get a bunch of directories; but if I kick off a MR job via the hive --service jar ... I am getting a

 

Exception in thread "main" java.io.FileNotFoundException: File /user does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:334)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1462)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1502)
at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:570)

 

because it looks like it is not picking up the right core-site.xml, etc. I have deployed the client configuration for both hive and hdfs. Anyone have an idea?

 

thanks

 

dave

1 ACCEPTED SOLUTION

avatar

Click on your MapReduce service, go to the instances tab, and add a Gateway to the host where you want configs. Then deploy client configuration.

 

In general, you can use gateway roles to deploy client configs to hosts. This works for services like Hive and HBase as well.

 

Thanks,

Darren

View solution in original post

3 REPLIES 3

avatar
Explorer
Nevermind, figured it out. This job used to run on a box with hdfs and MR installed. Only runs on a box with hdfs config. I was asking for the file system via the JobClient and was getting back the local file system.

avatar
Explorer

Is there a way in Cloudera Manager to put MR configs on a box that isn't a TT or Job tracker?

avatar

Click on your MapReduce service, go to the instances tab, and add a Gateway to the host where you want configs. Then deploy client configuration.

 

In general, you can use gateway roles to deploy client configs to hosts. This works for services like Hive and HBase as well.

 

Thanks,

Darren