question Re: MR Job using Hive libraries in Archives of Support Questions (Read Only)

MR Job using Hive libraries

stampy88 — Fri, 16 Sep 2022 08:50:46 GMT

Hi all,

I am using Cloudera manager

Cloudera Standard 4.7.2 (#135 built by jenkins on 20130918-2007 git: 72d3f9dfa797fe2c627d00dc6414a1e0151b91d6)

CDH 4.4.0-1.cdh4.4.0.p0.39

I have deployed hive and the hive2 server to one of the boxes on my cluster, along with it being a Journal Node and HDFS Balencer. When I do a hadoop fs -ls /user/, i get a bunch of directories; but if I kick off a MR job via the hive --service jar ... I am getting a

Exception in thread "main" java.io.FileNotFoundException: File /user does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:334)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1462)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1502)
at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:570)

because it looks like it is not picking up the right core-site.xml, etc. I have deployed the client configuration for both hive and hdfs. Anyone have an idea?

thanks

dave

Re: MR Job using Hive libraries

stampy88 — Tue, 03 Dec 2013 22:40:49 GMT

Nevermind, figured it out. This job used to run on a box with hdfs and MR installed. Only runs on a box with hdfs config. I was asking for the file system via the JobClient and was getting back the local file system.

Re: MR Job using Hive libraries

stampy88 — Tue, 03 Dec 2013 22:43:45 GMT

Is there a way in Cloudera Manager to put MR configs on a box that isn't a TT or Job tracker?

Re: MR Job using Hive libraries

Darren — Tue, 03 Dec 2013 23:17:30 GMT

Click on your MapReduce service, go to the instances tab, and add a Gateway to the host where you want configs. Then deploy client configuration.

In general, you can use gateway roles to deploy client configs to hosts. This works for services like Hive and HBase as well.

Thanks,

Darren