Support Questions
Find answers, ask questions, and share your expertise

Mapreduce and Hcatalog Integration fails to use MySql MetaStore

Contributor

Environment: HDP 2.3 Sandbox

Problem: I have created a table in hive with just 2 columns. Now i want to read this in my MR code using HCatalog integration. The MR Job fails to read the table from the MySql meta-store. It uses the Derby for some reason and hence it fails with "table not found" message.

Job Client code:

public class HCatalogMRJob extends Configured implements Tool {

   public int run(String[] args) throws Exception {
        Configuration conf = getConf();
        args = new GenericOptionsParser(conf, args).getRemainingArgs();
        String inputTableName = args[0];
        String outputTableName = args[1];
        String dbName = null;
        Job job = new Job(conf, "HCatalogMRJob");
        HCatInputFormat.setInput(job, dbName, inputTableName);
        job.setInputFormatClass(HCatInputFormat.class);
        job.setJarByClass(HCatalogMRJob.class);
        job.setMapperClass(HCatalogMapper.class);
        job.setReducerClass(HCatalogReducer.class);
        job.setMapOutputKeyClass(IntWritable.class);
        job.setMapOutputValueClass(IntWritable.class);
        job.setOutputKeyClass(WritableComparable.class);
        job.setOutputValueClass(DefaultHCatRecord.class);
        HCatOutputFormat.setOutput(job, OutputJobInfo.create(dbName, outputTableName, null));
        HCatSchema s = HCatOutputFormat.getTableSchema(conf);
        System.err.println("INFO: output schema explicitly set for writing:"
                + s);
        HCatOutputFormat.setSchema(job, s);
        job.setOutputFormatClass(HCatOutputFormat.class);
        return (job.waitForCompletion(true) ? 0 : 1);
    }
    public static void main(String[] args) throws Exception {
        int exitCode = ToolRunner.run(new HCatalogMRJob(), args);
        System.exit(exitCode);
    }
}

Job Run Command:

hadoop jar mr-hcat.jar input_table out_table

Before running this command, i have set the necessary hcatalog, hive jars in the class path using the hadoop_classpath variable.

Question:

Now, how do i make the job to use the hive-site.xml correctly?

I tried setting this in the classpath using the same hadoop_classpath as mentioned above., but still it fails.

1 ACCEPTED SOLUTION

Contributor

Issue resolved by setting /etc/hive/conf in the classpath Instead of /etc/hive/conf/*.

View solution in original post

1 REPLY 1

Contributor

Issue resolved by setting /etc/hive/conf in the classpath Instead of /etc/hive/conf/*.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.