Flume Hive sink error

[#1]

Hi,

I'm trying to ingest data with flume into hive table using appropriate hive sink. The table is partitioned with column date(e.g. "2016/05/26") and clustered into 5 buckets. Also all requirements for hive streaming are specified as described in StreamingRequirements

But all the time I get the same error:

FAILED: Hive Internal Error: java.lang.ClassNotFoundException(org.apache.atlas.hive.hook.HiveHook) java.lang.ClassNotFoundException: org.apache.atlas.hive.hook.HiveHook at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:60) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1376) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1360) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1589) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1126) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1116) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.runDDL(HiveEndPoint.java:456) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.createPartitionIfNotExists(HiveEndPoint.java:421) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:280) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:246) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:183) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:160) at org.apache.flume.sink.hive.HiveWriter$8.call(HiveWriter.java:376) at org.apache.flume.sink.hive.HiveWriter$8.call(HiveWriter.java:373) at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:425) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

I use hdp 2.4.2 sandbox with default flume version. How can I resolve this issue?

Thanks!

[#2 · Accepted solution]

it looks that atlas hook is enabled in your sandbox, could you please check your hive-site.xml change hive.exec.post.hooks as follows if you dont want to use atlas hooks

from

<property>

<name>hive.exec.post.hooks</name>

<value>org.apache.hadoop.hive.ql.hooks.ATSHook, org.apache.atlas.hive.hook.HiveHook</value>

</property>

to

<property>

<name>hive.exec.post.hooks</name>

<value>org.apache.hadoop.hive.ql.hooks.ATSHook</value>

</property>

[#3]

thanks, that fixed the issue!