Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume 1.5.2 Sink failure in HDP 2.5.0: wrong/missing jars for HDFS and Solr morphline

Highlighted

Flume 1.5.2 Sink failure in HDP 2.5.0: wrong/missing jars for HDFS and Solr morphline

New Contributor

Hi,

We have installed Flume 1.5.2 in a HDP 2.5.0 cluster using Ambari 2.4.1.0. The HDFS sink is failing with the following error:

30 Dec 2016 12:58:48,977 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:466) - process failed java.lang.NoSuchMethodError: org.apache.hadoop.net.unix.DomainSocketWatcher.<init>(I)V at org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.<init>(DfsClientShmManager.java:415)

We have make it work deleting the hadoop libs in $FLUME_HOME/lib and adding the ones in /usr/

[root@merc-horton-poc1-slave-0 ~]# ll /usr/hdp/2.5.0.0-1245/flume/lib/hadoop*

-rw-r--r-- 1 root root 17372 Dec 30 12:43 /usr/hdp/2.5.0.0-1245/flume/lib/hadoop-annotations-2.7.3.2.5.0.0-1245.jar

-rw-r--r-- 1 root root 78097 Dec 30 12:41 /usr/hdp/2.5.0.0-1245/flume/lib/hadoop-auth-2.7.3.2.5.0.0-1245.jar

lrwxrwxrwx 1 root root 34 Dec 30 12:42 /usr/hdp/2.5.0.0-1245/flume/lib/hadoop-auth.jar -> hadoop-auth-2.7.3.2.5.0.0-1245.jar -rw-r--r-- 1 root root 3607775 Dec 30 12:42 /usr/hdp/2.5.0.0-1245/flume/lib/hadoop-common-2.7.3.2.5.0.0-1245.jar

lrwxrwxrwx 1 root root 36 Dec 30 12:43 /usr/hdp/2.5.0.0-1245/flume/lib/hadoop-common.jar -> hadoop-common-2.7.3.2.5.0.0-1245.jar

-rw-r--r-- 1 root root 1535128 Jan 2 15:33 /usr/hdp/2.5.0.0-1245/flume/lib/hadoop-mapreduce-client-core-2.7.3.2.5.0.0-1245.jar

Also we installed Solr with ambari service pack and we had to symlink the jar to be able to use morphline sink

Have anyone found this behavior? We found this strange and maybe there is something we missed/did wrong on the installation step. Thoughts?

Kind Regards