Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume kudu sink

Highlighted

Flume kudu sink

Contributor

HI,

 

My flume plugin dir is 

 

 

/var/lib/flume-ng/plugins.d

 

put kudu sink jar there, but still log says

 

org.apache.flume.FlumeException: Unable to load sink type: org.apache.kudu.flume.sink.KuduSink, class: org.apache.kudu.flume.sink.KuduSink
at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:70)
at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:43)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:408)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:141)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.kudu.flume.sink.KuduSink

 

 

Thanks

Shannon

4 REPLIES 4

Re: Flume kudu sink

Contributor

Resolved this, the jar needs to be in certain path.   <plugins>/<custom jar 1>/lib/xxx.

 

 

 

 

Now i am getting this error, my shema path is local file file:///xxx

 

 

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.ShellBasedUnixGroupsMapping not org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.GroupMappingServiceProvider
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.Groups.<init>(Groups.java:80)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.Groups.<init>(Groups.java:74)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2812)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2802)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2668)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.kudu.flume.sink.AvroKuduOperationsProducer$1.load(AvroKuduOperationsProducer.java:108)
at org.apache.kudu.flume.sink.AvroKuduOperationsProducer$1.load(AvroKuduOperationsProducer.java:102)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
... 9 more
Caused by: java.lang.RuntimeException: class org.apache.hadoop.security.ShellBasedUnixGroupsMapping not org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.GroupMappingServiceProvider
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2221)
... 27 more

Re: Flume kudu sink

Contributor

 

looking at these two lines, it would be better if we check first only if hdfs then try to use hdfs filesystem to read?

 

FileSystem fs = FileSystem.get(URI.create(url), conf);
                if (url.toLowerCase(Locale.ENGLISH).startsWith("hdfs:/")) {

 

 

 

 try {
                FileSystem fs = FileSystem.get(URI.create(url), conf);
                if (url.toLowerCase(Locale.ENGLISH).startsWith("hdfs:/")) {
                  is = fs.open(new Path(url));
                } else {
                  is = new URL(url).openStream();
                }
                return parser.parse(is);
              } finally {
                if (is != null) {
                  is.close();
                }
              }
            }

 

 

 

Re: Flume kudu sink

Contributor

My schema is not in hdfs, how can i pass this error?

Re: Flume kudu sink

Contributor

I am running flume in cloudera (5.12).