Reply
Explorer
Posts: 116
Registered: ‎09-14-2016

Flume kudu sink

HI,

 

My flume plugin dir is 

 

 

/var/lib/flume-ng/plugins.d

 

put kudu sink jar there, but still log says

 

org.apache.flume.FlumeException: Unable to load sink type: org.apache.kudu.flume.sink.KuduSink, class: org.apache.kudu.flume.sink.KuduSink
at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:70)
at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:43)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:408)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:141)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.kudu.flume.sink.KuduSink

 

 

Thanks

Shannon

Explorer
Posts: 116
Registered: ‎09-14-2016

Re: Flume kudu sink

Resolved this, the jar needs to be in certain path.   <plugins>/<custom jar 1>/lib/xxx.

 

 

 

 

Now i am getting this error, my shema path is local file file:///xxx

 

 

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.ShellBasedUnixGroupsMapping not org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.GroupMappingServiceProvider
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.Groups.<init>(Groups.java:80)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.Groups.<init>(Groups.java:74)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2812)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2802)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2668)
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.kudu.flume.sink.AvroKuduOperationsProducer$1.load(AvroKuduOperationsProducer.java:108)
at org.apache.kudu.flume.sink.AvroKuduOperationsProducer$1.load(AvroKuduOperationsProducer.java:102)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
... 9 more
Caused by: java.lang.RuntimeException: class org.apache.hadoop.security.ShellBasedUnixGroupsMapping not org.apache.kudu.flume.sink.shaded.org.apache.hadoop.security.GroupMappingServiceProvider
at org.apache.kudu.flume.sink.shaded.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2221)
... 27 more

Explorer
Posts: 116
Registered: ‎09-14-2016

Re: Flume kudu sink

 

looking at these two lines, it would be better if we check first only if hdfs then try to use hdfs filesystem to read?

 

FileSystem fs = FileSystem.get(URI.create(url), conf);
                if (url.toLowerCase(Locale.ENGLISH).startsWith("hdfs:/")) {

 

 

 

 try {
                FileSystem fs = FileSystem.get(URI.create(url), conf);
                if (url.toLowerCase(Locale.ENGLISH).startsWith("hdfs:/")) {
                  is = fs.open(new Path(url));
                } else {
                  is = new URL(url).openStream();
                }
                return parser.parse(is);
              } finally {
                if (is != null) {
                  is.close();
                }
              }
            }

 

 

 

Explorer
Posts: 116
Registered: ‎09-14-2016

Re: Flume kudu sink

My schema is not in hdfs, how can i pass this error?

Highlighted
Explorer
Posts: 116
Registered: ‎09-14-2016

Re: Flume kudu sink

I am running flume in cloudera (5.12).

Announcements