Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume: HDFS sink: FileSystem doesn't support getNumCurrentReplicas; --HDFS 826 not available

Flume: HDFS sink: FileSystem doesn't support getNumCurrentReplicas; --HDFS 826 not available

New Contributor

I'm using flume to get data from Kafka to HDFS. (Kafka Source and HDFS Sink). These are the versions I'm using.

HDP 2.6.2.0-205
Flume -1.5.2.2.6.2-205

This is my flume.conf.

agent1.sources = kafka-source
agent1.channels = memory-channel
agent1.sinks = hdfs-sink

agent1.sources.kafka-source.type = org.apache.flume.source.kafka.KafkaSource
agent1.sources.kafka-source.batchSize = 5
agent1.sources.kafka-source.kafka.consumer.timeout.ms = 100
agent1.sources.kafka-source.kafka.topics = test
agent1.sources.kafka-source.kafka.bootstrap.servers = localhost:9092

agent1.channels.memory-channel.type = memory
agent1.channels.memory-channel.capacity = 10000
agent1.channels.memory-channel.transactionCapacity = 1000

agent1.sinks.hdfs-sink.type = hdfs
agent1.sinks.hdfs-sink.hdfs.path = /tmp/kafka/%{topic}/%y-%m-%d
agent1.sinks.hdfs-sink.channel = memory-channel


I'm getting the below. This keeps coming and I don't see data in HDFS.

2017-11-10 14:36:21,713 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:409)] Closing /tmp/kafka/test/17-11-10/FlumeData.1510353261651.tmp
2017-11-10 14:36:21,713 (hdfs-hdfs-sink-call-runner-1) [INFO - org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:675)] Renaming /tmp/kafka/test/17-11-10/FlumeData.1510353261651.tmp to /tmp/kafka/test/17-11-10/FlumeData.1510353261651
2017-11-10 14:36:21,716 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:284)] Creating /tmp/kafka/test/17-11-10/FlumeData.1510353261652.tmp
2017-11-10 14:36:21,722 (hdfs-hdfs-sink-call-runner-0) [INFO - org.apache.flume.sink.hdfs.AbstractHDFSWriter.reflectGetNumCurrentReplicas(AbstractHDFSWriter.java:184)] FileSystem's output stream doesn't support getNumCurrentReplicas; --HDFS-826 not available; fsOut=org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer; err=java.lang.NoSuchMethodException: org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.getNumCurrentReplicas()
2017-11-10 14:36:21,722 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.getRefIsClosed(BucketWriter.java:233)] isFileClosed() is not available in the version of the distributed filesystem being used. Flume will not attempt to re-close files if the close fails on the first attempt 
Don't have an account?
Coming from Hortonworks? Activate your account here