How long will the Hadoop edits file be generated? How long does it take to clean up? I have a cluster that has not yet entered production data, but found that the NameNode has a large number of edits files, and one is generated in about two minutes, and it has not been cleaned up; The number of edits files in the NN of another cluster is normal, but the number of edits files in the JN is very large, and no cleaning operation is done ... View more

The files in the folder are read by blob, the files are deleted in real time, but kafka is not entered, and kafka is also reporting an error. ... View more

I have a problem now, after starting the Flume agent, Flume immediately gives an error: 2019-01-28 14:28:52,406 ERROR org.apache.flume.source.kafka.KafkaSource: KafkaSource EXCEPTION, {} org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: ct103} at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200) at org.apache.flume.source.kafka.KafkaSource.doProcess(KafkaSource.java:132) at org.apache.flume.source.AbstractPollableSource.process(AbstractPollableSource.java:58) at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:137) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.flume.ChannelException: Put queue for MemoryTransaction of capacity 100 full, consider committing more frequently, increasing capacity or increasing thread count at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doPut(MemoryChannel.java:84) at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93) at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80) at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189) ... 4 more   My flume configuration: tier1.sources= rt103 tier1.channels= ct103 tier1.sinks= kt103 # Configure source tier1.sources.rt103.type = org.apache.flume.source.kafka.KafkaSource tier1.sources.rt103.channels = ct103 tier1.sources.rt103.zookeeperConnect =10.231.35.6:2181,10.231.35.15:2181,10.231.35.122:2181 tier1.sources.rt103.topic = zk103 tier1.sources.rt103.groupId = flume21a22 tier1.sources.rt103.kafka.consumer.timeout.ms = 100 tier1.sources.rt103.kafka.fetch.message.max.bytes = 512000000 # Configure channel tier1.channels.ct103.type=memory tier1.channels.ct103.capacity=20000000 tier1.channels.ct103.transactioncapacity=2000000 tier1.channels.ct103.byteCapacityBufferPercentage = 20 tier1.channels.ct103.byteCapacity = 8589934592 tier1.channels.ct103.threads=20 #Configure sink tier1.sinks.kt103.channel = ct103 tier1.sinks.kt103.type = hdfs tier1.sinks.kt103.hdfs.useLocalTimeStamp = true tier1.sinks.kt103.hdfs.kerberosPrincipal = bdi tier1.sinks.kt103.hdfs.kerberosKeytab = /bdi/conf/bdi.keytab tier1.sinks.kt103.hdfs.path = hdfs://nameservice-newremote1/user/bdi/4gdpi/11/103/%Y%m%d%H tier1.sinks.kt103.hdfs.filePrefix = 4gdpi_ip21_flume1_kt103_%Y%m%d%H%M%S tier1.sinks.kt103.hdfs.fileSuffix = .txt tier1.sinks.kt103.hdfs.rollInterval = 0 tier1.sinks.kt103.hdfs.rollSize = 134217728 tier1.sinks.kt103.hdfs.rollCount = 0 tier1.sinks.kt103.hdfs.idleTimeout = 60 tier1.sinks.kt103.hdfs.round = true tier1.sinks.kt103.hdfs.roundValue = 10 tier1.sinks.kt103.hdfs.callTimeout = 600000 tier1.sinks.kt103.hdfs.roundUnit = minute tier1.sinks.kt103.hdfs.writeFormat = Text tier1.sinks.kt103.hdfs.fileType = DataStream tier1.sinks.kt103.hdfs.batchSize = 10000 tier1.sinks.kt103.hdfs.maxOpenFiles = 5000   How has anyone encountered this problem? ... View more