Thank you very much for helping me, but I have some questions to help: 1. If the files are not moved to another folder (like questions 1 and 2 I mentioned), when the folder is too many files, for example 1 billion files, the server is full, I have to do that what? Maybe I have to reconfigure with another spool folder? 2. This is the configuration file I wanted to mention in question 5 # Sources, channels, and sinks are defined per
# agent name, in this case 'tier1'.
tier1.sources = source1
tier1.channels = channel1
tier1.sinks = sink1
# For each source, channel, and sink, set
# standard properties.
# source details
tier1.sources.source1.type = spooldir
tier1.sources.source1.spoolDir = /data/diem
tier1.sources.source1.fileHeader = false
tier1.sources.source1.fileSuffix = .COMPLETED
tier1.sources.source1.channels = channel1
tier1.sources.source1.interceptors = i1
tier1.sources.source1.interceptors.i1.type = regex_extractor
tier1.sources.source1.interceptors.i1.regex = \\[(.*?)\\]
tier1.sources.source1.interceptors.i1.serializers = s1
tier1.sources.source1.interceptors.i1.serializers.s1.type = org.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializer
tier1.sources.source1.interceptor.serializers.s1.name = timestamp
tier1.sources.source1.serializers.s1.pattern = yyyy-MM-dd HH:mm:ss
# channel details
tier1.channels.channel1.type = file
tier1.channels.channel1.capacity = 200000
tier1.channels.channel1.transactionCapacity = 1000
# sink details
tier1.sinks.sink1.type = HDFS
tier1.sinks.sink1.fileType = DataStream
tier1.sinks.sink1.hdfs.writeFormat = Text
tier1.sinks.sink1.channel = channel1
tier1.sinks.sink1.hdfs.path = hdfs://localhost:8020/user/cloudera/testFolder/%y-%m-%d/%H%M/%S
tier1.sinks.sink1.round = true
tier1.sinks.sink1.roundValue = 10
tier1.sinks.sink1.roundUnit = minute
tier1.sinks.sink1.hdfs.rollSize = 268435456
tier1.sinks.sink1.rollInterval = 0
tier1.sinks.sink1.hdfs.batchSize = 10000
And this is an error in the log file 2018-12-24 11:56:03,065 ERROR org.apache.flume.sink.hdfs.HDFSEventSink: process failed
java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:251)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:460)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:368)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:745)
2018-12-24 11:56:03,069 ERROR org.apache.flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:451)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:251)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:460)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:368)
... 3 more And once again thank you for helping me answer these questions
... View more
