Support Questions

Find answers, ask questions, and share your expertise

mulitple flume sinks

Master Collaborator

I have a working flume sink to hdfs from a kafka source. I want to add another sink to hive , is it possible?

flume1.sources = kafka-source-1
flume1.channels = hdfs-channel-1
flume1.sinks = hdfs-sink-1
flume1.sources.kafka-source-1.type = org.apache.flume.source.kafka.KafkaSource
flume1.sources.kafka-source-1.zookeeperConnect = hadoop1:2181
flume1.sources.kafka-source-1.topic =kafkahdfs
flume1.sources.kafka-source-1.batchSize = 100
flume1.sources.kafka-source-1.channels = hdfs-channel-1
### Sinks #######
flume1.sinks.hdfs-sink-1.channel = hdfs-channel-1
flume1.sinks.hdfs-sink-1.type = hdfs
flume1.sinks.hdfs-sink-1.hdfs.writeFormat = Text
flume1.sinks.hdfs-sink-1.hdfs.fileType = DataStream
flume1.sinks.hdfs-sink-1.hdfs.filePrefix = UFM
flume1.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true
flume1.sinks.hdfs-sink-1.hdfs.path = /tmp/kafka/%{topic}/%y-%m-%d
flume1.sinks.hdfs-sink-1.hdfs.rollCount=100
flume1.sinks.hdfs-sink-1.hdfs.rollSize=0
### Channels ######
flume1.channels.hdfs-channel-1.capacity = 10000
flume1.channels.hdfs-channel-1.type = memory
flume1.channels.hdfs-channel-1.transactionCapacity = 1000
flume1.channels.hdfs-channel-1.kafka.bootstrap.servers = hadoop1.example.com:6667
flume1.channels.hdfs-channel-1.topic = kafkahdfs
2 REPLIES 2

Contributor

@Sami Ahmad You can specify multiple sinks in the agent.sinks config. Something like this -

flume1.sinks = hdfs-sink-1 hive-sink-2
flume1.sinks.hive-sink-2.channel =  hdfs-channel-1
flume1.sinks.hive-sink-2.type = hive
flume1.sinks.hive-sink-2.hive.metastore = thrift://localhost:9083
flume1.sinks.hive-sink-2.hive.database = <flumedb>
flume1.sinks.hive-sink-2.hive.table = <flumetable>
flume1.sinks.hive-sink-2.serializer = DELIMITED
flume1.sinks.hive-sink-2.serializer.delimiter = ,
flume1.sinks.hive-sink-2.serializer.fieldnames = <field_names>
flume1.sinks.hive-sink-2.batchSize = 10

Modify and add the configs accordingly and please give it a try.

Contributor

@Sami Ahmad Did you get a chance to try this config and does it work? Please update.