Created 09-05-2017 02:07 PM
I have a working flume sink to hdfs from a kafka source. I want to add another sink to hive , is it possible?
flume1.sources = kafka-source-1 flume1.channels = hdfs-channel-1 flume1.sinks = hdfs-sink-1 flume1.sources.kafka-source-1.type = org.apache.flume.source.kafka.KafkaSource flume1.sources.kafka-source-1.zookeeperConnect = hadoop1:2181 flume1.sources.kafka-source-1.topic =kafkahdfs flume1.sources.kafka-source-1.batchSize = 100 flume1.sources.kafka-source-1.channels = hdfs-channel-1 ### Sinks ####### flume1.sinks.hdfs-sink-1.channel = hdfs-channel-1 flume1.sinks.hdfs-sink-1.type = hdfs flume1.sinks.hdfs-sink-1.hdfs.writeFormat = Text flume1.sinks.hdfs-sink-1.hdfs.fileType = DataStream flume1.sinks.hdfs-sink-1.hdfs.filePrefix = UFM flume1.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true flume1.sinks.hdfs-sink-1.hdfs.path = /tmp/kafka/%{topic}/%y-%m-%d flume1.sinks.hdfs-sink-1.hdfs.rollCount=100 flume1.sinks.hdfs-sink-1.hdfs.rollSize=0 ### Channels ###### flume1.channels.hdfs-channel-1.capacity = 10000 flume1.channels.hdfs-channel-1.type = memory flume1.channels.hdfs-channel-1.transactionCapacity = 1000 flume1.channels.hdfs-channel-1.kafka.bootstrap.servers = hadoop1.example.com:6667 flume1.channels.hdfs-channel-1.topic = kafkahdfs
Created 09-07-2017 01:58 PM
@Sami Ahmad You can specify multiple sinks in the agent.sinks config. Something like this -
flume1.sinks = hdfs-sink-1 hive-sink-2 flume1.sinks.hive-sink-2.channel = hdfs-channel-1 flume1.sinks.hive-sink-2.type = hive flume1.sinks.hive-sink-2.hive.metastore = thrift://localhost:9083 flume1.sinks.hive-sink-2.hive.database = <flumedb> flume1.sinks.hive-sink-2.hive.table = <flumetable> flume1.sinks.hive-sink-2.serializer = DELIMITED flume1.sinks.hive-sink-2.serializer.delimiter = , flume1.sinks.hive-sink-2.serializer.fieldnames = <field_names> flume1.sinks.hive-sink-2.batchSize = 10
Modify and add the configs accordingly and please give it a try.
Created 09-13-2017 04:26 AM
@Sami Ahmad Did you get a chance to try this config and does it work? Please update.