Reply
Highlighted
Explorer
Posts: 22
Registered: ‎08-23-2018
Accepted Solution

flume kafkasource, hdfs sink remove avro field

[ Edited ]

I want to create a table with the complex type removed from the avro data in the same schema. This is because Impala does not skipping complex types. Platform is CDH 6.0.1

For Example :

Employee(raw data)
  - name : string
  - age : int
  - additional-info : map<string, string>

Employee(Hive table 1)
  - name : string
  - age : int
  - additional-info : map<string, string>

Employee_For_Implala(Hive table 2)
  - name : string
  - age : int

Pipeline :

KafkaProducer(Avro Bytes) - Kafka - Flume - HDFS - Hive(Impala)

Flume : KafkaSource - Channel - Sink(AvroEventSerializer$Builder)

I tried changing the sink(serializer.schemaURL, remove Complex type field) but it failed.

I am trying to use morphine now. But this is also failing.

Is there a better way?

Cloudera Employee
Posts: 277
Registered: ‎01-09-2014

Re: flume kafkasource, hdfs sink remove avro field

Morphlines would be the preferred way to selectively choose the data that will be passing through the source to the sink. you can use the morphline removeFields command [1] to selectively drop the fields you don't want. If you need to review what is happening with the data you can turn on morphline TRACE by adding the following to the flume logging safety valve:
log4j.logger.org.kitesdk.morphline=TRACE

 

-pd

 

[1] http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html#removeFields

Explorer
Posts: 22
Registered: ‎08-23-2018

Re: flume kafkasource, hdfs sink remove avro field

Thanks. I'll try it the way you told me.