Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

flume kafkasource, hdfs sink remove avro field

avatar
Contributor

I want to create a table with the complex type removed from the avro data in the same schema. This is because Impala does not skipping complex types. Platform is CDH 6.0.1

For Example :

Employee(raw data)
  - name : string
  - age : int
  - additional-info : map<string, string>

Employee(Hive table 1)
  - name : string
  - age : int
  - additional-info : map<string, string>

Employee_For_Implala(Hive table 2)
  - name : string
  - age : int

Pipeline :

KafkaProducer(Avro Bytes) - Kafka - Flume - HDFS - Hive(Impala)

Flume : KafkaSource - Channel - Sink(AvroEventSerializer$Builder)

I tried changing the sink(serializer.schemaURL, remove Complex type field) but it failed.

I am trying to use morphine now. But this is also failing.

Is there a better way?

1 ACCEPTED SOLUTION

avatar

Morphlines would be the preferred way to selectively choose the data that will be passing through the source to the sink. you can use the morphline removeFields command [1] to selectively drop the fields you don't want. If you need to review what is happening with the data you can turn on morphline TRACE by adding the following to the flume logging safety valve:
log4j.logger.org.kitesdk.morphline=TRACE

 

-pd

 

[1] http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html#removeFields

View solution in original post

2 REPLIES 2

avatar

Morphlines would be the preferred way to selectively choose the data that will be passing through the source to the sink. you can use the morphline removeFields command [1] to selectively drop the fields you don't want. If you need to review what is happening with the data you can turn on morphline TRACE by adding the following to the flume logging safety valve:
log4j.logger.org.kitesdk.morphline=TRACE

 

-pd

 

[1] http://kitesdk.org/docs/1.1.0/morphlines/morphlines-reference-guide.html#removeFields

avatar
Contributor
Thanks. I'll try it the way you told me.