01-16-2019 11:16 PM - last edited on 01-17-2019 05:56 AM by cjervis
I want to create a table with the complex type removed from the avro data in the same schema. This is because Impala does not skipping complex types. Platform is CDH 6.0.1
For Example :
Employee(raw data) - name : string - age : int - additional-info : map<string, string> Employee(Hive table 1) - name : string - age : int - additional-info : map<string, string> Employee_For_Implala(Hive table 2) - name : string - age : int
KafkaProducer(Avro Bytes) - Kafka - Flume - HDFS - Hive(Impala)
Flume : KafkaSource - Channel - Sink(AvroEventSerializer$Builder)
I tried changing the sink(serializer.schemaURL, remove Complex type field) but it failed.
I am trying to use morphine now. But this is also failing.
Is there a better way?
01-23-2019 10:58 AM
Morphlines would be the preferred way to selectively choose the data that will be passing through the source to the sink. you can use the morphline removeFields command  to selectively drop the fields you don't want. If you need to review what is happening with the data you can turn on morphline TRACE by adding the following to the flume logging safety valve: