Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hdfs Avro format file need to insert into Hive ORCTable in nifi

Solved Go to solution

hdfs Avro format file need to insert into Hive ORCTable in nifi

hdfs Avro formatted file insert into HiveORC formatted table

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: hdfs Avro format file need to insert into Hive ORCTable in nifi

Super Guru
@Hariprasanth Madhavan

There are lot of ways to insert data into `HiveORC` table from NiFi.

Method1: Using PutHiveStreaming Processor:
Create transactional table and then feed the avro data to PutHivestreaming table.

As HiveStreaming processor converts the avro format data into ORC format and regards to all delta files you can use major compaction to create one base file.

Method2: ConvertAvroToORC in NiFi and store into HDFS:

Use ConvertAvroToORC processor to convert the avro format data into ORC format.

Store the data into HDFS and create an External hive table pointing to the same HDFS directory.

Method3: Create Avro table and load from Avro table to ORC table:

Based on the avro file we are having in NiFi we can create avro tables dynamically based on avro.schema.

Create an orc table and after storing the avro data into HDFS use PutHiveQL processor to run

insert into ORC table select * from Avro table

Refer to this link for more details regards to create avro table dynamically.

-

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

View solution in original post

1 REPLY 1
Highlighted

Re: hdfs Avro format file need to insert into Hive ORCTable in nifi

Super Guru
@Hariprasanth Madhavan

There are lot of ways to insert data into `HiveORC` table from NiFi.

Method1: Using PutHiveStreaming Processor:
Create transactional table and then feed the avro data to PutHivestreaming table.

As HiveStreaming processor converts the avro format data into ORC format and regards to all delta files you can use major compaction to create one base file.

Method2: ConvertAvroToORC in NiFi and store into HDFS:

Use ConvertAvroToORC processor to convert the avro format data into ORC format.

Store the data into HDFS and create an External hive table pointing to the same HDFS directory.

Method3: Create Avro table and load from Avro table to ORC table:

Based on the avro file we are having in NiFi we can create avro tables dynamically based on avro.schema.

Create an orc table and after storing the avro data into HDFS use PutHiveQL processor to run

insert into ORC table select * from Avro table

Refer to this link for more details regards to create avro table dynamically.

-

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here