Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

nifi ConvertAvroToJSON help needed

Highlighted

nifi ConvertAvroToJSON help needed

Master Collaborator

can someone guide me on how to divert PutHiveStreaming data into ConvertAvroToJSON processor ? I had a hadoop engineer on site for short training and he setup the flow but had to leave before he could tell me this step.

I am attaching the nifi flow xml .

thanks

inc-hive-import-temp.xml

10 REPLIES 10
Highlighted

Re: nifi ConvertAvroToJSON help needed

Master Guru

@Sami Ahmad

If i am understanding you correctly, the solution is as simple as dragging the circle that appears over the putHIveStreaming processor when you float your cursor over it to your ConvertAvroToJson processor.

In your connection configuration windows that appears, check the box next to the "success" relationship.

FlowFile that were successfully processed by the PutHiveStreaming will now get routed to the ConvertAvroToJson processor instead of being auto-terminated.

Thank you,

Matt

If you found this answer helpful, please take a moment to click "Accept" below.

Highlighted

Re: nifi ConvertAvroToJSON help needed

Master Collaborator

is it possible for you to modify the xml flow I send so I can import it ?

Highlighted

Re: nifi ConvertAvroToJSON help needed

Master Guru

@Sami Ahmad

What do you plan on doing next after ConvertAvrotoJson? Simply feeding the "success" connection to the next processor in a dataflow does not complete the dataflow. I am sure you intend to do something with the generated JSON files, correct?

If not FlowFiles, are just going to queue up on the success relationship between PutHiveStreaminga and ConvertAvroToJson until back pressure kicks in on the connection and causes your putHiveStreaming to stop running.

My suggestion is you play around with building the rest of the convertAvroToJson dataflow first to become more familiar with the NiFi interface and then connect your PutHiveStreaming processor to it,.

Thanks,

Matt

Highlighted

Re: nifi ConvertAvroToJSON help needed

Master Collaborator

sorry for late reply . I want to save the JSON data in hive .

Highlighted

Re: nifi ConvertAvroToJSON help needed

Super Guru

PutHiveStreaming currently requires Avro and does not accept JSON (there is a Jira to support JSON). You can use PutHDFS to put the JSON files in HDFS and then create the table using PutHiveQL.

Highlighted

Re: nifi ConvertAvroToJSON help needed

Super Guru
@Sami Ahmad

if you want to process the avro data that is stored by using PutHiveStreaming processor then Feed the success relation of PutHiveStreaming processor to ConvertAvroToJSON processor, then do your processing on the json data.

62438-converavrotojson.png

Highlighted

Re: nifi ConvertAvroToJSON help needed

Master Collaborator

since I need JSON data to put in hive ..it doesn't like AVRO I was told , shouldn't I be adding the ConvertAVroToJSON processor before PutHiveStreaming ?

Highlighted

Re: nifi ConvertAvroToJSON help needed

Super Guru
@Sami Ahmad

PutHiveStreaming processor require the incoming flowfile needs to be in Avro format and the table needs to exist in hive and all filed datatypes should be strings, orc format,bucketed,transactional ..etc are the requirements for the tables that are used in PutHiveStreaming processor.

62439-hivestreaming.png

Refer to below link how to stream data into hive using PutHiveStreaming processor

https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html

if you want put json data in hive then instead of using PutHiveStreaming processor use PutHDFS processor after ConvertAvtoToJson and then create Hive table with json serde pointing to HDFS location.

Flow:-

62440-flow.png

Reference how to create hive table on json data

https://community.hortonworks.com/questions/64658/create-hive-table-with-this-json-format.html

Highlighted

Re: nifi ConvertAvroToJSON help needed

Master Collaborator

hi Shu

the create hive table step would be outside of nifi ? how will I automate this then ?

Don't have an account?
Coming from Hortonworks? Activate your account here