Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Nifi: Check hiveql.args.n.value file attributes

avatar
New Contributor

I have to store data coming in from Kafka queue into a hive table.

However, some values are null. Eg:

{

    "id":"123",

    "name":"Sam",

    "dob":"02-01-2000",

    "gender":"m",

    "food_prefernce":null

}

 

When I pass this json through the ConvertJSONToSQL processor, I get sql.args.n.type for all 5 keys but sql..args.n.value for only 4 of them. The null value is not turned into sql.args.5.value.

For this, I manually need to add the missing attributes to the file.

 

The problem is that I have a total of 57 columns in the table and any of those values could be null.

I am using the UpdateAttribute processor to check whether a given attribute is null or not, if yes, then i want to add that attribute having null value.

 

How do I check whther hiveql.args.<>.value is null and then update all in one shot?

Any way to write a generic expression to check all such attribute values?

Thanks in advance!

@mburgess  @MattWho  @stevenmatison  @gazzer82 @TimothySpann @sunile_manjee 

4 REPLIES 4

avatar
New Contributor

OR,

 

How to loop over attribute values: (achieve something like this)

for(int i-0;i<n;i++) {

 if(${hiveql.args.i.value:isnull()) {

   //do something

  }

}

avatar
Master Guru

Use PutHive3Streaming instead from Kafka

or PutORC

avatar
Master Guru

switch to records, never use convertjsontosql 

avatar
New Contributor

Hi @TimothySpann,

Thank you so much for your reply!

 

I tried using the PutHiveStreaming processor but i'm getting this error:

Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='default', table='student_record', partitionVals=[] }

 

My Hive version is 2.3.7

 

Is it true that NiFi does not contain any processors that support Hive version 2.x? 

Is there any workaround?