Support Questions

Find answers, ask questions, and share your expertise

Is there any processor in NiFi which helps me add a unique identifier to each record?

avatar
Explorer

My processor is querying a DB2 table and loads data to S3 and then ingest it into Snowflake. I have unique account numbers in my table and I need to add a column with unique identifier for each unique account number.

1 ACCEPTED SOLUTION

avatar
Super Guru

@ang_coder  Depending on the number of unique values you need to add,  updateAttribute + expression language will allow you to create flowfile attribute based on the table results in a manner I would call "manually".   These can be used in routing or further manipulating the content (original database rows) according to your match logic.  For example with ReplaceText you can replace the original value with the original value + the new value.    

 

Additionally during your flow you can programmatically change the results of the content of the flowfile to add the new column using the attribute from above, or with a fabricated query.  In the latter example you would use a RecordReader/RecordWriter/UpdateRecord on your data.  In a nutshell you create a translation on the content that includes adding the new field.    This is a common use case for nifi and there are many different ways to achieve it.

 

 

To have a more complete reply that better matches your use case,  you should provide more information, sample input data, the expected output data, your flow, a template of your flow, and maybe what you have tried already. 

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

View solution in original post

2 REPLIES 2

avatar

Assuming your flowfile contains multiple records, this should probably be achievable with the UpdateRecord processor. 

 

Note that the expression language has a UUID function which may be helpful to use inside this.


- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.

avatar
Super Guru

@ang_coder  Depending on the number of unique values you need to add,  updateAttribute + expression language will allow you to create flowfile attribute based on the table results in a manner I would call "manually".   These can be used in routing or further manipulating the content (original database rows) according to your match logic.  For example with ReplaceText you can replace the original value with the original value + the new value.    

 

Additionally during your flow you can programmatically change the results of the content of the flowfile to add the new column using the attribute from above, or with a fabricated query.  In the latter example you would use a RecordReader/RecordWriter/UpdateRecord on your data.  In a nutshell you create a translation on the content that includes adding the new field.    This is a common use case for nifi and there are many different ways to achieve it.

 

 

To have a more complete reply that better matches your use case,  you should provide more information, sample input data, the expected output data, your flow, a template of your flow, and maybe what you have tried already. 

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ