Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to route fixed length records into multiple routes by using Route on Text in nifil

how to route fixed length records into multiple routes by using Route on Text in nifil

New Contributor

 How to route fixed width records into multiple Routes to use individual copybooks,

 I used earlier Route on content processor if my data starts as below and its working as expected.                                              
Example Data


00ABD0000000100000103200A08LL30003    0000000000220700840940
10ABD0000000210000100060A08LL30003    0000000000220700840940


In nifi we used earlier
recordtype01=00

recordtype02=10
route01 =(${recordtype01}.*)

route02 =(${recordtype02}.*)

 

Based on above regex starts with 00 records will go to one route and 10 records will go to different 

 

New requirement


But now we are getting as below for other feed, so to determine which copybook to use for each row on the file, We would need to look at position 12:2 and route based on the value 

 

for example if 12,2 =00 then those records should go to route01 ,if 12,2=10 then those records should go to route02 

 

Sample content of flow file 
ABD0000000100000103200A08LL30003    0000000000220700840940
ABD0000000210000100060A08LL30003    0000000000220700840940

 

any idea how to Route on content of flow file to different routes to add required copy book  or any thoughts with different processors in nifi also helpful 

 

Thanks in advance 

 

3 REPLIES 3
Highlighted

Re: how to route fixed length records into multiple routes by using Route on Text in nifil

Super Guru

@sureshpathipati 

one way would be using Route Text processor to get first field value until first space.. and then apply NiFi expression language substring function on the extracted attribute value to check/route the line based on the value..

Regex to capture first field value until first space..

 

([^ ]+)(.*)

 

Another method:

 

As you are having fixed width file so by using ReplaceText Processor we can create delimited file and by using QueryRecord processor add new SQL query to check

substring(firstfield_name,Start,endposition)="00"

Add two new queries in QueryRecord processor to check substring value and route the records dynamically based on matched criteria.

-

If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

 

Re: how to route multiple fixed length(Complex Flat file )records into multiple routes by using Route on Text in nifi

New Contributor

@Shu_ashu  

 

 Please help me on this issue as our work got Hold due to not able to split the file into multiple files 

 

Option -1: I tried to use Route Text processor to get first field value until first space, But in file as i have millions of records, i need to get each group into individual flow 

I have record types of 25 categories , so for example in my file if i have 100 records for  record type =00 then those all 100 should go to route1 and  if i have 250 records for  record type =10 then those all 250 should go to route2 ...... etc

 

 If i use route text processor , how many i need to write routes  ? 

 

For  example i am writing as below ,its not giving my expected results

 

Route01 = ([^ ]+)(.*)

 

Sample content of flow file

 

ABD0000000100000103200A08LL30003       0000000000220700840940
ABD0000000210000100060A08LL30003       0000000000220700840940

 

for every record as seen above i have space after 30003 to next  value 

 

I am not getting as expected based on below regex.Could you please let me know  how i can i do this one 

 

Route on Text processor.jpg

Option 2. I also tried to use Extract Text Processor to extract first field value until first space as i have l multiple record types i am not getting all records extracted into my attribute to route in next processor 

Extract Text processor.jpg

 

 once we are able to get sub string from file content then we want to assign individual sachems as below as each record type has different schema

Routing rows to individual flows.jpg

 

 

Please suggest us if anything wrong i am doing as i am new to nifi

Re: how to route multiple fixed length(Complex Flat file )records into multiple routes by using Route on Text in nifi

New Contributor

Any help on this is a great help for us as we are unable to route the records based on substring value on the flow file content

Don't have an account?
Coming from Hortonworks? Activate your account here