Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi: How to split , convert and concat dynamic String to an insert into statement ?

Solved Go to solution
Highlighted

Nifi: How to split , convert and concat dynamic String to an insert into statement ?

New Contributor

Hi,

i have the following input single row FlowFiles:

Aug 9 10:34:12 sysxyz-p2 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 10 10:34:12 sysxxx.xxx.xx sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 11 10:34:12 sysxxx.xxx.xx sudo: pam_unix(sudo:session): session closed for user root
Aug 12 10:34:12 sysxxx.xxx.xyz sudo: pam_unix(sudo:session): session closed for user root
Aug 13 10:34:01 sysxxx.xxx.xx CRON[26312]: pam_unix(cron:session): session opened for user monadmin by (uid=0)

1. Split the Flow into timestamp, host, message fields:

timestamp host message

Aug 9 10:34:12 sysxyz-p2 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 10 10:34:12 sysxxx.xxx.xx sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 11 10:34:12 sysxxx.xxx.xx sudo: pam_unix(sudo:session): session closed for user root
Aug 12 10:34:12 sysxxx.xxx.xyz sudo: pam_unix(sudo:session): session closed for user root
Aug 13 10:34:01 sysxxx.xxx.xx CRON[26312]: pam_unix(cron:session): session opened for user monadmin by (uid=0)

2. Convert timestamp Format:

Aug 9 10:34:12 - > 'YYYY.MM.DD HH:MI:SS'

3. Concat the extracted files to insert into ... string:

example:

insert into table ('2017-08-09 10:34:12', 'sysxyz-p2', 'sudo: pam_unix(sudo:session): session opened for user root by (uid=0)';

Thanks for you help!
Timo

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Nifi: How to split , convert and concat dynamic String to an insert into statement ?

New Contributor

Got the solution:

1. Replace String with actual year

search value: <(.*?)>
Replacement Value: ${now():format('yyyy ')}

2. Extract Timestamp and Message
time = (^.{20})
message = (^.*$)

3. Replace String with insert into clause:
search value: (?s)(^.*$)
Replacement Value: insert into syslog values ('${time:substring(0,20):toDate('yyyy MMM dd HH:mm:ss'):format('yyyy-MM-dd HH:mm:ss')}','${message:substring(21):substringBefore(' ')}','${message:substring(21):substringAfter(' ')}');

1 REPLY 1

Re: Nifi: How to split , convert and concat dynamic String to an insert into statement ?

New Contributor

Got the solution:

1. Replace String with actual year

search value: <(.*?)>
Replacement Value: ${now():format('yyyy ')}

2. Extract Timestamp and Message
time = (^.{20})
message = (^.*$)

3. Replace String with insert into clause:
search value: (?s)(^.*$)
Replacement Value: insert into syslog values ('${time:substring(0,20):toDate('yyyy MMM dd HH:mm:ss'):format('yyyy-MM-dd HH:mm:ss')}','${message:substring(21):substringBefore(' ')}','${message:substring(21):substringAfter(' ')}');

Don't have an account?
Coming from Hortonworks? Activate your account here