Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Doing Math and replacingtext Line-byLine on ExtractText

Explorer

I have the below flowfile contents that I would like to manipulate without splitting the flowfile.  There may be a million of these lines.

{"ConfigBuild":"1007.8.0010703.9","ConfigStateHash":"1386539727","ContextProcessId":"317927403096194486","ContextThreadId":"0","ContextTimeStamp":"1603206060.314","Entitlements":"15","RawProcessId":"161622","TargetProcessId":"317927403096194486","aid":"1c41ff1d8eff49809ddb44d997ef2fa6","aip":"204.194.122.230","cid":"dff76f718745469994bd39c6a38579aa","event_platform":"Lin","event_simpleName":"TerminateProcess","id":"10c7f679-12e5-11eb-a7cc-0ab5dad5c022","name":"TerminateProcessLinV2","timestamp":"1603206060328"}
{"CommandLine":"logger -p cron.notice -t run-parts(/etc/cron.hourly)[115758] starting 0anacron","ConfigBuild":"1007.8.0010703.9","ConfigStateHash":"1386539727","Entitlements":"15","GID":"0","ImageFileName":"/usr/bin/logger","MD5HashData":"79922c9c30f225086d8a7331f54427ae","ParentBaseFileName":"bash","ParentProcessId":"317927531071092650","ProcessEndTime":"1603206061.046","ProcessGroupId":"317927531071092650","ProcessStartTime":"1603206061.045","RGID":"0","RUID":"0","RawProcessId":"115761","SHA1HashData":"0000000000000000000000000000000000000000","SHA256HashData":"9ed6a92880974f41eef1dd11b80d883c26faf1951a4b6a12720ccce8d195042c","SVGID":"0","SVUID":"0","SessionProcessId":"317927531071092652","SourceProcessId":"317927531071092650","SourceThreadId":"0","TargetProcessId":"317927531071092653","UID":"0","aid":"3a6f13022fff41a7b016537ea3862935","aip":"204.194.122.230","cid":"dff76f718745469994bd39c6a38579aa","event_platform":"Lin","event_simpleName":"ProcessRollup2","id":"11869eb7-12e5-11eb-9385-02ed8d7af942","name":"ProcessRollup2LinV6","timestamp":"1603206061577"}
{"CommandLine":"cat /var/spool/anacron/cron.daily","ConfigBuild":"1007.8.0010703.9","ConfigStateHash":"1386539727","Entitlements":"15","GID":"0","ImageFileName":"/usr/bin/cat","MD5HashData":"98b51c2a3efd088fe2ec632980c28e6c","ParentBaseFileName":"bash","ParentProcessId":"317927530540353195","ProcessEndTime":"1603206061.179","ProcessGroupId":"317927530540353191","ProcessStartTime":"1603206061.178","RGID":"0","RUID":"0","RawProcessId":"11630","SHA1HashData":"0000000000000000000000000000000000000000","SHA256HashData":"6825557ed76e1055d07a2cfd6ef753a931885daf4e102414724f9e812f48d538","SVGID":"0","SVUID":"0","SessionProcessId":"317927530540353193","SourceProcessId":"317927530540353195","SourceThreadId":"0","TargetProcessId":"317927532637505196","UID":"0","aid":"0a127135ef064baf8a4004991c43f2b7","aip":"204.194.122.230","cid":"dff76f718745469994bd39c6a38579aa","event_platform":"Lin","event_simpleName":"ProcessRollup2","id":"1199c591-12e5-11eb-b366-064134f5d9c0","name":"ProcessRollup2LinV6","timestamp":"1603206061703"}

In each row there is a timestamp, for example in the first row it is "timestamp":"1603206061703"

I'm doing an ExtractText manipulate the flowfile line by line with the goal of each line replaced with

 {"time": "THE_TIMESTAMP_VALUE/1000", "attribute a", "attribute_stuff_a", "event": $1}

So I want to do a calculation based on a value found in the line-by-line and add a couple attributes.  I've been able to do everything but add the calculated "time" value.

 

I tried separating the ExtractText and adding all the attributes in one and then adding the timestamp using the following ExtractText

2020-10-20_14-37-27.png

 
 
 
1 REPLY 1

Explorer

Maybe this can't be done with ExtractText and there are other options

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.