Hello   I am having an issue retrieving bucket contents into NiFi using the FetchS3Object processor. I have configured the ListS3 processor to pull in a json array containing all the bucket information, I'm happy that's working ok... example output:   I have configured the FetchS3Object processor with the same Accesskey, Secret etc but I get the following error, I haven't been able to find much online ref this error, can anyone see where I'm going wrong? FetchS3Object Config:   Any assistance would be greatly appreciated, I've seen a lot of questions asked about these two processors but haven't found anything about this particular error.   Cheers Andy ... View more

Hello all   Within NiFi, updateAttribute processor I am trying to change an attribute called 'hive_database' based on the value of another attribute called 'hostname'.   For instance if: #1: hostname = Mickey Mouse #2: hostname = James Bond   I want to use a 'contains' statement (or similar ) to change the output of new attribute 'hive_database'   property: hive_database value: ${hostname:contains('Mickey'):<OUTPUT>('cartoon')} <OR> ${hostname:contains('Mickey'):<OUTPUT>('movie')}   So that #1 output would be: property= hive_database value= cartoon   The parts I'm stuck on is the OUTPUT and OR parts above, do you know if this would be possible using the update attribute processor? I am trying to avoid having to use routeOnAttribute to break out multiple different 'hosname' values to send data to the relevant 'hive_database'    Hope this makes sense, unable to share screenshots etc as sensitive vales.   Any help greatly appreciated.  ... View more

Is there a processor that can detect hidden characters in NiFi?   In our test environment I am using an API invoke to deliver a JSON payload, for added security we want to filter out any hidden and potentially malicious text.   I have been trying to use RouteText, RouteOnContent and ExtractText processor to only allow alphanumeric and punctuation characters through but can't seem to get the regex to work when looking for 'uncommon' text or characters?   any help would be appreciated.   Andy  ... View more

Good afternoon   I am trying to make improvements to the way we make our Nifi flows by implementing Record processing.   Historically we: split text > get timestamps using regex > merge on 'corellation_id' (attribute from timestamp: format- yyyy-MM-dd-HH) > extract sourcetype using regex (flow splits at this point for each sourcetype) > merge again on 'correlation_id' > out to HDFS/Splunk etc   flow structured like this: Old Split/Merge Flow (each spine is a different sourcetype)   I recently watched this series of videos https://www.youtube.com/channel/UCcNsK56MZRXzPbRfe8sODWw I was particularly interested in the first one where Record Processing is used in place of split/merge/regex etc.    As a result I have started to re-design the flow above, the logs come into us in CEF format from a syslogListener (TCP).   My progress so far in processors:   GenerateFlowfile - using some copied data from original flow. SplitText - to split text/plain content into single logs (not figured out a way to get around splitting) ParseCEF - parses CEF into JSON format JoltTransformJSON - 'default' operation - to add some info to the JSON 'extension' content (feedname etc.)  JoltTransformJSON - ''shift' operation - to strip out unwanted lines, just keeping lines I later need as attributes  + 'raw_content' line (I haven't figured out how to achieve these two steps using one Jolt processor? is there a way?) EvaluateJsonPath - pulls the above attributes from the content. MergeRecord - Currently merging on 'sourcetype'.....   That's as far as I have got,  the flow is using a lot less processors and achieving (nearly) the same results. I have also compared the flow file linage duration, my new flow is around 5secs and the old style takes around 35secs (although could be tweaked).   WIP Process Record Flow     My question is, is there a way to merge on two attributes? 'sourcetype' and 'correlation_id' in my case I don't want different sourcetypes ("deviceVendor" in JSON format) merged together in HDFS and I also want to merge/group by timestamps (correlation_id) in one hour bins.   I want to try to get away from having 11 different spines for each different sourcetype, is the mergeRecord clever enough to group by common content and correlate from my time (correlation_id) attribute? or is this asking too much?   Hope this makes sense, any assistance would be greatly appreciated.  ... View more