@SandeepG01    Ahh no fun with bad filenames.  Space in filename is highly not recommended in these days and times.  That said,  a solution you might try is to \  (backslash) the space.   Especially in the context of passing the filename in flowfile attributes.   If you still need to allow spaces and cannot resolve upstream (do not use spaces), i might suggest submitting your experience over on the NiFI jira as a bug:   https://issues.apache.org/jira/projects/NIFI/issues   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

@stevenmatisonthe solution that i find is to get the Oauth2 token from slaesforec by using command Curl. like is explained in this page : https://www.jitendrazaa.com/blog/salesforce/using-curl-with-salesforce-rest-api/ So i create a ExecuteProces NiFi Procesor. And as parameter i put : the file C:/loginInfo.txt contains : grant_type=password& client_id= 3MVG9iTxZANhwsdsdsdsdspr0LstjR3sRat & client_secret=21961212323233121943 & username=jitendra.zaa@demo.com & password=myPWDAndSecurityToken and the i get a response with the authentication token 🙂 (you can use the cmd command Curl -x post -d @LoginInfo.txt Https://test.salesforce.com/.... to test the connection between the local machine and salesforce )   ... View more

The problem is that you need something to store the dynamic schemas in.  That is where the Schema Registry comes in as it provides a UI and api to add/update/delete schemas.  These can then be refrenced from NiFi.       It looks like AvroSchemaRegistry allows you to do the similar,  minus the ui/api.   So you would need to create your schema in your flow, as attribute, and send that to AvroRecorderReader configured against AvroSchemaRegistry.  You could use some other data store to store these schemas, but you would need to pull them out into an attribute of the same name configured in the Reader and Registry.   https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-registry-nar/1.12.1/org.apache.nifi.schemaregistry.services.AvroSchemaRegistry/index.html   The latter method does not give you a way to manage all the schemas, which is why I reference the Hortonworks Schema Registry which does include ability to manage, version actual schemas.       ... View more

@P_Rat98   Per our PM discussion,  in your flow use DetectDuplicate before sending an email.   This should rate limit the # of messages you send based on your configuration of detectDuplicate.  Additionally when this is linked in your flow, and duplicates are auto-terminated it will drain the flow and stop it from filling up the queue.    Also as suggested you can chose to retain the duplicates, but move them into a much bigger Queue which isnt going to back up the main flow.  Then once you see the email, you can go look at flow, see what flowfiles were causing issues, and take some corrective action.   If you really need to monitor flow for a queue being full,  you would need to use the nifi API to check the status of the queue.  This maybe more work than it is worth, when you can solve as above much easier.  However, i would recommend you check the api out,  there are a lot of api capabilities and I am beginning to use nifi api calls within my flow to monitor, stop, start, and take actions automatically that would normally require a human doing them in the UI.   If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.   Thanks, Steven ... View more

After I have not been able to find a solution that would be easy to implement inside of NiFi, I've written a small perl (yuk) script that can be used to adjust timestamps in a CSV file to be in ISO8601 format.  Maybe it is useful to someone else: #!/bin/perl -w # This perl script adds timezone information to timestamps without a # timezone. All timestamps in the input file that follow the format # "YYYY-MM-DD HH:MM:SS" are converted to ISO8601 timestamps. use strict; use DateTime::Format::Strptime; my $time_zone = 'Europe/Amsterdam'; my $parser = DateTime::Format::Strptime->new( pattern => '%Y-%m-%d %T', time_zone => $time_zone ); my $printer = DateTime::Format::Strptime->new( pattern => '%FT%T%z', time_zone => $time_zone ); while (<>) { s/(?<=")(\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d)(?=")/ my $dt = $parser->parse_datetime($1); $printer->format_datetime($dt); /ge; print; }   ... View more

At the end of the day I used Steve Matison and 'PVVK's solutions.  First I used PVVK's to modify my flowfile into valid html, then I successfully was able to use SplitJson processor to break up the flowfiles cleanly.  Then since there were multiple 'logEvent' entries in some of the records I saved off the header items as attributes and then used SteveMatison's method to break down each logEvent and added the header attributes back to it for processing.   Thanks all! ... View more

@aniket5003  Check out the following post for a few different ways to locate or update the password:   https://community.cloudera.com/t5/Support-Questions/Default-password-for-Metron-UI-Version-0-4-0/td-p/199277     If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.     Thanks, Steven ... View more