Hi, I get a rar file from a SFTP server, have it in a flowfile I'm trying to run unrar on the flowfile with an executeStreamCommand withtout persisting it with no luck, tried setting working directory to /tmp too, unrar is in path an executing: Is there any way to do this or I have to dump the files to disk, decompress them there with this processor and get the files again? Thanks! ... View more

Maybe it's not implemented, I'm logged in on Nifi, commit a processor, go into nifi-registry and it shows the commit was made by anonymous (I have no authentication on nifi-registry), but I have OpenID in NiFi, should it use this user to log the commit? Thank you! ... View more

Hi i'm having a problem with self S2S reports, it's a NIFI 1.11.3 standalone (no cluster) linux deployment, this is the related configuration:   # Site to Site properties nifi.remote.input.host=localhost nifi.remote.input.secure=true nifi.remote.input.socket.port=10443 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec nifi.remote.contents.cache.expiration=30 secs # web properties # nifi.web.war.directory=./lib nifi.web.http.host= nifi.web.http.port= nifi.web.http.network.interface.default= nifi.web.https.host=my.domain.com nifi.web.https.port=8443 nifi.web.https.network.interface.default= nifi.web.jetty.working.directory=./work/jetty nifi.web.jetty.threads=200 nifi.web.max.header.size=16 KB nifi.web.proxy.context.path= nifi.web.proxy.host= nifi.security.keystore=./conf/keystore.jks nifi.security.keystoreType=jks nifi.security.keystorePasswd=xxxxxxxxxxxx nifi.security.keyPasswd=xxxxxxxxxxxxxxxx nifi.security.truststore=./conf/truststore.jks nifi.security.truststoreType=jks nifi.security.truststorePasswd=xxxxxxxxxxxxxxxxx nifi.security.user.authorizer=managed-authorizer nifi.security.user.login.identity.provider= nifi.security.ocsp.responder.url= nifi.security.ocsp.responder.certificate=   And the processor configuration, StandardRestrictedSSLContextService (using the self signed keystore from nifi that allows it to work securely) And the SiteToSiteBulletinReportingTask: with that configuration I receive this errors: SiteToSiteBulletinReportingTask[id=017111a7-83c2-1c18-25d3-ad4d5f780eb1] Error running task SiteToSiteBulletinReportingTask[id=017111a7-83c2-1c18-25d3-ad4d5f780eb1] due to org.apache.nifi.processor.exception.ProcessException: Failed to send Bulletins to destination due to IOException:null SiteToSiteBulletinReportingTask[id=017111a7-83c2-1c18-25d3-ad4d5f780eb1] org.apache.nifi.remote.client.PeerSelector@1e7445c6 Unable to refresh Remote Group's peers due to null If i change http to https I receive: SiteToSiteBulletinReportingTask[id=017111a7-83c2-1c18-25d3-ad4d5f780eb1] Error running task SiteToSiteBulletinReportingTask[id=017111a7-83c2-1c18-25d3-ad4d5f780eb1] due to org.apache.nifi.processor.exception.ProcessException: Failed to send Bulletins to destination due to IOException:sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target   To test if Nifi could see itself I tried this, it's about the self signed certificate, so it seems ok:   curl https://host123.internal:8443/nifi curl: (60) SSL certificate problem: self signed certificate in certificate chain More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.   Ignoring the certificate I reach Jetty:   curl --insecure -I https://host123.internal:8443/nifi HTTP/1.1 302 Found Date: Fri, 15 May 2020 18:21:49 GMT Location: https://host123.internal:8443/nifi/ Content-Length: 0 Server: Jetty(9.4.19.v20190610)    To secure the connection on installation (http to https) I used the security toolkit and copied the files to their locations under the nifi/conf folder as seen on the configuration file at top of the post:   ./bin/tls-toolkit.sh standalone -n 'localhost(1)' -C 'CN=my_user,OU=NIFI' -O -o ../security_output   Any idea of what can I be doing wrong with the S2S configuration? Thanks! ... View more

Hi, I just deployed a new NiFi instance, I want to have a good management of credentials, there are some that I can store in the Parameter Context, but there are others that have to be posted using an invokeHTTP (like oauth secrets), not a sensitive data field, and don't want to sit there in plaintext.   How do you store this tipe of credentials? I thought about AWS secrets manager, but don't know if I can access them from NiFi, I can have a file on S3 too, but again it's stored in plain text, not decoded on usage.   I think I need something like Parameter Context that let me get the unencrypted value when I need to use it outside sensitive fields. Any idea? how do you manage this? unencrypt a file on the fly? Thank you! ... View more

I am working in an AWS environment with NifI on the core of my ETL process, this is a new installation so I have no monitoring setup yet. Because this is a fresh installation I want to hear the practices you use to monitor NiFi, I used to put some Slack messages for FAIL and SUCCESS but there are very long flows and I don't seem viable putting a slack call on every possible point of failure, those can be a LOT of processors.   How do you do? insert to mysql to monitor start/end and alarm if something doesn't end? LOT of slack/mail processors? Dashboards (how do you feed them)? any suggestions?   Thank you! ... View more

Hi, i'm having a problem with this, I decompress a gz, it gives me 10k+ json files with 1 element each. I make a transformation with JOLT and SHIFT only to reorder the fields. Now I have this problem, when trying to InferAvroSchema, it founds spaces on some keys, the keys can change, so I don't want to force the schema. How can I replace all the keys with spaces (replace with "_") and keeping the current tree of the JSON? don't want to hardcode anything if possible, because some of the keys are actions, and when a new element is created on the app wich generates the json, a new child key is added.   My last try was this: https://community.cloudera.com/t5/Support-Questions/How-to-do-JOLT-replace-on-all-JSON-keys-in-Nifi/td-p/178416 But it doesn't modify the child elements, only root ones, and if I tell on wich element is what I want to replace it "shift" it to root and all the other elements dissapear.   The JSON is something like:   { "server_received_time": "2019-08-01", "app": 8, "event_id": int, "event_properties":{"event buy":"one", "event try":"yes"} "event_state":{"event profile":"absent", "event_fail":"yes", "event text":"will try again"} } The json has a lot more keys, a space will not appear in the root elements but it can in the child elements, so I only need to replace those keeping the others intact. The transformations I try make me lost the schema format or name all the child elements (that I want to keep this way because I dont know if new ones will appear.   The idea is to remove this spaces on keys so I can InferAvroSchema without loosing elements and continue to process and store them Any idea? Thank you! ... View more

Hi, I have an API call to a service, that service have different type of processes, the API cannot filter them. So when I query the API it returns JSON with 30 common keys, and another 15 that varies for each case, the question is, is there a way to automatically adapt the schema so I can use MergeRecords? The result goes to S3. There are a lot of cases, and it seems they are adding more once in a while, is there a "generic" way to tell MergeRecords to "if you see any change, make another group", I don't have a key with value to determine the group. Any idea? Thanks! ... View more