Hi! I've using NiFi for around a year, every day I like it more, very flexible, processors for every need, have some nice reusable templates, etc. So, here goes the question, why there is so more hype around Airflow? I didn't have a deep dive on the platform, watched some tutorials for ETL tasks, and still don't feel it as friendly as NiFi for the same tasks. I started to think, why do everyone defaults to Airfow and don't even try NiFi? should I be testing Airflow to do same tasks? Both seem to have a nice user base (it's a strange metric, but looking on linkedin for both platfoms Airflow gives me double the results 4k vs 2k). Meybe the NiFi support forum on cloudera is somehow biased, but I love to read some opinions. Thanks!     ... View more

I have my credential stored in AWS Secret Manager, use a PutLamda to retrieve it and send it to a jsonextract and then invokehttp processor where I use the credentials in the body of a post message to get a bearer token. Between those processors the secret travels in plaintext. Is there a way to hide the credentials? The idea is to rotate the credentials on aws and the only way to view those is using aws console, doing this we can log the access to the secret, when and who. Thank you!   ... View more

Hi i'm having a problem with self S2S reports, it's a NIFI 1.11.3 standalone (no cluster) linux deployment, this is the related configuration:   # Site to Site properties nifi.remote.input.host=localhost nifi.remote.input.secure=true nifi.remote.input.socket.port=10443 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec nifi.remote.contents.cache.expiration=30 secs # web properties # nifi.web.war.directory=./lib nifi.web.http.host= nifi.web.http.port= nifi.web.http.network.interface.default= nifi.web.https.host=my.domain.com nifi.web.https.port=8443 nifi.web.https.network.interface.default= nifi.web.jetty.working.directory=./work/jetty nifi.web.jetty.threads=200 nifi.web.max.header.size=16 KB nifi.web.proxy.context.path= nifi.web.proxy.host= nifi.security.keystore=./conf/keystore.jks nifi.security.keystoreType=jks nifi.security.keystorePasswd=xxxxxxxxxxxx nifi.security.keyPasswd=xxxxxxxxxxxxxxxx nifi.security.truststore=./conf/truststore.jks nifi.security.truststoreType=jks nifi.security.truststorePasswd=xxxxxxxxxxxxxxxxx nifi.security.user.authorizer=managed-authorizer nifi.security.user.login.identity.provider= nifi.security.ocsp.responder.url= nifi.security.ocsp.responder.certificate=   And the processor configuration, StandardRestrictedSSLContextService (using the self signed keystore from nifi that allows it to work securely) And the SiteToSiteBulletinReportingTask: with that configuration I receive this errors: SiteToSiteBulletinReportingTask[id=017111a7-83c2-1c18-25d3-ad4d5f780eb1] Error running task SiteToSiteBulletinReportingTask[id=017111a7-83c2-1c18-25d3-ad4d5f780eb1] due to org.apache.nifi.processor.exception.ProcessException: Failed to send Bulletins to destination due to IOException:null SiteToSiteBulletinReportingTask[id=017111a7-83c2-1c18-25d3-ad4d5f780eb1] org.apache.nifi.remote.client.PeerSelector@1e7445c6 Unable to refresh Remote Group's peers due to null If i change http to https I receive: SiteToSiteBulletinReportingTask[id=017111a7-83c2-1c18-25d3-ad4d5f780eb1] Error running task SiteToSiteBulletinReportingTask[id=017111a7-83c2-1c18-25d3-ad4d5f780eb1] due to org.apache.nifi.processor.exception.ProcessException: Failed to send Bulletins to destination due to IOException:sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target   To test if Nifi could see itself I tried this, it's about the self signed certificate, so it seems ok:   curl https://host123.internal:8443/nifi curl: (60) SSL certificate problem: self signed certificate in certificate chain More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.   Ignoring the certificate I reach Jetty:   curl --insecure -I https://host123.internal:8443/nifi HTTP/1.1 302 Found Date: Fri, 15 May 2020 18:21:49 GMT Location: https://host123.internal:8443/nifi/ Content-Length: 0 Server: Jetty(9.4.19.v20190610)    To secure the connection on installation (http to https) I used the security toolkit and copied the files to their locations under the nifi/conf folder as seen on the configuration file at top of the post:   ./bin/tls-toolkit.sh standalone -n 'localhost(1)' -C 'CN=my_user,OU=NIFI' -O -o ../security_output   Any idea of what can I be doing wrong with the S2S configuration? Thanks! ... View more

I am working in an AWS environment with NifI on the core of my ETL process, this is a new installation so I have no monitoring setup yet. Because this is a fresh installation I want to hear the practices you use to monitor NiFi, I used to put some Slack messages for FAIL and SUCCESS but there are very long flows and I don't seem viable putting a slack call on every possible point of failure, those can be a LOT of processors.   How do you do? insert to mysql to monitor start/end and alarm if something doesn't end? LOT of slack/mail processors? Dashboards (how do you feed them)? any suggestions?   Thank you! ... View more

Hi, I have an API call to a service, that service have different type of processes, the API cannot filter them. So when I query the API it returns JSON with 30 common keys, and another 15 that varies for each case, the question is, is there a way to automatically adapt the schema so I can use MergeRecords? The result goes to S3. There are a lot of cases, and it seems they are adding more once in a while, is there a "generic" way to tell MergeRecords to "if you see any change, make another group", I don't have a key with value to determine the group. Any idea? Thanks! ... View more