Hi Obaid, please take a look at https://community.hortonworks.com/articles/49467/integrating-apache-nifi-with-aws-s3-and-sqs.html The missing piece in your description is SQS. ... View more

I have traced the root cause to be the low entropy on new VM instances, especially if they are headless (a typical cloud server today). To test if one is affected by the problem: head -1 /dev/urandom If the above command doesn't return immediately with some garbage output, but rather hangs, your server is affected by the problem. Java's SecureRandom initializes by reading from /dev/urandom. Some solution online suggest modifying JCE settings to use /dev/random, but this is less desirable: It's not guaranteed to always work There can be multiple JVMs, and admin may not always know which install is used by a specific process, and JAVA_HOME might not even be set, leaving him/her guessing It would require manual intervention, which hinders e.g. blueprints functionality (fully automated Ambari install) One solution which worked great for me and didn't require any JDK or code changes was to install the Haveged entropy daemon, which was designed for this problem specifically: http://www.issihosts.com/haveged/ Here's a process for CentOS 7. Similar steps are available for Ubuntu, etc. Haveged packages are in the EPEL repo. Need to install version-specific one for CentOS rpm -Uvh http://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm yum install -y haveged chkconfig haveged on service haveged start After this my secure NiFI cluster reliably restarts within expected time windows. ... View more

I wonder if you're looking for MiNiFi?https://nifi.apache.org/minifi/index.html ... View more

Hi Oliver. One would need to configure SSL context and add the self-signed certificate to the keystore used by it. Take a look at https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.ssl.StandardSSLContextService/index.html The components which support SSL will have a controller service property to reference. You would configure all SSL details and keystores in there, to be used by other processors. ... View more

Sam, the statements should be in the flowfile content/payload. Please see https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hive.PutHiveQL/index.html Also, it's not limited to INSERT statements only, any Hive DDL will be executed by this processor as well. This becomes very useful for creating new partitions in a table, etc. ... View more

Hi, I'm not yet sure if that can be done in JOLT spec. However, how about approaching it from the flow-based programming standpoint? The steps should be: Transform with JOLT - do all, no filtering EvaluateJsonPath into an attribute - extract values that will be used to make a decision RouteOnAttribute - use an expression against the above set of attributes to implement the filtering logic. E.g. discard or re-route records which don't match the criteria. ... View more