You can execute any shell script using ExecuteProcess Processor in Nifi. For example I have a very simple shell script below which writes "Hello world" to a file. ... View more

For non-SSL enabled NiFi below should work: curl --tlsv1.2 -i -H 'Content-Type: application/json' -XPUT -d '{"id":"cdb54c9a-0158-1000-5566-c45ca9692f85","state":"RUNNING"}' localhost:8080/nifi-api/flow/process-groups/cdb54c9a-0158-1000-5566-c45ca9692f85 Start Process Group for SSL enabled NiFi : Generate access token: [root@nifi-ambari-01 ~]# curl --tlsv1.2 https://000.00.00.000:9091/nifi-api/access/token --data 'username=awadhwani&password=password' -k O/P: eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJjbj1BcnRpIFdhZGh3YW5pLG91PVBlb3BsZSxkYz1zbWUsZGM9aHd4IiwiaXNzIjoiTGRhcFByb3ZpZGVyIiwiYXVkIjoiTGRhcFByb3ZpZGVyIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiQXJ0aSBXYWRod2FuaSIsImtpZCI6MywiZXhwIjoxNDg5MjMyNDQyLCJpYXQiOjE0ODkxODkyNDJ9.17iHL3XX7Bw6dXv5lCByimu_asQOaSwW11o2IQEFO0s[root@nifi-ambari-01~]# Start PG by passing the above access token [root@nifi-ambari-01 ~]# curl --tlsv1.2 -ik -H 'Content-Type: application/json' -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJjbj1BcnRpIFdhZGh3YW5pLG91PVBlb3BsZSxkYz1zbWUsZGM9aHd4IiwiaXNzIjoiTGRhcFByb3ZpZGVyIiwiYXVkIjoiTGRhcFByb3ZpZGVyIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiQXJ0aSBXYWRod2FuaSIsImtpZCI6MywiZXhwIjoxNDg5MjMyNDQyLCJpYXQiOjE0ODkxODkyNDJ9.17iHL3XX7Bw6dXv5lCByimu_asQOaSwW11o2IQEFO0s' -XPUT -d '{"id":"2f092b07-0157-1000-0000-00005f526fbc","state":"RUNNING"}' https://000.00.00.000:9091/nifi-api/flow/process-groups/2f092b07-0157-1000-0000-00005f526fbc O/P: HTTP/1.1 200 OKDate: Fri, 10 Mar 2017 23:42:14 GMTServer: Jetty(9.3.9.v20160517)Cache-Control: private, no-cache, no-store, no-transformX-ProxiedEntitiesAccepted: trueDate: Fri, 10 Mar 2017 23:42:14 GMTContent-Type: application/jsonContent-Length: 63{"id":"2f092b07-0157-1000-0000-00005f526fbc","state":"RUNNING"}[root@nifi-ambari-01 ~]# Earlier before starting: After starting: Stop process Group for SSL enabled NiFi : (use the same token generated above) [root@nifi-ambari-01 ~]# curl --tlsv1.2 -ik -H 'Content-Type: application/json' -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJjbj1BcnRpIFdhZGh3YW5pLG91PVBlb3BsZSxkYz1zbWUsZGM9aHd4IiwiaXNzIjoiTGRhcFByb3ZpZGVyIiwiYXVkIjoiTGRhcFByb3ZpZGVyIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiQXJ0aSBXYWRod2FuaSIsImtpZCI6MywiZXhwIjoxNDg5MjMyNDQyLCJpYXQiOjE0ODkxODkyNDJ9.17iHL3XX7Bw6dXv5lCByimu_asQOaSwW11o2IQEFO0s' -XPUT -d '{"id":"2f092b07-0157-1000-0000-00005f526fbc","state":"STOPPED"}' https://000.00.00.000:9091/nifi-api/flow/process-groups/2f092b07-0157-1000-0000-00005f526fbc HTTP/1.1 200 OKDate: Fri, 10 Mar 2017 23:42:46 GMTServer: Jetty(9.3.9.v20160517)Cache-Control: private, no-cache, no-store, no-transformX-ProxiedEntitiesAccepted: trueDate: Fri, 10 Mar 2017 23:42:46 GMTContent-Type: application/jsonContent-Length: 63{"id":"2f092b07-0157-1000-0000-00005f526fbc","state":"STOPPED"}[root@nifi-ambari-01 ~] Earlier before stopping: After stopping: ... View more

While designing your flow, one will think of handling failures as well. Some may ignore (terminate) it while some may want to analyze their failed flow files. You can view/download any flowfile in the connection via NiFi UI. However if there are multiple flowfiles then this becomes little tedious since you have to do it for each flowfile. . In that case, I used below flow to put all failed flow files to a temp location on my nifi node and then access it from there for further analysis. In this I am generating random 2B text file and passing it to FTP server. If FTP server is unreachable it will put all flow files to failure/reject queue. You can list all files in this queue. I have viewed one such file in image below. Then all these files are passed to PutFile processor which is configured to write all the files it receives to /tmp/test location on the nifi node. Please note: There could be multiple ways to do this. This is the approach used by me to quickly download all failed flow files in my flow. download-failed-flowfiles-2.png download-failed-flowfiles-3.png download-failed-flowfiles-4.png download-failed-flowfiles-5.png ... View more

Using NiFi REST API for unsecured cluster is straight-forward like below: [root@<nifi-host> ~]# curl -v -X GET http://<nifi-host>:<port>/nifi-api/flow/current-user * About to connect() to <nifi-host> port <port> (#0) * Trying <IP address>... * Connected to <nifi-host> (<IP address>) port <port> (#0) > GET /nifi-api/flow/current-user HTTP/1.1 > User-Agent: curl/7.29.0 > Host: <nifi-host>:<port> > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 30 Jun 2017 22:15:09 GMT < X-Frame-Options: SAMEORIGIN < X-Frame-Options: SAMEORIGIN < X-Frame-Options: SAMEORIGIN < Cache-Control: private, no-cache, no-store, no-transform < Server: Jetty(9.4.3.v20170317) < Vary: Accept-Encoding, User-Agent < Date: Fri, 30 Jun 2017 22:15:09 GMT < Date: Fri, 30 Jun 2017 22:15:09 GMT < Content-Type: application/json < Content-Length: 439 < * Connection #0 to host <nifi-host> left intact {"identity":"anonymous","anonymous":true,"provenancePermissions":{"canRead":true,"canWrite":true},"countersPermissions":{"canRead":true,"canWrite":true},"tenantsPermissions":{"canRead":true,"canWrite":true},"controllerPermissions":{"canRead":true,"canWrite":true},"policiesPermissions":{"canRead":true,"canWrite":true},"systemPermissions":{"canRead":true,"canWrite":true},"restrictedComponentsPermissions":{"canRead":true,"canWrite":true}} However if this cluster is using Kerberos for authentication then the curl call will need a Kerberos authentication token as below: First do a kinit (using appropriate keytab/principal) on the nifi node you are logged into. Now get a token using below API call: token=`curl -k -X POST --negotiate -u : https://<nifi-node>:<port>/nifi-api/access/kerberos` Second you need to pass above generated token to the actual API call: curl -k --header "Authorization: Bearer $token" https://<nifi-host>:<port>/nifi-api/flow/cluster/summary ... View more

This is a known issue https://issues.apache.org/jira/browse/NIFI-3800 which is fixed in Apache NiFi 1.2.0 (HDF 2.1.4 and later) Workaround: You can downgrade your Java version to openjdk-1.8.0_121 if you are using something higher than that. ... View more

How to enable SSL for Storm UI on an unsecured cluster: 1. Generate keystore and certificate: root@beautiful-storm2 ~]# /usr/jdk64/jdk1.8.0_77/bin/keytool -genkeypair -alias certificatekey -keyalg RSA -validity 7 -keystore keystore.jks Enter keystore password: Re-enter new password: What is your first and last name? [Unknown]: storm What is the name of your organizational unit? [Unknown]: storm What is the name of your organization? [Unknown]: storm What is the name of your City or Locality? [Unknown]: storm What is the name of your State or Province? [Unknown]: storm What is the two-letter country code for this unit? [Unknown]: storm Is CN=storm, OU=storm, O=storm, L=storm, ST=storm, C=storm correct? [no]: yes Enter key password for <certificatekey> (RETURN if same as keystore password): Re-enter new password: 2. Add below properties via Ambari to custom storm-site ui.https.key.password=bigdata ui.https.keystore.password=bigdata ui.https.keystore.path=/keystore.jks (This is path to your keystore.jks generated in above step) ui.https.keystore.type=jks ui.https.port=8740 3. Sanity check: List your keystore: <code>[root@beautiful-storm2 ~]# /usr/jdk64/jdk1.8.0_77/bin/keytool -list -keystore keystore.jks Enter keystore password: Keystore type: JKS Keystore provider: SUN Your keystore contains 1 entry certificatekey, Dec 13, 2016, PrivateKeyEntry, Certificate fingerprint (SHA1): 4D:8A:C1:0E:8F:4A:4B:26:0C:27:4C:DD:39:96:00:83:CE:F4:B3:6E 4. Now hit Storm https UI: https://<storm nimbus IP address>:8740/index.html (http does not work now) 5. You will see below in storm ui.log: 2016-12-13 18:47:20.011 o.a.s.j.s.Server [INFO] jetty-7.x.y-SNAPSHOT 2016-12-13 18:47:20.036 o.a.s.j.s.h.ContextHandler [INFO] started o.a.s.j.s.ServletContextHandler{/,null} 2016-12-13 18:47:20.481 o.a.s.j.u.s.SslContextFactory [INFO] Enabled Protocols [SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] 2016-12-13 18:47:20.493 o.a.s.j.s.AbstractConnector [INFO] Started SslSocketConnector@0.0.0.0:8740 ... View more

Has anyone seen a situation where the Ranger Kafka plugin is enabled and connection test is also successful, however the json file created in the kafka policycache on the node is empty! Also tried using the REST API call to get the policy details but no output. Due to this, the kafka logs are being filled with below Error continuously: 2016-03-02 16:47:34,607 ERROR [kafka-request-handler-6] apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer (RangerKafkaAuthorizer.java:202) - Ranger Plugin returned null. Returning false ... View more