@Aditya Sirna Can you please suggest how to remove params. As I tried but unable to save the configuration and restart storm. ... View more

Hello, Certificates were not created properly. I have compared another working certificate with this certificate and found mismatch. I have verified certificate through openssl command and then I have copied required certificates from other working application server to issued one. Issue is resolved now but still unable find why below commands doesn't works on server sudo /usr/jdk64/jdk1.8.0_112/bin/keytool -import -trustcacerts -noprompt -storepass xxxx -alias abc-sha2 -file /home/ec2-user/abc-sha2.cer -keystore /usr/jdk64/jdk1.8.0_112/jre/lib/security/cacerts ... View more

@JZ I have replaced keystore/truststore with below commands. Where Ab-ssl-sha2.cer is a certificate used to convert to keystore/truststore. /usr/jdk64/jdk1.8.0_112/bin/keytool -import -file /home/Ab-ssl-sha2.cer -keystore /etc/nifi/3.0.1.1-5/0/keystore.jks -alias keystore_internal /usr/jdk64/jdk1.8.0_112/bin/keytool -import -file /home/Ab-ssl-sha2.cer -keystore /etc/nifi/3.0.1.1-5/0/truststore.jks -alias truststore_internal Can you please suggest, where need to do changes? ... View more

@JZ I'm facing similar error. I am using Nifi 1.2.0. with HTTPS and LDAPS. Recently I have updated the certificated and started facing below error. I can access Nifi webgui. When I'm trying to copy files from Nifi gui to S3, I'm getting the below errors. Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) at sun.security.validator.Validator.validate(Validator.java:260) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491) ... 50 common frames omitted Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) <br> I have kept, cacert files in java path /usr/jdk64/jdk1.8.0_112/jre/lib/security/cacerts<br> and keystore/trustore files /etc/nifi/3.0.1.1-5/0/keystore.jks /etc/nifi/3.0.1.1-5/0/truststore.jks<br> I not getting clear, where exactly valid certification path is located. If you know, please suggest. ... View more

I Have faced similar error. In my case Nifi was running fine but in cluster, nodes was not connected. In nifi-app.log found below errors. ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss Solution - ZK services was not running. I have started first then started Nifi cluster. Now Nifi nodes are connected properly in a cluster and cluster is running fine. ... View more

@JZ Can you please suggest - as per https://community.hortonworks.com/questions/167502/nifi-ssl-unable-to-find-valid-certification-path-t.html ... View more

Thanks @Jay Kumar SenSharma Do we need to use --mysqlschema=ambari instead of --postgresschema=ambari -s at the end will help to do rest of the ambari server installation? ... View more

@Jay Kumar SenSharma What about customized installation? I wanted to select mysql instead of postgres and related to further options. ... View more

I want user authentication(read only access) to run sql queries over amazon athena. Athena will be used to run read only queries over s3. Apache Hue will be used for user authentication instead of direct access to Athena. So I think with the help of Apache Hue, this can be achievable. But unable to find the clear way to do apache hue integration with Athena. Similarly Nifi needs to get integrated with Athena. ... View more

HDF 3.0.1.1 is used ... View more

I followed the steps given above but getting below error. 8080 port is not listening as ambari-server service is not running. I tried to start the service but it fails. Please suggest curl -H "X-Requested-By: ambari" -X POST -u admin:admin http://192.168.10.10:8080/api/v1/blueprints/single-node-hdp-cluster -d @cluster_configuration.json curl: (7) Failed connect to 192.168.10.10:8080; Connection refused ... View more

First you've to change the hostname at your node level. Then at the time of hosts registration of ambari setup through UI, you've to mention correct hostname. You can change it from there. ... View more

@Kuldeep Kulkarni Can you please elaborate step 4? How do you created blueprint? Where need to put this file "api/v1/blueprints" from this url "http://<ambari-hostname>:8080/api/v1/blueprints/<blueprint-name" ? ... View more

@Matt Clarke Thank you I found the solution. Issue is fixed now. In my case, one of LDAP username is 'dvteam' but in LDAP database there was full description of username as 'architecture dev team, locations, team details, etc'. Error messages I found in nifi-user.log. is 'architecture dev team' user was trying to authenticate with nifi nodes. Authentication was successful but authorizations not happening. The username which I've mentioned in initial admin identity was 'dvteam'.(cn=dvteam,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com) Then as per logs, I changed it to (cn=architecture dev team,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com) Also there was some mismatch about host names in node identities section. 'hostname -f' shows a hostname ip-zz-xx-ec2-internal. So, I have given 'ip-zz-xx-ec2-internal' in node identities section but that was not working. Then I have changed the hostnames to 'nifi1.abc.local' and mentioned in node identities. In 'Template for login-identity-providers.xml' I've made some changes. Earlier I had set 'use_username' in '<property name="Identity Strategy">USE_DN</property>' this section. later I've changed to use_dn. because as per nifi-user log authentication is happening with LDAP user 'architecture dev team'. So in my case user_username was not working for authentications. Every configurations changes I used to remove authorizations.xml and users.xml file from my all nifi nodes. Also There was confusion on about 'OU' in Node identities section. What does it mean OU in node identities section? I don't know yet. Later I've mentioned 'OU=nifi' and also gave host names as 'nifi1.abc.local' , 'nifi2.abc.local', etc. I have added AD/LDAP user in Initial Admin Identity(cn=architecture dev team,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com) After setting above all, I was facing an error about setting nifi.security.identity.mapping.pattern.dn. There was a challenge about the pattern definition. There was 4 'ou' I have defined in initial admin identities and login-identity-providers.xml. So I've used below pattern and it worked well. ^cn=(.?),ou=(.?),ou=(.?),ou=(.?),ou=(.?),dc=(.?),dc=(.?)$ Note: I have removed Ranger completely. Thanks, Suraj ... View more

I found the solution. Issue is fixed now. In my case, one of LDAP username is 'dvteam' but in LDAP database there was full description of username as 'architecture dev team, locations, team details, etc'. Error messages I found in nifi-user.log. is 'architecture dev team' user was trying to authenticate with nifi nodes. Authentication was successful but authorizations not happening. The username which I've mentioned in initial admin identity was 'dvteam'.(cn=dvteam,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com) Then as per logs, I changed it to (cn=architecture dev team,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com) Also there was some mismatch about host names in node identities section. 'hostname -f' shows a hostname ip-zz-xx-ec2-internal. So, I have given 'ip-zz-xx-ec2-internal' in node identities section but that was not working. Then I have changed the hostnames to 'nifi1.abc.local' and mentioned in node identities. In 'Template for login-identity-providers.xml' I've made some changes. Earlier I had set 'use_username' in '<property name="Identity Strategy">USE_DN</property>' this section. later I've changed to use_dn. because as per nifi-user log authentication is happening with LDAP user 'architecture dev team'. So in my case user_username was not working for authentications. Every configurations changes I used to remove authorizations.xml and users.xml file from my all nifi nodes. Also There was confusion on about 'OU' in Node identities section. What does it mean OU in node identities section? I don't know yet. Later I've mentioned 'OU=nifi' and also gave host names as 'nifi1.abc.local' , 'nifi2.abc.local', etc. I have added AD/LDAP user in Initial Admin Identity(cn=architecture dev team,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com) After setting above all, I was facing an error about setting nifi.security.identity.mapping.pattern.dn. There was a challenge about the pattern definition. There was 4 'ou' I have defined in initial admin identities and login-identity-providers.xml. So I've used below pattern and it worked well. ^cn=(.?),ou=(.?),ou=(.?),ou=(.?),ou=(.?),dc=(.?),dc=(.?)$ Note: I have removed Ranger completely. Thanks, Suraj ... View more

Thank you @Andy LoPresto I found the solution. Issue is fixed now. In my case, one of LDAP username is 'dvteam' but in LDAP database there was full description of username as 'architecture dev team, locations, team details, etc'. Error messages I found in nifi-user.log. is 'architecture dev team' user was trying to authenticate with nifi nodes. Authentication was successful but authorizations not happening. The username which I've mentioned in initial admin identity was 'dvteam'.(cn=dvteam,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com) Then as per logs, I changed it to (cn=architecture dev team,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com) Also there was some mismatch about host names in node identities section. 'hostname -f' shows a hostname ip-zz-xx-ec2-internal. So, I have given 'ip-zz-xx-ec2-internal' in node identities section but that was not working. Then I have changed the hostnames to 'nifi1.abc.local' and mentioned in node identities. In 'Template for login-identity-providers.xml' I've made some changes. Earlier I had set 'use_username' in '<property name="Identity Strategy">USE_DN</property>' this section. later I've changed to use_dn. because as per nifi-user log authentication is happening with LDAP user 'architecture dev team'. So in my case user_username was not working for authentications. Every configurations changes I used to remove authorizations.xml and users.xml file from my all nifi nodes. Also There was confusion on about 'OU' in Node identities section. What does it mean OU in node identities section? I don't know yet. Later I've mentioned 'OU=nifi' and also gave host names as 'nifi1.abc.local' , 'nifi2.abc.local', etc. I have added AD/LDAP user in Initial Admin Identity(cn=architecture dev team,ou=xx,ou=xx,ou=xx,ou=xx,dc=abc,dc=com) After setting above all, I was facing an error about setting nifi.security.identity.mapping.pattern.dn. There was a challenge about the pattern definition. There was 4 'ou' I have defined in initial admin identities and login-identity-providers.xml. So I've used below pattern and it worked well. ^cn=(.?),ou=(.?),ou=(.?),ou=(.?),ou=(.?),dc=(.?),dc=(.?)$ Thanks, Suraj ... View more

@Andy LoPresto Hello Andy, I'm also getting similar problem. Can you please suggest how to do changes in users.xml for below user identifier. In which section need to define node details? <users> <user identifier="eef8e5ba-348e-3bc3-877f-7d48d870a862" identity="dbr-lncn-01.int.payoff.com, OU=NIFI"/> <user identifier="7a6460a6-2944-388c-a947-1f098e13c112" identity="dbr-lncn-02.int.payoff.com, OU=NIFI"/> </users> ... View more

@Geoffrey Shelton Okot /proxy is also tested earlier. Anonymous users is already set to false . [root@ip- ~]# cat /etc/nifi/3.0.1.1-5/0/authorizers.xml|grep -i anon <property name="Allow Anonymous">false</property> [root@ip-10-248-13-199 ~]# [root@ip- ~]# cat /usr/hdf/current/nifi/conf/nifi.properties |grep -i login nifi.login.identity.provider.configuration.file=/usr/hdf/current/nifi/conf/login-identity-providers.xml nifi.security.user.login.identity.provider=ldap-provider [root@ip- ~]# Can you suggest, from where anonymous user is coming? Is it from Ranger policy(LDAP) or is it a default user in Nifi ? ... View more

@Jobin George Can you please suggest how to remove anonymous user by getting default login to Nifi UI. I can login Nifi UI with my LDAP user but Nifi is also accessible with anonymous user without password. I wanted to disable it. In ranger policy if I remove {user} from user section then I cannot login Nifi UI with LDAP user and also it doesn't get default login with anonymous. Please suggest. Brief description is mentioned on below link. https://community.hortonworks.com/questions/142667/how-to-give-permissions-to-users-to-access-nifi-ui.html?childToView=145984#answer-145984 ... View more

@Geoffrey Shelton Okot Thank you for the clearing the doubt. But /var/lib/nifi/conf/users.xml and /var/lib/nifi/conf/authorizations.xml files are not created as well. Can you please suggest what exactly need to do to create them. Also can you please suggest that, is it possible to remove the anonymous default login ? Am I searching in right direction, because I have tried multiple ways but still I do not get the expected result. Anything I'm missing in configurations? ... View more

@Geoffrey Shelton Okot @Pierre Villard @Matt Clarke @Jonas Straub @Yolanda M. Davis Complete setup scenario: In a cluster ( HDF 3.0.1 - Ambari, Nifi, zookeeper, Ranger, DB - Mysql ), all componants are running fine. Nifi UI is configured with HTTPS but do not get successful login page in Nifi UI. (To configure Nifi UI with HTTPS - converted keystore.jks file into pks12 format and loaded the pks12 file into browser) Ranger is integrated with LDAP successfully. Ranger UI is accessible through LDAP users. Copied Nifi's keystore and trustore file from Nifi server to Ranger server to build the trust between them. (copied at /usr/hdf/current/ranger-admin/conf) Then Ranger Policy is created and added LDAP users in it. Also given Read and Write permissions to added LDAP users in Ranger policy. Note: Ranger UI is not on HTTPS Now there is one issue. If I add some LDAP users in the Ranger policy then I cannot access Nifi UI. I got 'insuffecient permissions and unable to access the page' kind of errors. Logs shows authentication is success for LDAP users but authorization is failed. Getting below error as per screenshot. But If I gave {users} in user's section of Ranger Policy, then I can login Nifi UI with my LDAP user. Also Nifi UI can be accessible by anonymous user. I dont know from where anonymous user is coming. But If I remove {user} from user section then I cannot login with LDAP user as well as anonymous user. As per some blogs, I found it could be the related from authorizations.xml and users.xml files. But those files are missing from Nifi servers. How to create/generate those files ? Nifi Config. nifi.security.user.login.identity.provider ldap-provider Template for login-identity-providers.xml <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Identity Strategy">USE_USERNAME</property> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">CN=zxc_oi,OU=fox,DC=abc,DC=com</property> <property name="Manager Password">xxx</property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://ldap.abc.com:389</property> <property name="User Search Base">DC=abc,DC=com</property> <property name="User Search Filter">sAMAccountName={0}</property> <property name="Authentication Expiration">12 hours</property> </provider> Please suggest, How I can remove anonymous user by getting logged in Nifi UI. I have gone through below links and some others as well. https://community.hortonworks.com/articles/60001/hdf-20-integrating-secured-nifi-with-secured-range.html https://pierrevillard.com/2017/01/24/integration-of-nifi-with-ldap/comment-page-1/#comment-1114 If any more details is required, please let us know. ... View more

@Geoffrey Shelton Okot @Matt Clark Below files are not in Nifi's conf folder. authorizations.xml users.xml Those files are not found anywhere on the system. Do we need those files ? If yes then how to generate/create both .xml files ? Also as per https://pierrevillard.com/2017/01/24/integration-of-nifi-with-ldap/comment-page-1/ user's should have specific permissions to access the nifi. I have full permissions to the users but still I'm getting the same permission issue. What could be the issue? ... View more

Hello @Geoffrey Shelton Okot Do you find anything related to this issue? ... View more

/data/log/nifi/nifi-user.log <em>==> /data/log/nifi/nifi-user.log <== 2017-10-30 06:11:54,514 WARN [main] o.a.n.a.util.IdentityMappingUtil Identity Mapping property nifi.security.identity.mapping.pattern.kerb was found, but was empty 2017-10-30 06:11:55,605 WARN [main] o.a.n.a.util.IdentityMappingUtil Identity Mapping property nifi.security.identity.mapping.pattern.kerb was found, but was empty 2017-10-30 06:11:55,652 WARN [main] o.a.n.a.util.IdentityMappingUtil Identity Mapping property nifi.security.identity.mapping.pattern.kerb was found, but was empty 2017-10-30 06:13:06,886 INFO [NiFi Web Server-22] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response. 2017-10-30 06:13:07,135 INFO [NiFi Web Server-94] o.a.n.w.a.c.AccessDeniedExceptionMapper anonymous does not have permission to access the requested resource. Unable to view the user interface. Returning Unauthorized response. 2017-10-30 06:14:03,287 INFO [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (eyJhcGxlLG91PU1VTV9NdW1iYWkgSW5kaWEsb3U9QXNpYSxvdT1QZW9wbGUgYW5kIFdvcmtzdGF0aW9ucyxkYz1tb3Jua) GET https://10.248.13.199:9091/nifi-api/flow/current-user (source ip: 10.90.18.237) 2017-10-30 06:14:03,290 INFO [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for cn=Danny Leo,ou=Asia,ou=People and Workstations,dc=ex,dc=com 2017-10-30 06:14:03,293 INFO [NiFi Web Server-20] o.a.n.w.a.c.AccessDeniedExceptionMapper cn=Danny Leo,ou=People,ou=Asia,ou=People and Workstations,dc=ex,dc=com does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response.<br></em> I have followed multiple links to resolve this issue. https://community.hortonworks.com/articles/60001/hdf-20-integrating-secured-nifi-with-secured-range.html https://community.hortonworks.com/articles/57980/hdf-20-apache-nifi-integration-with-apache-ambarir.html https://community.hortonworks.com/articles/61729/nifi-identity-conversion.html ... View more

@Geoffrey Shelton Okot I am not using Kerberos. I dont know from where kerberos entries are coming in logs. Nifi is already configured with SSL. Also I have created policies in Ranger. Please check below screenshot. I have added users in all above policies. Template for login-identity-providers.xml <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Identity Strategy">USE_USERNAME</property> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">CN=hadoop_prd_ad_user,OU=Service Accounts,OU=Hadoop,OU=Servers and Services,DC=ex,DC=com</property> <property name="Manager Password">xxx</property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://ldap.ex.com:389</property> <property name="User Search Base">DC=ex,DC=com</property> <property name="User Search Filter">sAMAccountName={0}</property> <property name="Authentication Expiration">12 hours</property> </provider> nifi.security.user.login.identity.provider=ldap-provider In the Ranger UI your parameter Nifi Resource Identifier I have removed * from policy. authorizers.xml <authorizer> <identifier>ranger-provider</identifier> <class>org.apache.nifi.ranger.authorization.RangerNiFiAuthorizer</class> <property name="Ranger Audit Config Path">/usr/hdf/current/nifi/conf/ranger-nifi-audit.xml</property> <property name="Ranger Security Config Path">/usr/hdf/current/nifi/conf/ranger-nifi-security.xml</property> <property name="Ranger Service Type">nifi</property> <property name="Ranger Application Id">nifi</property> <property name="Allow Anonymous">false</property> <property name="Ranger Admin Identity"></property> <property name="Ranger Kerberos Enabled">false</property> Still my issue is not get resolved. How it is automatically gets logged in with anonymous? ... View more

@Geoffrey Shelton Okot have you succeeded in giving your Nifi users UI access through ranger? Ans> yes In Ranger policy, when I give {user} in select user tab I can login with LDAP users. Please check screenshot. Also when I hit the Nifi UI (https://nifihost:9091/nifi) automatically it gets logged in with anonymous user. I do not get login page, but there is option to do login. As per attached screenshot. My question is, I wanted to remove this default anonymous login. Please suggest, where I can do changes in configurations. Also, In Ranger Policy if I remove {user} from select user tab and give specific LDAP users then I cannot login Nifi UI. Even it doesn't login with anonymous. I get below in logs <em>==> /data/log/nifi/nifi-user.log <== 2017-10-30 06:11:54,514 WARN [main] o.a.n.a.util.IdentityMappingUtil Identity Mapping property nifi.security.identity.mapping.pattern.kerb was found, but was empty 2017-10-30 06:11:55,605 WARN [main] o.a.n.a.util.IdentityMappingUtil Identity Mapping property nifi.security.identity.mapping.pattern.kerb was found, but was empty 2017-10-30 06:11:55,652 WARN [main] o.a.n.a.util.IdentityMappingUtil Identity Mapping property nifi.security.identity.mapping.pattern.kerb was found, but was empty 2017-10-30 06:13:06,886 INFO [NiFi Web Server-22] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response. 2017-10-30 06:13:07,135 INFO [NiFi Web Server-94] o.a.n.w.a.c.AccessDeniedExceptionMapper anonymous does not have permission to access the requested resource. Unable to view the user interface. Returning Unauthorized response. 2017-10-30 06:14:03,287 INFO [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (eyJhcGxlLG91PU1VTV9NdW1iYWkgSW5kaWEsb3U9QXNpYSxvdT1QZW9wbGUgYW5kIFdvcmtzdGF0aW9ucyxkYz1tb3Jua) GET https://10.248.13.199:9091/nifi-api/flow/current-user (source ip: 10.90.18.237) 2017-10-30 06:14:03,290 INFO [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for cn=Danny Leo,ou=Asia,ou=People and Workstations,dc=ex,dc=com 2017-10-30 06:14:03,293 INFO [NiFi Web Server-20] o.a.n.w.a.c.AccessDeniedExceptionMapper cn=Danny Leo,ou=People,ou=Asia,ou=People and Workstations,dc=ex,dc=com does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response.<br></em> It seems I have authorizations/permissions issue. Can you please suggest where I'm missing the configurations. Thanks, Suraj ... View more