Hi all, I have recently implemented LDAP for Nifi (works fine) - the problem is it is still using its own CA for certificates and presents the "This connection is not secure" message. Is it possible to create company signed certs to be used by Nifi although NOT for authorisation purposes? I have followed this guide and have been able to generate my own certs but these were used for authorisation and not as standard "web certificate" https://community.hortonworks.com/content/supportkb/151106/nifi-how-to-create-your-own-certs-for-securing-nif.html In short - How can I achieve Nifi LDAP while using a company signed cert? Thanks, ... View more

Hi guys, I have successfully integrated Ambari UI through through KNOX. In addition LDAP is also setup with Ambari and users can login with their LDAP credentials when accessing Ambari through the knox gateway. However, every time a user navigates to the quicklinks of a different UI and attempts to open a new UI (for example Yarn resource manager UI) an authentication popup jumps on screen asking for login credentials. The LDAP credentials the user just used to log into Ambari are no longer valid. Why is this? Here is the topology config: <topology> <gateway> <provider> <role>authentication</role> <name>ShiroProvider</name> <enabled>true</enabled> <param> <name>sessionTimeout</name> <value>30</value> </param> <param> <name>main.ldapRealm</name> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> </param> <param> <name>main.ldapRealm.userDnTemplate</name> <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value> </param> <param> <name>main.ldapRealm.contextFactory.url</name> <value>ldap://{{knox_host_name}}:33389</value> </param> <param> <name>main.ldapRealm.contextFactory.authenticationMechanism</name> <value>simple</value> </param> <param> <name>urls./**</name> <value>authcBasic</value> </param> </provider> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> <provider> <role>authorization</role> <name>AclsAuthz</name> <enabled>true</enabled> </provider> </gateway> <service> <role>NAMENODE</role> <url>hdfs://{{namenode_host}}:{{namenode_rpc_port}}</url> </service> <service> <role>JOBTRACKER</role> <url>rpc://{{rm_host}}:{{jt_rpc_port}}</url> </service> <service> <role>WEBHDFS</role> {{webhdfs_service_urls}} </service> <service> <role>WEBHCAT</role> <url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url> </service> <service> <role>OOZIE</role> <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url> </service> <service> <role>WEBHBASE</role> <url>http://{{hbase_master_host}}:{{hbase_master_port}}</url> </service> <service> <role>HIVE</role> <url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url> </service> <service> <role>RESOURCEMANAGER</role> <url>http://{{rm_host}}:{{rm_port}}/ws</url> </service> <service> <role>DRUID-COORDINATOR-UI</role> {{druid_coordinator_urls}} </service> <service> <role>DRUID-COORDINATOR</role> {{druid_coordinator_urls}} </service> <service> <role>DRUID-OVERLORD-UI</role> {{druid_overlord_urls}} </service> <service> <role>DRUID-OVERLORD</role> {{druid_overlord_urls}} </service> <service> <role>DRUID-ROUTER</role> {{druid_router_urls}} </service> <service> <role>DRUID-BROKER</role> {{druid_broker_urls}} </service> <service> <role>ZEPPELINUI</role> {{zeppelin_ui_urls}} </service> <service> <role>ZEPPELINWS</role> {{zeppelin_ws_urls}} </service> <service> <role>AMBARI</role> <url>http://XXX.XXX.XXX.XXX:8080</url> </service> <service> <role>AMBARIUI</role> <url>http://XXX.XXX.XXX.XXX:8080</url> </service> <service> <role>HBASE</role> <url>http://XXX.XXX.XXX.XXX:16010</url> </service> <service> <role>HBASEUI</role> <url>http://XXX.XXX.XXX.XXX:16010</url> </service> <service> <role>YARN</role> <url>http://XXX.XXX.XXX.XXX:8088</url> </service> <service> <role>YARNUI</role> <url>http://XXX.XXX.XXX.XXX:8088</url> </service> </topology> ... View more

Hi guys, I know this question has been answered before although none of the procedures/guides I have tried so far have been successful. I will list my attempts below: Cluster Info - Ubuntu 14: Ambari-2.5.2.0 HDP-2.6.2.14 HDF-3.0 hdf-ambari-mpack-3.0.1.1-5 Method 1 - Using the Nifi Toolkit to generate the required components: Using the guide: https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy I have then replaced the output files into the usr/hdf/current/nifi/conf folder I have not followed the "Cluster Configuration" section as I have only 1 instance of Nifi. Which is already integrated into the Ambari cluster with the correct zookeeper settings already configured. I have copied the same Authorizer settings as shown. - In this example it shows Nifi auto-configuring to port 9443 on startup..no idea how that is set. Regardless when I attempt port 9091, I get nothing. Nifi is still running as http on port 9090. Method 2 - Configuring the components manually: Following this video tutorial: https://www.youtube.com/watch?v=7DM1ZuWmcAQ I ran the following commands: #: keytool -genkey -alias NifiTest -keyalg RSA -keysize 1024 -dname "CN=NiFi NifiTest, OU=SCC,O=SCC,L=Annapolis,S =Maryland,C=US" -keypass test1234 -keystore NifiTest.jks -storepass test1234 -validity 360 #: keytool -importkeystore -srckeystore NifiTest.jks -destkeystore NifiTest.p12 -srcstoretype JKS -deststoretype PKCS12 -srcstorepass test1234 -deststorepass test1234 -srcalias NifiTest -destalias NifiTest -srckeypass test1234 -destkeypass test1234 -noprompt #: keytool -export -keystore NifiTest.jks -storepass test1234 -alias NifiTest -file NifiTest.cer #: keytool -import -trustcacerts -file NifiTest.cer -alias NifiTest -keystore truststore.jks -storepass test1234 -noprompt and updated the following settings in Ambari - Nifi: Keystore path: /usr/hdf/current/nifi/conf/NifiTest.jks Keystore password: test1234 Keystore type: JKS Truststore path: /usr/hdf/current/nifi/conf/truststore.jks Truststore password: test1234 Truststore type: JKS - I added the custom cert into my Firefox browser Nifi still runs on http port 9090. If I check "Enable SSL?" under "Advanced nifi-ambari-ssl-config. Nifi does not restart and is left with this error: 2018-04-24 10:51:15,160 - Generating NiFi Keystore and Truststore 2018-04-24 10:51:15,160 - File['/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.1-5/bin/tls-toolkit.sh'] {'mode': 0755} Command failed after 1 tries It appears to be trying to re-generate what I have already generated.. Not sure how to proceed, any advice would be greatly appreicated. Thanks, Luka ... View more

HDP 2.6.2 I have managed to add the Hbase Master UI through Knox. The following tabs: Home, Table Details, Local Logs, Log Level, Debug Dump, Metrics Dump and Hbase Configuration all work when accessed except for 'Procedures'. My topology settings (hbaseui.xml): <topology> <gateway> <provider> <role>authentication</role> <name>Anonymous</name> <enabled>true</enabled> </provider> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>false</enabled> </provider> </gateway> <service> <role>HBASE</role> <url>http://XXX.XXX.XXX.XXX:16010</url> </service> <service> <role>HBASEUI</role> <url>http://XXX.XXX.XXX.XXX:16010</url> </service> </topology> When I access the hbase master ui the url is as follows: https://XXX.XXX.XXX.XXX:8443/gateway/hbaseui/hbase/webui/master-status When I click on the 'Procedures' Tab it reverts back to: https://XXX.XXX.XXX.XXX:8443/procedures.jsp Am I missing a setting? ... View more

HDP: 2.6.2.14-5 As the title suggests, Knox is only creating a gateway for ambari. If I access any other service UI it reverts back to the original IP. It doesn't seem to be mapping at all. Taking YARN as an example, this is the config I have in the topology: <service> <role>YARN</role> <url>http://192.168.XXX.XXX:8088</url> </service> <service> <role>YARNUI</role> <url>http://192.168.XXX.XXX:8088</url> </service> ... View more

Hi all, My Livy service (version 0.3) is configured as shown in this tutorial: http://henning.kropponline.de/2016/11/06/connecting-livy-to-a-secured-kerberized-hdp-cluster/ The following command is able to access the livy server and give me a result: curl --negotiate -u : http://fqdn.host.name:8998/sessions My problem is when I try to launch a job from my API, I get an authentication error. Is this a problem due to configurations or because my code isn't asking for any negotiation methods/isn't using principals ? Before running the jobs I have tried doing a 'kinit' for SPNEGO, Zeppelin and Livy principals..no luck.. ... View more

Through tinkering I have been able to partially launch a spark submit job using the following command, however soon after starting it crashes and gives me the exception outlined below: Spark-Submit Command: su spark -c 'export SPARK_MAJOR_VERSION=2; spark-submit \ --verbose \ --master yarn \ --driver-cores 5 \ --num-executors 3 --executor-cores 6 \ --principal spark@test.com \ --keytab /etc/security/keytabs/spark.headless.keytab \ --driver-java-options "-Djava.security.auth.login.config=kafka_client_jaas.conf"\ --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf" \ --files "/tmp/kafka_client_jaas.conf,/tmp/kafka.service.keytab" \ --class au.com.XXX.XXX.spark.test.test test.jar application.properties' EXCEPTION: Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not logn:the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user WARN KerberosLogin: [Principal=kafka/test.com@test.com]: TGT renewal thread has been interrupted and will exit. How can I get Kerberos to KINIT two principals at the same time? I'm assuming that is the problem here? I have tried adding another set of --principal/--keytab to the initial command, although this presented more permission issues within HDFS. ... View more

Hi guys, I have installed Nifi as standalone. I have also created the user 'nifi' for that particular server and assigned it to all files/folders regarding Nifi. In the bootstrap.conf configuration file 'run.as' is set to nothing, although every time I try to access the web UI I'm met with the following error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /nifi-api/flow/current-user was not found on this server.</p> </body></html> I have also gotten the following output from the nifi-user log: 2017-10-10 17:54:00,621 INFO [NiFi Web Server-43] org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET http://XXX.XXX.XXX.XXX:8080/nifi-api/flow (source ip: XXX.XXX.XXX.XXX) 2017-10-10 17:54:00,622 INFO [NiFi Web Server-43] o.a.n.w.a.c.WebApplicationExceptionMapper javax.ws.rs.WebApplicationException. Returning 405 response. Has anyone experienced this before? Thanks in advance ... View more

[Apache Ambari version 2.5.1.0.] Hi guys, My current setup has all service UI's piped through a reverse proxy for security reasons. I'm currently having trouble accessing the Storm UI. Im given the following error in the browser (Chrome): org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag) After some digging it was mentioned that SPNEGO needs to be enabled for Storm UI, so I followed the following steps: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_security/content/_configuring_http_authentication_for_HDFS_YARN_MapReduce2_HBase_Oozie_Falcon_and_Storm.html This just results in a broken YARN (NodeManager Health alerts) and the same browser error.. Thank you in advance, ... View more