Member since
08-08-2017
43
Posts
1
Kudos Received
0
Solutions
05-31-2018
07:35 AM
Hi @Umair Khan I am getting this error when using CA signed certs: o.a.n.c.c.node.NodeClusterCoordinator Failed to determine which node is elected active Cluster Coordinator: ZooKeeper reports the address as F.Q.D.N:9088, but there is no node with this address. Attempted to determine the node's information but failed to retrieve its information due to org.apache.nifi.cluster.protocol.ProtocolException: Failed to request Node Identifer from F.Q.D.N:9088
... View more
05-31-2018
05:15 AM
Hi @Umair Khan, do you have any guides on how to create the truststores/keystores specifically for Nifi? I have tried with my team although Nifi does not accept the new stores..
... View more
05-28-2018
05:06 AM
Hi all, I have recently implemented LDAP for Nifi (works fine) - the problem is it is still using its own CA for certificates and presents the "This connection is not secure" message. Is it possible to create company signed certs to be used by Nifi although NOT for authorisation purposes? I have followed this guide and have been able to generate my own certs but these were used for authorisation and not as standard "web certificate" https://community.hortonworks.com/content/supportkb/151106/nifi-how-to-create-your-own-certs-for-securing-nif.html In short - How can I achieve Nifi LDAP while using a company signed cert? Thanks,
... View more
Labels:
- Labels:
-
Apache NiFi
05-07-2018
07:50 AM
Hi @adash I enabled SSO and it works at redirecting any connections to the Knox login page although it still doesn't accept LDAP logins. Are there any other configs I need to make? This is the advanced knoxsso-topology config <topology>
<gateway>
<provider>
<role>webappsec</role>
<name>WebAppSec</name>
<enabled>true</enabled>
<param><name>xframe.options.enabled</name><value>true</value></param>
</provider>
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<name>sessionTimeout</name>
<value>30</value>
</param>
<param>
<name>redirectToUrl</name>
<value>/gateway/knoxsso/knoxauth/login.html</value>
</param>
<param>
<name>restrictedCookies</name>
<value>rememberme,WWW-Authenticate</value>
</param>
<param>
<name>main.ldapRealm</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
</param>
<param>
<name>main.ldapContextFactory</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
</param>
<param>
<name>main.ldapRealm.contextFactory</name>
<value>$ldapContextFactory</value>
</param>
<param>
<name>main.ldapRealm.userDnTemplate</name>
<value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.url</name>
<value>ldap://X.X.X.X:33389</value>
</param>
<param>
<name>main.ldapRealm.authenticationCachingEnabled</name>
<value>false</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
<value>simple</value>
</param>
<param>
<name>urls./**</name>
<value>authcBasic</value>
</param>
</provider>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
</gateway>
<application>
<name>knoxauth</name>
</application>
<service>
<role>KNOXSSO</role>
<param>
<name>knoxsso.cookie.secure.only</name>
<value>false</value>
</param>
<param>
<name>knoxsso.token.ttl</name>
<value>30000</value>
</param>
<param>
<name>knoxsso.redirect.whitelist.regex</name>
<value>^https?:\/\/(localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$</value>
</param>
</service>
</topology>
<br>
... View more
05-07-2018
06:40 AM
Hi @adash Thanks for that, before I try this solution I noticed it says "Use the following steps to configure Knox SSO for Ranger" despite the title saying "Setting up Knox SSO for Ambari" --- Is Ranger required for this solution? I would prefer to implement SSO without ranger if possible? Thanks, LV
... View more
05-03-2018
07:09 AM
Hi guys, I have successfully integrated Ambari UI through through KNOX. In addition LDAP is also setup with Ambari and users can login with their LDAP credentials when accessing Ambari through the knox gateway. However, every time a user navigates to the quicklinks of a different UI and attempts to open a new UI (for example Yarn resource manager UI) an authentication popup jumps on screen asking for login credentials. The LDAP credentials the user just used to log into Ambari are no longer valid. Why is this? Here is the topology config: <topology>
<gateway>
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<name>sessionTimeout</name>
<value>30</value>
</param>
<param>
<name>main.ldapRealm</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
</param>
<param>
<name>main.ldapRealm.userDnTemplate</name>
<value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.url</name>
<value>ldap://{{knox_host_name}}:33389</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
<value>simple</value>
</param>
<param>
<name>urls./**</name>
<value>authcBasic</value>
</param>
</provider>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
<provider>
<role>authorization</role>
<name>AclsAuthz</name>
<enabled>true</enabled>
</provider>
</gateway>
<service>
<role>NAMENODE</role>
<url>hdfs://{{namenode_host}}:{{namenode_rpc_port}}</url>
</service>
<service>
<role>JOBTRACKER</role>
<url>rpc://{{rm_host}}:{{jt_rpc_port}}</url>
</service>
<service>
<role>WEBHDFS</role>
{{webhdfs_service_urls}}
</service>
<service>
<role>WEBHCAT</role>
<url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url>
</service>
<service>
<role>OOZIE</role>
<url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url>
</service>
<service>
<role>WEBHBASE</role>
<url>http://{{hbase_master_host}}:{{hbase_master_port}}</url>
</service>
<service>
<role>HIVE</role>
<url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url>
</service>
<service>
<role>RESOURCEMANAGER</role>
<url>http://{{rm_host}}:{{rm_port}}/ws</url>
</service>
<service>
<role>DRUID-COORDINATOR-UI</role>
{{druid_coordinator_urls}}
</service>
<service>
<role>DRUID-COORDINATOR</role>
{{druid_coordinator_urls}}
</service>
<service>
<role>DRUID-OVERLORD-UI</role>
{{druid_overlord_urls}}
</service>
<service>
<role>DRUID-OVERLORD</role>
{{druid_overlord_urls}}
</service>
<service>
<role>DRUID-ROUTER</role>
{{druid_router_urls}}
</service>
<service>
<role>DRUID-BROKER</role>
{{druid_broker_urls}}
</service>
<service>
<role>ZEPPELINUI</role>
{{zeppelin_ui_urls}}
</service>
<service>
<role>ZEPPELINWS</role>
{{zeppelin_ws_urls}}
</service>
<service>
<role>AMBARI</role>
<url>http://XXX.XXX.XXX.XXX:8080</url>
</service>
<service>
<role>AMBARIUI</role>
<url>http://XXX.XXX.XXX.XXX:8080</url>
</service>
<service>
<role>HBASE</role>
<url>http://XXX.XXX.XXX.XXX:16010</url>
</service>
<service>
<role>HBASEUI</role>
<url>http://XXX.XXX.XXX.XXX:16010</url>
</service>
<service>
<role>YARN</role>
<url>http://XXX.XXX.XXX.XXX:8088</url>
</service>
<service>
<role>YARNUI</role>
<url>http://XXX.XXX.XXX.XXX:8088</url>
</service>
</topology>
... View more
Labels:
- Labels:
-
Apache Knox
-
Apache YARN
03-29-2018
01:03 AM
Thank you @Geoffrey Shelton Okot, What would then be the required changes for the settings below? Will Knox work with AD auth? <provider>
<role>authentication</role>
<name>Anonymous</name>
<enabled>true</enabled>
</provider>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>false</enabled>
</provider><br>
... View more
03-28-2018
03:42 AM
@Geoffrey Shelton Okot That worked perfectly! - If I wish to use LDAP authentication - Do I first need to setup Ambari with LDAP and then specify the following to use the LDAP auth? <gateway>
<provider>
<role>authentication</role>
<name>Anonymous</name>
<enabled>true</enabled>
... View more
03-28-2018
03:29 AM
Hi @Geoffrey Shelton Okot I started up the Knox LDAP Demo and it immediately worked using: user: guest password: guest-password I will try your method without the LDAP demo. Thanks
... View more
03-28-2018
02:18 AM
Hi guys, I have attempted to add the YARN service into Knox Gateway. These are the settings in my advanced topology xml file: <service>
<role>YARN</role>
<url>http://<HOSTNAME>:8088</url>
</service>
<service>
<role>YARNUI</role>
<url>http://<HOSTNAME>:8088</url>
</service>
- I'm not entirely sure those ports are correct. The configs show 8050 for yarn.resourcemanager.address 8141 for yarn.resourcemanager.admin.address 8088 for yarn.resourcemanager.webapp.address 8090 for yarn.resourcemanager.webapp.https.address I have also updated the following quicklinks.json files on the Ambari Server Host: /var/lib/ambari-server/resources/stacks/HDP/2.0.6/services/YARN/quicklinks/quicklinks.json
/var/lib/ambari-server/resources/stacks/HDP/2.3/services/YARN/quicklinks/quicklinks.json The config change looks like this: {
"name": "resourcemanager_ui",
"label": "ResourceManager UI",
"requires_user_name": "false",
"component_name": "RESOURCEMANAGER",
"url": "https://FQDN(KNOX HOST):8443/gateway/default/yarnui",
"port":{
"http_property": "yarn.resourcemanager.webapp.address",
"http_default_port": "8088",
"https_property": "yarn.resourcemanager.webapp.https.address",
"https_default_port": "8090",
"regex": "\\w*:(\\d+)",
"site": "yarn-site"
}
The problem is when I use the URL: https://<FQDN KNOX HOST>:8443/gateway/default/yarnui/ I get a page not found error, although when I use the URL: https://<FQDN KNOX HOST>:8443/gateway/default/yarn/ I get a popup asking me for a username and password - I have not set these anywhere. I have also tried admin/admin and admin/admin-password with no luck. Am I missing something?
... View more
Labels:
- Labels:
-
Apache Knox
-
Apache YARN
03-06-2018
03:13 AM
Hi @Jay Kumar SenSharma that worked! Although it throws a browser popup asking me for a username and password - admin/admin doesn't work nor does the Knox master secret password work and I don't recall setting any other authentication credentials. Are additional authentication configs required?
... View more
03-02-2018
01:20 AM
Hi @Jay Kumar SenSharma Do I need to change the quicklink in all of the above listed locations?
... View more
03-01-2018
08:37 AM
Thanks for that @Jay Kumar SenSharma I made the following changes: {
"name": "resourcemanager_ui",
"label": "ResourceManager UI",
"requires_user_name": "false",
"component_name": "RESOURCEMANAGER",
"url": "https://FQDN(KNOX HOST):8443/gateway/default/yarnui",
"port":{
"http_property": "yarn.resourcemanager.webapp.address",
"http_default_port": "8088",
"https_property": "yarn.resourcemanager.webapp.https.address",
"https_default_port": "8090",
"regex": "\\w*:(\\d+)",
"site": "yarn-site"
}
},<br> Still no luck. - I restarted both ambari server and the ambari agent for the YARN host.
... View more
03-01-2018
07:51 AM
Hi @Jay Kumar SenSharma Using the FQDN didn't work either. With quicklinks, the only files available are in these directories /var/lib/ambari-agent/cache/common-services/ /var/lib/ambari-agent/cache/stacks/HDP/2.3/services/ /var/lib/ambari-agent/cache/stacks/BigInsights/4.2.5/services/ Which doesn't look right: https://cwiki.apache.org/confluence/display/AMBARI/Quick+Links "By default, the JSON file is called quicklinks.json and is located in the quicklinks directory under the service root directory. For example, for Oozie, the file is OOZIE/quicklinks/quicklinks.json. You can also name the file differently as well as put it in a custom directory under the service root directory."
... View more
03-01-2018
06:31 AM
HDP: 2.6.2.14-5 As the title suggests, Knox is only creating a gateway for ambari. If I access any other service UI it reverts back to the original IP. It doesn't seem to be mapping at all. Taking YARN as an example, this is the config I have in the topology: <service>
<role>YARN</role>
<url>http://192.168.XXX.XXX:8088</url>
</service>
<service>
<role>YARNUI</role>
<url>http://192.168.XXX.XXX:8088</url>
</service>
... View more
Labels:
- Labels:
-
Apache Knox
03-01-2018
06:10 AM
Hi @Jay Kumar SenSharma That worked, although when I go and access other services and try view their UI's it just diverts back to their original IP address. Lets take YARN as an example, this is the configuration I have currently entered into the topology: <service>
<role>YARN</role>
<url>http://192.168.XXX.XXX:8088</url>
</service>
<service>
<role>YARNUI</role>
<url>http://192.168.XXX.XXX:8088</url>
</service>
... View more
02-27-2018
03:20 AM
@Jay Kumar SenSharma I have set the Ambari Server URL as follows: https://$AMBARI_HOST:8443 I am unable to import any certificates into the Knox keystore. Using the following command: keytool -importcert -file test.csr -keystore gateway.jks -alias "gateway-identity" along with the Knox Master Secret password although I get the error: keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect. Note I have created the alias with this command: knoxcli.sh create-alias gateway-identity-passphrase --value {value} I have even removed and re-installed Knox and reset the Master Secret password, again same error.
... View more
02-27-2018
02:26 AM
@Jay Kumar SenSharma I am unable to access Ambari UI at all when trying through Knox. I'm met with a HTTP 500 error
... View more
02-27-2018
02:17 AM
Hi @Jay Kumar SenSharma I have configured the truststore and enabled SSL for Ambari. It is currently listening on port 8443 and I am able to access Ambari UI normally (now via HTTPS). I am still receiving the same error when trying to access it via Knox. The log is from /var/log/knox/ (Not located on the ambari host server) Full Log: gateway.txt
... View more
02-27-2018
01:26 AM
@Jay Kumar SenSharma Snippet: Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491)
... 76 more
Its strange because originally it was working. I will give that guide a go now. Thanks! EDIT: This was from the Knox Gateway log /var/log/knox/gateway.log
... View more
02-27-2018
01:19 AM
@Jay Kumar SenSharma - I did not have AMBARI added, only AMBARIUI. I have added that in now although I am now getting this error: Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target @Deepak Sharma I was able to view the ambari homepage, there were just little things missing as I mentioned earlier.
... View more
02-26-2018
08:27 AM
Hi @Deepak Sharma did you have any advice regarding this issue?
... View more
01-09-2018
06:47 AM
HDP: 2.6.2.14-5 I used the default admin account - verified that it is still an admin with the correct priveleges
... View more
01-08-2018
08:08 AM
I have followed this guide to enable Knox for the Ambari cluster: https://community.hortonworks.com/articles/78361/configure-knox-to-access-ambari-ui.html While I am able to log into the Ambari UI via Knox. I cannot view the hosts nor am I able to access the preconfigured 'Views': - Hive View - Files View - Smartsense View - Tez View etc.. I have also noticed that alerts and warnings do not show either. Why am I a limited user despite logging in with an admin account? Thanks in advance
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Knox
12-05-2017
07:40 AM
Hi all, My Livy service (version 0.3) is configured as shown in this tutorial: http://henning.kropponline.de/2016/11/06/connecting-livy-to-a-secured-kerberized-hdp-cluster/ The following command is able to access the livy server and give me a result: curl --negotiate -u : http://fqdn.host.name:8998/sessions My problem is when I try to launch a job from my API, I get an authentication error. Is this a problem due to configurations or because my code isn't asking for any negotiation methods/isn't using principals ? Before running the jobs I have tried doing a 'kinit' for SPNEGO, Zeppelin and Livy principals..no luck..
... View more
Labels:
- Labels:
-
Apache Spark
11-10-2017
01:30 AM
Through tinkering I have been able to partially launch a spark submit job using the following command, however soon after starting it crashes and gives me the exception outlined below: Spark-Submit Command:
su spark -c 'export SPARK_MAJOR_VERSION=2; spark-submit \
--verbose \
--master yarn \
--driver-cores 5 \
--num-executors 3 --executor-cores 6 \
--principal spark@test.com \
--keytab /etc/security/keytabs/spark.headless.keytab \
--driver-java-options "-Djava.security.auth.login.config=kafka_client_jaas.conf"\
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf" \
--files "/tmp/kafka_client_jaas.conf,/tmp/kafka.service.keytab" \
--class au.com.XXX.XXX.spark.test.test test.jar application.properties' EXCEPTION:
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not logn:the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
WARN KerberosLogin: [Principal=kafka/test.com@test.com]: TGT renewal thread has been interrupted and will exit.
How can I get Kerberos to KINIT two principals at the same time? I'm assuming that is the problem here? I have tried adding another set of --principal/--keytab to the initial command, although this presented more permission issues within HDFS.
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Spark
10-11-2017
02:13 AM
Hi guys, I have installed Nifi as standalone. I have also created the user 'nifi' for that particular server and assigned it to all files/folders regarding Nifi. In the bootstrap.conf configuration file 'run.as' is set to nothing, although every time I try to access the web UI I'm met with the following error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /nifi-api/flow/current-user was not found on this server.</p> </body></html> I have also gotten the following output from the nifi-user log: 2017-10-10 17:54:00,621 INFO [NiFi Web Server-43] org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET http://XXX.XXX.XXX.XXX:8080/nifi-api/flow (source ip: XXX.XXX.XXX.XXX) 2017-10-10 17:54:00,622 INFO [NiFi Web Server-43] o.a.n.w.a.c.WebApplicationExceptionMapper javax.ws.rs.WebApplicationException. Returning 405 response. Has anyone experienced this before? Thanks in advance
... View more
Labels:
- Labels:
-
Apache NiFi
10-10-2017
03:29 AM
[Apache Ambari version 2.5.1.0.] Hi guys, My current setup has all service UI's piped through a reverse proxy for security reasons. I'm currently having trouble accessing the Storm UI. Im given the following error in the browser (Chrome): org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag) After some digging it was mentioned that SPNEGO needs to be enabled for Storm UI, so I followed the following steps: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_security/content/_configuring_http_authentication_for_HDFS_YARN_MapReduce2_HBase_Oozie_Falcon_and_Storm.html This just results in a broken YARN (NodeManager Health alerts) and the same browser error.. Thank you in advance,
... View more
Labels:
- Labels:
-
Apache Storm
09-07-2017
09:49 AM
Hi @Sindhu, The cipher I want to disable (RSA_WITH_3DES_EDE_CBC_SHA) falls under the TLSv1.2 cipher suite. Is RSA_WITH_3DES_EDE_CBC_SHA and SSL_RSA_WITH_3DES_EDE_CBC_SHA the same cipher?
... View more
09-04-2017
10:43 AM
Hi guys, - Ambari 2.5.X, RHEL7 I'm attempting to disable the cipher: RSA_WITH_3DES_EDE_CBC_SHA I have added it to the ambari.properties file as both: RSA_WITH_3DES_EDE_CBC_SHA and TLS_RSA_WITH_3DES_EDE_CBC_SHA (They are separated by the '|' symbol) Although every time I run TestSSLServer2 on Ambari port..the ciphers show up. I have also added these ciphers to the servers java.security file with no luck. Am I missing something?
... View more
- Tags:
- Security
Labels:
- Labels:
-
Apache Ambari
-
Security