Member since
09-24-2014
29
Posts
2
Kudos Received
3
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 13330 | 04-25-2015 09:56 AM | |
| 8704 | 03-07-2015 03:03 PM | |
| 4030 | 03-01-2015 10:25 AM |
04-15-2015
08:51 PM
I was looking some more to confirm that the issue is between Cloudera Navigator host and Cloudera Manager host: 2015-04-15 23:20:50,677 WARN 236787520@scm-web-23643:org.mortbay.log: SSL renegotiate denied: java.nio.channels.SocketChannel[connected local=/{CM_SERVER_HOST}:7183 remote=/{NAVIGATOR_HOST}:50359] 2015-04-15 23:20:57,174 WARN 236787520@scm-web-23643:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
... View more
04-15-2015
08:38 PM
Troubleshooting SSL/TLS Connectivity. Verified connectivity. **{HOSTNAME} refers to the hostname listed in the logs, where Cloudera Manager Server lives # openssl s_client -connect {HOSTNAME}:7183 CONNECTED(00000003) depth=1 O = PLATFORM.{OUR_DOMAIN], CN = Certificate Authority verify error:num=19:self signed certificate in certificate chain verify return:0 --- Certificate chain 0 s:/O=PLATFORM.{OUR_DOMAIN]/CN={HOSTNAME} i:/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority 1 s:/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority i:/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority --- Server certificate -----BEGIN CERTIFICATE----- ...here goes our certificate -----END CERTIFICATE----- subject=/O=PLATFORM.{OUR_DOMAIN]/CN={HOSTNAME} issuer=/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority --- No client certificate CA names sent Server Temp Key: ECDH, ___, 521 bits --- SSL handshake has read 2508 bytes and written 511 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.2 Cipher : ___ Session-ID: ___ Session-ID-ctx: Master-Key: ___ Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1429155044 Timeout : 300 (sec) Verify return code: 19 (self signed certificate in certificate chain) ---
... View more
04-15-2015
08:26 PM
After having succesfully enabled TLS encryption between Server and Agents, I am unable to load Cloudera Navigator UI. The log is pointing at issues with SSL handshake. I understand I need to configure SSL for Cloudera Navigator in addition to this, so I followed guidelines from Cloudera documentation: Open the Cloudera Manager Admin Console and navigate to the Cloudera Management Service. Click Configuration. Go to the Navigator Metadata Server Default Group > Advanced category, and add the following strings to the Navigator Metadata Server Advanced Configuration Snippet (Safety Valve) for cloudera-navigator.properties property. nav.http.enable_ssl=true nav.ssl.keyStore=<path to jks keystore with signed server certificate installed> nav.ssl.keyStorePassword=<password> Click Save Changes. Restart the Navigator Metadata server. After I added cloudera-navigator.properties to Safety Valve and restarted, Cloudera Management Services became unhealthy and I had to revert my change. I would like to clarify what values exactly go into nav.ssl.keyStore and nav.ssl.keyStorePassword. I have set nav.ssl.keyStore to same value as ssl.client.truststore.location, since this is where my keystore file lives. 2015-04-15 17:54:02,572 WARN com.cloudera.enterprise.EnterpriseService: Exception in scheduled runnable. javax.ws.rs.client.ClientException: org.apache.cxf.interceptor.Fault: Could not send Message. at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:548) at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:534) at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:545) at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:206) at com.sun.proxy.$Proxy35.readRoles(Unknown Source) at com.cloudera.nav.cm.CmApiClient.getMgmtRoleByType(CmApiClient.java:224) at com.cloudera.navigator.ActivityPollingService.getAmonNozzle(ActivityPollingService.java:189) at com.cloudera.navigator.ActivityPollingService.run(ActivityPollingService.java:108) at com.cloudera.enterprise.PeriodicEnterpriseService$UnexceptionablePeriodicRunnable.run(PeriodicEnterpriseService.java:67) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.cxf.interceptor.Fault: Could not send Message. at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:607) at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:543) ... 7 more Caused by: javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://{HOSTNAME}:7183/api/v4/cm/service/roles: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.reflect.GeneratedConstructorAccessor51.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1338) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1322) at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:622) at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) ... 10 more Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1341) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868) at sun.security.ssl.Handshaker.process_record(Handshaker.java:804) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1300) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:260) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1517) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1490) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1309) ... 13 more Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) at sun.security.validator.Validator.validate(Validator.java:260) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1323) ... 29 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196) at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268) at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380) ... 35 more
... View more
Labels:
03-07-2015
03:03 PM
Got everything to work! Thanks all for useful tips. Uninstalled and did another clean install - this time ensured that Sentry server is installed on same node where Hue, Hive and Impala services are also installed. My FreeIPA user does not have sufficient privileges to create roles but I will fix that 🙂
... View more
03-07-2015
02:26 PM
Got Sentry service successfully added for Hue, Hive and Impala services. I am however now seeing a connection error when I try to load Sentry Tables: timed out (code THRIFTSOCKET): None The Hue error.log: kerberos_ ERROR handle_other(): Mutual authentication unavailable on 200 response Sentry's log on node it is installed on: ERROR sentry.org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message. java.lang.RuntimeException: sentry.org.apache.thrift.transport.TTransportException at sentry.org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at sentry.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: sentry.org.apache.thrift.transport.TTransportException at sentry.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at sentry.org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at sentry.org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182) at sentry.org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at sentry.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) at sentry.org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:1) at sentry.org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ... 4 more In Cloudera Manager Hue safety valve I have: [libsentry] # Hostname or IP of server. hostname=cdh-foyer.platform.infochimps # Port the sentry service is running on. port=8038 # Sentry configuration directory, where sentry-site.xml is located. sentry_conf_dir=/etc/sentry/conf sentry-site.xml has default settings for "sentry.service.security.mode". It seems I should need to specify Kerberos here instead of "none", not sure if that is a requirement here is this config. <?xml version="1.0" encoding="UTF-8"?> <configuration> <property> <name>sentry.service.security.mode</name> <value>none</value> </property> <property> <name>sentry.service.admin.group</name> <value>admin1</value> </property> <property> <name>sentry.service.allow.connect</name> <value>impala,hive,solr</value> </property> <property> <name>sentry.store.jdbc.url</name> <value>jdbc:derby:;databaseName=sentry_store_db;create=true</value> </property> <property> <name>sentry.store.jdbc.driver</name> <value>org.apache.derby.jdbc.EmbeddedDriver</value> </property> </configuration>
... View more
03-06-2015
03:30 PM
Followed all steps outlined here http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/sg_sentry_service_config.html#concept_z5b_42s_p4_unique_1 (confuguring Sentry with Cloudera Manager) and Sentry is still not running.
... View more
03-06-2015
08:32 AM
I am going over all steps outlined here http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/sg_sentry_service_config.html 1. Permissions $ sudo -u hdfs hdfs dfs -chmod -R 771 /user/hive/warehouse
$ sudo -u hdfs hdfs dfs -chown -R hive:hive /user/hive/warehous 2. Disabled impersonation for HiveServer2 in the Cloudera Manager Admin Console 3. Set the Minimum User ID for Job Submission property to zero 4. Ensured the Allowed System Users property includes the hive user 5. Unchecked the Enable Sentry Authorization using Policy Files configuration property for both Hive and Impala under the Service-Wide > Policy File Based Sentry In the Service-Wide category for Hue/Hive/Impala, I will need to set the Sentry Service property to Sentry, but the option is not listed still. Going over all pre-req's again.
... View more
03-06-2015
06:32 AM
Added hostname to safety valve and restarted Hue service. Sentry service is definitely not running: # ps auxfww | grep sentry and # netstat -anp | grep 8038 ...return nothing, which explains my error in Hue. However, I was able to test HiveServer2 with beeline shell using above recommendation for the string. # beeline Beeline version 0.13.1-cdh5.3.0 by Apache Hive beeline> !connect jdbc:hive2://MY_FQDN_HOSTNAME:10000/default;principal=hive/MY_FQDN_HOSTNAME@MY_DOMAIN scan complete in 2ms Connecting to jdbc:hive2://MY_FQDN_HOSTNAME:10000/default;principal=hive/MY_FQDN_HOSTNAME@MY_DOMAIN Enter username for jdbc:hive2://MY_FQDN_HOSTNAME:10000/default;principal=hive/MY_FQDN_HOSTNAME@MY_DOMAIN: Enter password for jdbc:hive2://cjdbc:hive2://MY_FQDN_HOSTNAME:10000/default;principal=hive/MY_FQDN_HOSTNAME@MY_DOMAIN: Connected to: Apache Hive (version 0.13.1-cdh5.3.0) Driver: Hive JDBC (version 0.13.1-cdh5.3.0) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://MY_FQDN_HOSTNAME:10> show databases; +----------------+--+ | database_name | +----------------+--+ | default | | test | +----------------+--+ 2 rows selected (0.178 seconds)
... View more
03-05-2015
07:56 PM
# beeline Beeline version 0.13.1-cdh5.3.0 by Apache Hive beeline> !connect jdbc:hive2://localhost:10000 org.apache.hive.jdbc.HiveDriver scan complete in 2ms Connecting to jdbc:hive2://localhost:10000 Enter password for jdbc:hive2://localhost:10000: Error: Could not open connection to jdbc:hive2://localhost:10000: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0) 0: jdbc:hive2://localhost:10000 (closed)> With the cluster being Kerberized, looks to me like some config still needs to be added here to enable kerberos vs plain as shown above in an error message.
... View more
03-05-2015
07:15 PM
I am going to troubleshoot HiveServer2 to make sure everything is running as expected there, as the error could point at some misconfigurations. I do think though that Cloudera Manager would have picked up if an issue with HiveServer2 was detected...
... View more
- « Previous
-
- 1
- 2
- Next »