Re: Can't connect to Hive Metastore in cluster wit...

nataliaking_csc · ‎09-26-2015

I am having issues with configuring external authentication for Cloudera Navigator with FreeIPA (OpenLDAP compatible). Following instructions "Configuring Cloudera Navigator Authentication Using an OpenLDAP-compatible Server" found here http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cn_sg_external_auth.html#cmug_topic_13_9_2_unique_4 In the External Authentication Type, select LDAP - DONE In the LDAP URL property, provide the URL of the LDAP server and (optionally) the base Distinguished Name (DN) (the search base) as part of the URL — for example ldap://ldap-server.corp.com/dc=corp,dc=com. - DONE (experimented with both ldaps://MY_FREEIPA_SERVER and ldaps://MY_FREEIPA_SERVER /dc=platform,dc=*** In the Bind Distinguished Name property, enter the distinguished name of the user to bind as. This is used to connect to the LDAP server for searching groups and to get other user information. - DONE (I used my own IPA user ID to get this to work first, I am an admin user) In the LDAP Bind Password property, enter the password for the bind user entered above. Other configurations: LDAP Distinguished Name Pattern: uid={0} LDAP User Search Base: cn=accounts,dc=platform,dc=*** I was able to troubleshoot using the ldapsearch command (requires the openldap-clients package) on the host where LDAP authentication or authorization issues are being seen: $ ldapsearch -D 'uid=MY_USER,cn=users,cn=accounts,dc=platform,dc=***' -W -b 'cn=users,cn=accounts,dc=platform,dc=***' localhost uid The reason why I am using LDAPS (I did try just an "ldap" at first) is because URI is configured by our FreeIPA scripts at launch (http://www.freeipa.org/page/HowTo/LDAP): $ cat /etc/openldap/ldap.conf #File modified by ipa-client-install URI ldaps:// MY_FREEIPA_SERVER BASE dc=platform,dc=*** TLS_CACERT /etc/ipa/ca.crt $ ls /etc/openldap/ldap.conf /etc/openldap/ldap.conf We are using Release 3 (http://www.freeipa.org/page/Releases/3.0.0) and this is working for external auth for Cloudera Manager, HUE and other non-Hadoop components integrated within the platform. Still unable to login though

nataliaking_csc · ‎07-30-2015

Find current process number for Hue service. As a root user navigate to directory /var/run/cloudera-scm-agent/process/ and find subdirectory XX-hue-HUE_SERVER with the highest value, i.e. “61-hue-HUE_SERVER” Inside XX-hue-HUE_SERVER directory, confirm that TTL value has been added to hue_safety_valve.ini file. # grep ttl /var/run/cloudera-scm-agent/process/{PROCESS_NUMBER}-hue-HUE_SERVER/hue_safety_valve.ini ttl=900

nataliaking_csc · ‎07-14-2015

Great, thank you for clarification. I didn't realize it logs me out even if I remain active. Good to know!

nataliaking_csc · ‎07-14-2015

This is for period of inactivity. If you are actively using HUE, you won't be logged off. In various scenarios - like compliant and/or secure clusters - it would be required to set up automated timeout for idle users.

nataliaking_csc · ‎04-25-2015

I was looking further into this and it appears the problem comes up when I first try loading data to newly created table, and throws: Fetching results ran into the following error(s): java.io.IOException: java.io.IOException: HTTP status [500], message [Internal Server Error]

nataliaking_csc · ‎04-25-2015

Connected with beeline 1: jdbc:hive2://{hostname_of_hive_server}:10> select * from students; +-----------------+---------------+---------------+--+ | students.sname | students.age | students.gpa | +-----------------+---------------+---------------+--+ +-----------------+---------------+---------------+--+ No rows selected (0.802 seconds) 1: jdbc:hive2://{hostname_of_hive_server}:10> select * from sales limit 5; Error: java.io.IOException: java.io.IOException: HTTP status [500], message [Internal Server Error] (state=,code=0)

nataliaking_csc · ‎04-25-2015

Using CDH 5.3, with Kerberos and TLS enabled when we got to testing loading data, noticed that connection to Hive Metastore fails. Cloudera Manager is not indicating any issues with principals and their keytabs. What may I be missing here? 2015-04-25 11:02:26,197 ERROR org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:724) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:721) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:356) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:721) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199) at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:262) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ... 10 more 2015-04-25 11:02:27,200 ERROR org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:724) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:721) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:356) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:721) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199) at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:262) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ... 10 more

nataliaking_csc · ‎04-17-2015

Thank you! We got this to work finally. Now just need to wire to LDAP. It looks like the keystore passed by following the instructions on Cloudera's site wasn't used for some reason. Following instructions here: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_create_key_trust.html#concept_u35_w2m_l4_unique_1 The upshot is that the keystore at /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/jssecacerts (or ${JAVA_HOME}/lib/security/jssecacerts, where $JAVA_HOME is the home of the version of Java used by Cloudera Navigator, we used "ps" to find out where) should contain the root certificate. We then restarted both Cloudera Navigator services, and able to navigate to https:// Integration with FreeIPA was sort of confusing initially. Thanks for your help in understanding the mechnisms of this functionality.

nataliaking_csc · ‎04-16-2015

The the CA certificate that issued the certificate, imported into the truststores I've discussed already, establishes inherent trust within java services, for SSL certificates created by that CA. We have already created a truststore on Namenode (node where Cloudera Manager is installed), when TLS was set up for all agents, that's what was used - /etc/cloudera-scm-server/keystore - and this file was copied to all nodes in Cloudera Hadoop cluster, including Navigator.

nataliaking_csc · ‎04-16-2015

"Trust" is established differently between the two implementations, Navigator, being Java based, will derive trust through the default JDK mechanisms I pointed out We are looking into whether we need to use keytool utility to generate those or use our FreeIPA server to generate certs for Navigator...

nataliaking_csc · ‎04-15-2015

So far it appears that only Navigator is unhappy with the keystore. So I believe TLS/SSL was set up correctly, otherwise. We are using FreeIPA as certificate authority and below is a quick overview of steps taken to set it up, since there is a somewhat deviation from standard protocol. On the Namenode (same host the Cloudera Manager lives on), I generated a certificate and key to be used by Cloudera Manager # kinit -kt /etc/krb5.keytab # ipa-getcert request -f cmhost.pem -k cmhost.key -r # chmod 600 cmhost* Then I copied the newly created cm-keys directory to each host. $ for x in {LIST_OF_CDH_HOSTS}; do scp -r cm-keys $x:; done $ for x in {LIST_OF_CDH_HOSTS}; do ssh -tty $x sudo bash -c "'mkdir -p /opt/cloudera/security/x509; mv cm-keys/* /opt/cloudera/security/x509; chown cloudera /opt/cloudera/security/x509/*'"; done Next, I set up Puppet to configure Cloudera to use TLS.

nataliaking_csc · ‎04-15-2015

I was looking some more to confirm that the issue is between Cloudera Navigator host and Cloudera Manager host: 2015-04-15 23:20:50,677 WARN 236787520@scm-web-23643:org.mortbay.log: SSL renegotiate denied: java.nio.channels.SocketChannel[connected local=/{CM_SERVER_HOST}:7183 remote=/{NAVIGATOR_HOST}:50359] 2015-04-15 23:20:57,174 WARN 236787520@scm-web-23643:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

nataliaking_csc · ‎04-15-2015

Troubleshooting SSL/TLS Connectivity. Verified connectivity. **{HOSTNAME} refers to the hostname listed in the logs, where Cloudera Manager Server lives # openssl s_client -connect {HOSTNAME}:7183 CONNECTED(00000003) depth=1 O = PLATFORM.{OUR_DOMAIN], CN = Certificate Authority verify error:num=19:self signed certificate in certificate chain verify return:0 --- Certificate chain 0 s:/O=PLATFORM.{OUR_DOMAIN]/CN={HOSTNAME} i:/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority 1 s:/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority i:/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority --- Server certificate -----BEGIN CERTIFICATE----- ...here goes our certificate -----END CERTIFICATE----- subject=/O=PLATFORM.{OUR_DOMAIN]/CN={HOSTNAME} issuer=/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority --- No client certificate CA names sent Server Temp Key: ECDH, ___, 521 bits --- SSL handshake has read 2508 bytes and written 511 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.2 Cipher : ___ Session-ID: ___ Session-ID-ctx: Master-Key: ___ Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1429155044 Timeout : 300 (sec) Verify return code: 19 (self signed certificate in certificate chain) ---

nataliaking_csc · ‎04-15-2015

After having succesfully enabled TLS encryption between Server and Agents, I am unable to load Cloudera Navigator UI. The log is pointing at issues with SSL handshake. I understand I need to configure SSL for Cloudera Navigator in addition to this, so I followed guidelines from Cloudera documentation: Open the Cloudera Manager Admin Console and navigate to the Cloudera Management Service. Click Configuration. Go to the Navigator Metadata Server Default Group > Advanced category, and add the following strings to the Navigator Metadata Server Advanced Configuration Snippet (Safety Valve) for cloudera-navigator.properties property. nav.http.enable_ssl=true nav.ssl.keyStore=<path to jks keystore with signed server certificate installed> nav.ssl.keyStorePassword=<password> Click Save Changes. Restart the Navigator Metadata server. After I added cloudera-navigator.properties to Safety Valve and restarted, Cloudera Management Services became unhealthy and I had to revert my change. I would like to clarify what values exactly go into nav.ssl.keyStore and nav.ssl.keyStorePassword. I have set nav.ssl.keyStore to same value as ssl.client.truststore.location, since this is where my keystore file lives. 2015-04-15 17:54:02,572 WARN com.cloudera.enterprise.EnterpriseService: Exception in scheduled runnable. javax.ws.rs.client.ClientException: org.apache.cxf.interceptor.Fault: Could not send Message. at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:548) at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:534) at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:545) at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:206) at com.sun.proxy.$Proxy35.readRoles(Unknown Source) at com.cloudera.nav.cm.CmApiClient.getMgmtRoleByType(CmApiClient.java:224) at com.cloudera.navigator.ActivityPollingService.getAmonNozzle(ActivityPollingService.java:189) at com.cloudera.navigator.ActivityPollingService.run(ActivityPollingService.java:108) at com.cloudera.enterprise.PeriodicEnterpriseService$UnexceptionablePeriodicRunnable.run(PeriodicEnterpriseService.java:67) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.cxf.interceptor.Fault: Could not send Message. at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:607) at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:543) ... 7 more Caused by: javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://{HOSTNAME}:7183/api/v4/cm/service/roles: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.reflect.GeneratedConstructorAccessor51.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1338) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1322) at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:622) at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) ... 10 more Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1341) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868) at sun.security.ssl.Handshaker.process_record(Handshaker.java:804) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1300) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:260) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1517) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1490) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1309) ... 13 more Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) at sun.security.validator.Validator.validate(Validator.java:260) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1323) ... 29 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196) at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268) at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380) ... 35 more

nataliaking_csc · ‎04-06-2015

Some of the values you cannot find in Configuration for a particular service in Cloudera Manager may need to be changed via Safety Valve feature. For example, to add some custom value: Click a service. Select Configuration. Expand a category group and click an Advanced subcategory. In the Property column, choose a property that contains the safety valve value. In Cloudera Manager 5, they are called Advanced Configuration Snippet . Specify the properties. Save Changes. Restart the service or redeploy client configurations as prompted. I used this to add some custom configs for Hue, Sentry, etc. This will also ensure that with each config deploy, these values won't be wiped out. For example, if you add values to Hue config via safety valve option, you will be able to verify by locating appropariate .ini file. CM generates all the hue.ini for you. Value overridden in Cloudera Manager will be added to hue_safety_valve.ini, if you want to verify on the nodes manually, you can locate hue_safety_valve.ini file and ensure values have added successfully. To find main config file for Hue that is being used by current process run: $ ls -alrt /var/run/cloudera-scm-agent/process | grep HUE | tail -1 | awk ‘{print $9}’`/hue.ini

nataliaking_csc · ‎03-24-2015

This is probably not a very common scenario but can certainly happen to many who use Kerberos along with some OpenLDAP solution. We are using FreeIPA for users management, and have recently successfully configured Kerberos following Cloudera's documentation for configuring authentication using the Coudera Manager wizard (**because our cluster uses FreeIPA and FreeIPA does not allow direct access to Kerberos commands used to create principals, some deviations were necessary). Because the server running FreeIPA server had to be rebuilt, we ended up with a fresh install and lost all of the FreeIPA config files stored there. We have made sure all Cloudera nodes have rejoined the cluster. We are, however, unable to start all services via Cloudera Manager, and the following warning shows up in the logs (from Namenode): 10:35:05.198 PM WARN StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread Failed to evict stale entities java.lang.RuntimeException: javax.persistence.RollbackException: Error while committing the transaction at com.cloudera.cmf.persist.DatabaseExecutor.execReadWriteTaskNE(DatabaseExecutor.java:77) at com.cloudera.cmf.command.components.CommandManager.batchDeleteCommands(CommandManager.java:48) at com.cloudera.server.cmf.StaleEntityEvictionThread.reapDeletedCommands(StaleEntityEvictionThread.java:237) at com.cloudera.server.cmf.StaleEntityEvictionThread.innerLoop(StaleEntityEvictionThread.java:375) at com.cloudera.server.cmf.StaleEntityEvictionThread.run(StaleEntityEvictionThread.java:123) Caused by: javax.persistence.RollbackException: Error while committing the transaction at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:92) at com.cloudera.enterprise.AbstractWrappedEntityManager.commit(AbstractWrappedEntityManager.java:110) at com.cloudera.cmf.persist.CmfEntityManager.commit(CmfEntityManager.java:366) at com.cloudera.cmf.persist.ReadWriteDatabaseTaskCallable.call(ReadWriteDatabaseTaskCallable.java:37) at com.cloudera.cmf.persist.DatabaseExecutor.execTask(DatabaseExecutor.java:92) at com.cloudera.cmf.persist.DatabaseExecutor.execReadWriteTask(DatabaseExecutor.java:66) at com.cloudera.cmf.persist.DatabaseExecutor.execReadWriteTaskNE(DatabaseExecutor.java:75) ... 4 more Caused by: javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute statement at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310) at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:80) ... 10 more Caused by: org.hibernate.exception.ConstraintViolationException: could not execute statement at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:129) at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:136) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:3346) at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:3546) at org.hibernate.action.internal.EntityDeleteAction.execute(EntityDeleteAction.java:100) at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:377) at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:369) at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:293) at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:339) at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:52) at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1234) at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:404) at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.beforeTransactionCommit(JdbcTransaction.java:101) at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:175) at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:75) ... 10 more Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on table "commands" violates foreign key constraint "fk_command_parent" on table "commands" Detail: Key (command_id)=(2437) is still referenced from table "commands". at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:133) ... 23 more Also: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown Re-generating Kerberos credentials in Cloudera Manager fails with (the script referenced in below error is custom script written by our team, it takes some configuration input from the Cloudera manager, creates principals for Hadoop, and saves their keytabs (passwords) in locations that are accessible to Hadoop services): /usr/share/cmf/bin/gen_credentials.sh failed with exit code 1 and output of << kinit: Password incorrect while getting initial credentials ipa: ERROR: Kerberos error: ('Unspecified GSS failure. Minor code may provide more information', 851968)/('Decrypt integrity check failed', -1765328353) SASL Bind failed Local error (-2) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Decrypt integrity check failed)! chmod: cannot access `/var/run/cloudera-scm-server/cmf266477689285916699.keytab': No such file or directory >>

nataliaking_csc · ‎03-07-2015

Got everything to work! Thanks all for useful tips. Uninstalled and did another clean install - this time ensured that Sentry server is installed on same node where Hue, Hive and Impala services are also installed. My FreeIPA user does not have sufficient privileges to create roles but I will fix that 🙂

nataliaking_csc · ‎03-07-2015

Got Sentry service successfully added for Hue, Hive and Impala services. I am however now seeing a connection error when I try to load Sentry Tables: timed out (code THRIFTSOCKET): None The Hue error.log: kerberos_ ERROR handle_other(): Mutual authentication unavailable on 200 response Sentry's log on node it is installed on: ERROR sentry.org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message. java.lang.RuntimeException: sentry.org.apache.thrift.transport.TTransportException at sentry.org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at sentry.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: sentry.org.apache.thrift.transport.TTransportException at sentry.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at sentry.org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at sentry.org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182) at sentry.org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at sentry.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) at sentry.org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:1) at sentry.org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ... 4 more In Cloudera Manager Hue safety valve I have: [libsentry] # Hostname or IP of server. hostname=cdh-foyer.platform.infochimps # Port the sentry service is running on. port=8038 # Sentry configuration directory, where sentry-site.xml is located. sentry_conf_dir=/etc/sentry/conf sentry-site.xml has default settings for "sentry.service.security.mode". It seems I should need to specify Kerberos here instead of "none", not sure if that is a requirement here is this config. <?xml version="1.0" encoding="UTF-8"?> <configuration> <property> <name>sentry.service.security.mode</name> <value>none</value> </property> <property> <name>sentry.service.admin.group</name> <value>admin1</value> </property> <property> <name>sentry.service.allow.connect</name> <value>impala,hive,solr</value> </property> <property> <name>sentry.store.jdbc.url</name> <value>jdbc:derby:;databaseName=sentry_store_db;create=true</value> </property> <property> <name>sentry.store.jdbc.driver</name> <value>org.apache.derby.jdbc.EmbeddedDriver</value> </property> </configuration>

nataliaking_csc · ‎03-06-2015

Followed all steps outlined here http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/sg_sentry_service_config.html#concept_z5b_42s_p4_unique_1 (confuguring Sentry with Cloudera Manager) and Sentry is still not running.

nataliaking_csc · ‎03-06-2015

I am going over all steps outlined here http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/sg_sentry_service_config.html 1. Permissions $ sudo -u hdfs hdfs dfs -chmod -R 771 /user/hive/warehouse $ sudo -u hdfs hdfs dfs -chown -R hive:hive /user/hive/warehous 2. Disabled impersonation for HiveServer2 in the Cloudera Manager Admin Console 3. Set the Minimum User ID for Job Submission property to zero 4. Ensured the Allowed System Users property includes the hive user 5. U nchecked the Enable Sentry Authorization using Policy Files configuration property for both Hive and Impala under the Service-Wide > Policy File Based Sentry In the Service-Wide category for Hue/Hive/Impala, I will need to set the Sentry Service property to Sentry, but the option is not listed still. Going over all pre-req's again.

nataliaking_csc · ‎03-06-2015

Added hostname to safety valve and restarted Hue service. Sentry service is definitely not running: # ps auxfww | grep sentry and # netstat -anp | grep 8038 ...return nothing, which explains my error in Hue. However, I was able to test HiveServer2 with beeline shell using above recommendation for the string. # beeline Beeline version 0.13.1-cdh5.3.0 by Apache Hive beeline> !connect jdbc:hive2://MY_FQDN_HOSTNAME:10000/default;principal=hive/MY_FQDN_HOSTNAME@MY_DOMAIN scan complete in 2ms Connecting to jdbc:hive2://MY_FQDN_HOSTNAME:10000/default;principal=hive/MY_FQDN_HOSTNAME@MY_DOMAIN Enter username for jdbc:hive2://MY_FQDN_HOSTNAME:10000/default;principal=hive/MY_FQDN_HOSTNAME@MY_DOMAIN: Enter password for jdbc:hive2://cjdbc:hive2://MY_FQDN_HOSTNAME:10000/default;principal=hive/MY_FQDN_HOSTNAME@MY_DOMAIN: Connected to: Apache Hive (version 0.13.1-cdh5.3.0) Driver: Hive JDBC (version 0.13.1-cdh5.3.0) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://MY_FQDN_HOSTNAME:10> show databases; +----------------+--+ | database_name | +----------------+--+ | default | | test | +----------------+--+ 2 rows selected (0.178 seconds)

nataliaking_csc · ‎03-05-2015

# beeline Beeline version 0.13.1-cdh5.3.0 by Apache Hive beeline> !connect jdbc:hive2://localhost:10000 org.apache.hive.jdbc.HiveDriver scan complete in 2ms Connecting to jdbc:hive2://localhost:10000 Enter password for jdbc:hive2://localhost:10000: Error: Could not open connection to jdbc:hive2://localhost:10000: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0) 0: jdbc:hive2://localhost:10000 (closed)> With the cluster being Kerberized, looks to me like some config still needs to be added here to enable kerberos vs plain as shown above in an error message.

nataliaking_csc · ‎03-05-2015

I am going to troubleshoot HiveServer2 to make sure everything is running as expected there, as the error could point at some misconfigurations. I do think though that Cloudera Manager would have picked up if an issue with HiveServer2 was detected...

nataliaking_csc · ‎03-05-2015

Okay, that worked, I was able to add values I needed. Prior I was mistakenly trying to add config to Hue Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml instead of Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini I am still, however, getting transport error (my theory was that enabling Sentry should have solved that): Could not connect to localhost:8038 (code THRIFTTRANSPORT): TTransportException('Could not connect to localhost:8038',)

nataliaking_csc · ‎03-05-2015

I am looking to add Sentry service to Hue. How to I add below config using Cloudera Manager Safety Valve to enable Sentry? [libsentry] # Hostname or IP of server. hostname=localhost # Port the sentry service is running on. port=8038 # Sentry configuration directory, where sentry-site.xml is located. sentry_conf_dir=/etc/sentry/conf Add'l details: Kerberos (I know it is a pre-req) - YES Cloudera version - CDH 5.3

nataliaking_csc · ‎03-01-2015

I have discovered that Hadoop JobHistory service has not been running, that explains why no logs have been moved to HDFS. Will have to fix that, and I should have functionality I am looking for.

nataliaking_csc · ‎02-27-2015

I am looking for the best possible method to gather filesystem counters, job counters and Mapreduce framework details of all the jobs that ran on a specific date. Since upon completion of a job, the logs for the job are stored in HDFS and the information about the job is shipped off to a dedicated server called the JobHistory Server, I am looking at the node that's running Jobhistory server and port 19888 is currently locked down. I am looking for a way to either: 1) query HDFS to get data I need, or 2) open the port and use Jobhistory web UI on port 19888 3) other methods CDH v5.1.x We are currently not using Cloudera Manager.

nataliaking_csc · ‎02-24-2015

Thank you! This worked perfectly.

nataliaking_csc · ‎02-23-2015

I need to add "ttl" value to Hue configuration. We are using Cloudera Manager (which would mean that Hue config would be set in /var/run/cloudera-scm-agent/process/XXX-hue-HUE_SERVER/hue.ini), thus manually updating hue.ini in /etc/hue/conf will not activate the change. However, value "ttl" in "session" is not available in Cloudera Manager UI. The following packages are being used in a cluster: cloudera-manager-server-5.3.0-1.cm530.p0.166.el6.x86_64 cloudera-manager-server-db-2-5.3.0-1.cm530.p0.166.el6.x86_64 cloudera-manager-agent-5.3.0-1.cm530.p0.166.el6.x86_64 cloudera-manager-daemons-5.3.0-1.cm530.p0.166.el6.x86_64 Hue v3.7 How can "ttl" parameter be implemented in this scenario? [desktop] [[session]] ttl=900

Online	Offline
Last Visited	‎12-16-2015 04:05 PM

Date Registered	‎09-24-2014 01:28 PM
Last Visited	‎12-16-2015 04:05 PM
Total Messages Posted	29
Total Kudos Received	2

Re: Can't connect to Hive Metastore in cluster wit...

Re: Configuring Sentry via Hue Service Advanced Co...

Re: Query JobHistory Server in YARN to obtain data...

Cloudera Navigator External Authentication with Fr...

Re: Need to add value [[session]] ttl=900 to Hue c...

Re: Need to add value [[session]] ttl=900 to Hue c...

Re: Need to add value [[session]] ttl=900 to Hue c...

Re: Can't connect to Hive Metastore in cluster wit...

Re: Can't connect to Hive Metastore in cluster wit...

Can't connect to Hive Metastore in cluster with Ke...

Re: SSL handshake error when configuring SSL for C...

Re: SSL handshake error when configuring SSL for C...

Re: SSL handshake error when configuring SSL for C...

Re: SSL handshake error when configuring SSL for C...

Re: SSL handshake error when configuring SSL for C...

Re: SSL handshake error when configuring SSL for C...

SSL handshake error when configuring SSL for Cloud...

Re: How to make changes in .xml files using cloude...

Getting Kerberos to work after reinstalling FreeIP...

Re: Configuring Sentry via Hue Service Advanced Co...

Re: Configuring Sentry via Hue Service Advanced Co...

Re: Configuring Sentry via Hue Service Advanced Co...

Re: Configuring Sentry via Hue Service Advanced Co...

Re: Configuring Sentry via Hue Service Advanced Co...

Re: Configuring Sentry via Hue Service Advanced Co...

Re: Configuring Sentry via Hue Service Advanced Co...

Re: Configuring Sentry via Hue Service Advanced Co...

Configuring Sentry via Hue Service Advanced Config...

Re: Query JobHistory Server in YARN to obtain data...

Query JobHistory Server in YARN to obtain data for...

Re: Need to add value [[session]] ttl=900 to Hue c...

Need to add value [[session]] ttl=900 to Hue confi...