According to this https://issues.apache.org/jira/browse/HIVE-14217 Druid integration is fixed in Hive 2.2.0. CDH5 is based on Hive 1.x as mentioned before, and I cannot find the specific patch in the release notes. Probably CDH6 (which is based on Hive 2.1.0) will support it. We have to wait. ... View more

Have you checked the trust store file that it is copied to the host? ... View more

Everything seem to be ok. I have the same configuration and cannot reproduce your issue. I have CM 5.14.3 installed so my jar is /usr/share/cmf/lib/agent-5.14.3.jar What version are you using? ... View more

hi Krishna, rolling upgrade of Java is not supported according to https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cdh_cm_upgrading_to_jdk8.html Cloudera does not support a rolling upgrade to JDK 1.8. You must shut down the entire cluster. You should either upgrade all nodes to Java 8, or downgrade to Java 7.   ... View more

Usually, when we create test clusters, we are more elastic on user permissions. With this configuration, we are increasing the possibility, that a malicious user can take advance and gain access to data on production cluster, that he/she normally should not have.   Again, this is only my personal opinion. Of course you can ignore it, as you know your needs. ... View more

IMHO, this is not very good idea. You probably have some reasons to do that. In that case, since you share almost everything, you should consider manage both clusters from the same Cloudera Manager. In that case all of your points are fully satisfied. ... View more

You can check the last of the points you have listed as it is almost sure it will fail to hearbeat, if you have not manually did that step (set use_tls=1 and restart agent) If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here. If you did this already, then make sure that the configured keystore and truststore files, have been copied to the new host. ... View more

It is strange. Your config, seems to be ok. I don't know what is the problem. I could recommend check the output of hostname -f and host -t A 10.142.0.4 if the output is the FQDN, then probably something is wrong with your version of Dnstest.   Can you also check the "/etc/nsswitch.conf" file? Usually the hosts line is: hosts: files dns Can you check if something else is before the above? E.g. "sss files dns". The order matters. If "files" is first, then the local "/etc/hosts" will be checked first http://man7.org/linux/man-pages/man5/nsswitch.conf.5.html ... View more

Can you post you /etc/hosts file? ... View more

@RajeshBodollaunfortunatelly you are correct and I have realized the hard way. I have upgraded the CDH during the previous week and this week I was trying to configure some wildcard topics only to find out that this is not possible.   When I wrote the previous post, it was clearly mentioned in the release notes, that is supported. I had copied this part which was saying: * Wildcard usage for Kafka-Sentry components You can specify an asterisk (*) in a Kafa-Sentry command for the TOPIC component of a privilege to refer to any topic in the privilege. Supported with CDH 5.14.1. You can also use an asterisk (*) in a Kafka-Sentry command for the CONSUMERGROUPS component of a privilege to refer to any consumer groups in the privilege. This is useful when used with Spark Streaming, where a generated group.id may be needed. Supported with CDH 5.14.1. Now, this part is gone from the documentation.   I apologize that I have not tested it before.   But as you can see in http://archive.cloudera.com/cdh5/cdh/5/sentry-1.5.1-cdh5.14.2.CHANGES.txt it is still mentioned as commited: commit e9efe1b3b38912af8799d37a67679295d98ebe63 Author: amishra <amishra@cloudera.com> Date: Thu Feb 8 15:16:15 2018 +0530 CDH-57131 CDH-61471: Add consumergroup and topic wildcard for Kafka privilege validation Change-Id: I19cc4b8b047eac668721e85131287f56b6f66fcd Reviewed-on: http://gerrit.sjc.cloudera.com:8080/30142 Tested-by: Jenkins User Reviewed-by: Viktor Somogyi <viktor.somogyi@cloudera.com> Reviewed-by: Sergio Pena <sergio.pena@cloudera.com>   ... View more

To be honest, I have not used lzo in spark. I suppose that you have Spark running under yarn and not stand-alone. In that case, the first thing I would check, is that lzo is configured in YARN available codecs "io.compression.codecs". Moreover, have you configured HDFS https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_gpl_extras.html#xd_583c10bfdbd326ba--6eed2fb8-14349d04bee--7c3e ... View more

The error is about CDH and not Cloudera Manager. The specific version of Cloudera Kafka needs both to be at least, version 5.13. ... View more

This is not expected at all. Tables should be created under db directory. E.g. /user/hive/warehouse/database1.db/table1 /user/hive/warehouse/database2.db/tableA etc.   This case is very weird. In such cases I suspect user error. Can you provide the command you have used? ... View more

The link you have provided refers to possible data loss in case of disabling security but it is in "authorization" part of security. Enabling/disabling Kerberos is in the scope of "authentication". Personally I have done it to enable/disable with no data loss. There is no reason to lose data. But if you have defined authorization rules, then it is diferent. ... View more

According to your logs   18/04/18 15:09:05 WARN security.UserGroupInformation: PriviledgedActionException as:m0162109 (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 18/04/18 15:09:05 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 18/04/18 15:09:05 WARN security.UserGroupInformation: PriviledgedActionException as:m0162109 (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 18/04/18 15:09:05 WARN security.UserGroupInformation     This is a Kerberos issue. If you are using HUE notebook, Kerberos authentication has been done from hue user which is also allowed to impersonate your account "m0162109". If you are using other Notebook, then probably you do the authentication when you start the notebook. ... View more

According to your logs  18/04/18 15:09:05 WARN security.UserGroupInformation: PriviledgedActionException as:m0162109 (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 18/04/18 15:09:05 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 18/04/18 15:09:05 WARN security.UserGroupInformation: PriviledgedActionException as:m0162109 (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 18/04/18 15:09:05 WARN security.UserGroupInformation ... View more

I sedond to that this is an issue with Kerberos. If kinit didn't help then try to use: --principal PRINCIPAL Principal to be used to login to KDC, while running on secure HDFS. --keytab KEYTAB The full path to the file that contains the keytab for the principal specified above. This keytab will be copied to the node running the Application Master via the Secure Distributed Cache, for renewing the login tickets and the delegation tokens periodically. Moreover, some more infor regarding how do you use pyspark etc. would be helpful. ... View more

There are various possible issues for that. Have you checked connectivity from agent node to cloudera manager? The agent connects to CM on port 7182. Have you tested that you get a reply and there is no network or firewall issue? ... View more

According to the output of "systemctl status cloudera-scm-server" the service is down. Have you checked both "cloudera-scm-server.out" and "cloudera-scm-server.log"? Can you provide those files? ... View more

If you delete the log files that end with a number (e.g. mgmt-cmf-mgmt-NAVIGATORMETASERVER-my.node.com.log.out.1), then you will have no issue. Don't delete the ones with ".log" suffix, because they are open for writing by services.   Suppose that you have Max Logs Size = 200MB Maximum Log File Backups = 10   That means that each time your .log file reaches 200MB will be rolled, for ten times in total (will be copied to .log.1, .log.2,.....,.log.10). So in total, the log files of a service can occupy up to ((10* 200MB) + 200MB)=2200MB.     Depending on your disk size you can reduce only one or both of them, depending on what is better for you. E.g. if you don't want so big files you can set "Max Log Size"=100MB, that means ((10*100MB) + 100MB) = 1100 MB. So just reducing this parameter into half, you save 1100MB of disk space.   The same applies to all services. ... View more