add this in spark service in CM > Spark > Configuration >Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh =export HADOOP_CONF_DIR=/etc/hadoop/conf ... View more

The good news is that is that we shipped date support in Impala in CDP. https://docs.cloudera.com/runtime/7.0.3/impala-sql-reference/topics/impala-date.html ... View more

Good article and is still relevant for me. I would also add the following for Kudu tables create table hive.TABLE_PARAMS_BKUP as select * from hive.TABLE_PARAMS ; UPDATE hive.TABLE_PARAMS SET PARAM_VALUE = 'masterhostname01.fqdn,masterhostname02.fqdn,masterhostname03.fqdn' WHERE PARAM_KEY = 'kudu.master_addresses' and PARAM_VALUE like '%oldmasterhostname%';  And also for Sentry URI's that contain the HDFS namespace create table sentry.SENTRY_DB_PRIVILEGE_BKUP as select * from sentry.SENTRY_DB_PRIVILEGE ; UPDATE sentry.SENTRY_DB_PRIVILEGE SET URI = REPLACE(URI, 'hdfs://oldcluster-ns', 'hdfs://newcluster-ns/') WHERE URI like '%oldcluster-ns%';   ... View more

Hi,   We are facing same kind of issue..are you able to resolve? Please find below logs for reference.   cdsw status Sending detailed logs to [/tmp/cdsw_status_HOe8Jj.log] ... CDSW Version: [1.5.0.849870:4b1d6ac] OK: Application running as root check OK: NFS service check OK: System process check for CSD install OK: Sysctl params check OK: Kernel memory slabs check ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | NAME | STATUS | CREATED-AT | VERSION | EXTERNAL-IP | OS-IMAGE | KERNEL-VERSION | GPU | STATEFUL | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | dvwuaspnhad03.ams.com | True | 2019-07-23 15:22:18+00:00 | v1.8.12-1+44f60fa9b27304-dirty | None | Red Hat Enterprise Linux | 3.10.0-693.2.2.el7.x86_64 | 0 | True | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1/1 nodes are ready. ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | NAME | READY | STATUS | RESTARTS | CREATED-AT | POD-IP | HOST-IP | ROLE | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | etcd-dvwuaspnhad03.ams.com | 1/1 | Running | 0 | 2019-07-23 15:23:22+00:00 | 159.127.45.148 | 159.127.45.148 | None | | kube-apiserver-dvwuaspnhad03.ams.com | 1/1 | Running | 0 | 2019-07-23 15:23:39+00:00 | 159.127.45.148 | 159.127.45.148 | None | | kube-controller-manager-dvwuaspnhad03.ams.com | 1/1 | Running | 0 | 2019-07-23 15:23:37+00:00 | 159.127.45.148 | 159.127.45.148 | None | | kube-dns-78dcf4b9d9-4qlmt | 3/3 | Running | 0 | 2019-07-23 15:23:49+00:00 | 100.66.0.4 | 159.127.45.148 | None | | kube-proxy-72npf | 1/1 | Running | 0 | 2019-07-23 15:23:52+00:00 | 159.127.45.148 | 159.127.45.148 | None | | kube-scheduler-dvwuaspnhad03.ams.com | 1/1 | Running | 0 | 2019-07-23 15:23:30+00:00 | 159.127.45.148 | 159.127.45.148 | None | | tiller-deploy-775556c68-ntgxs | 1/1 | Running | 0 | 2019-07-23 15:22:36+00:00 | 100.66.0.2 | 159.127.45.148 | None | | weave-net-6w4cc | 2/2 | Running | 1 | 2019-07-23 15:22:36+00:00 | 159.127.45.148 | 159.127.45.148 | None | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- All required pods are ready in cluster kube-system. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | NAME | READY | STATUS | RESTARTS | CREATED-AT | POD-IP | HOST-IP | ROLE | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | cron-5df865cd67-8v9gq | 1/1 | Running | 0 | 2019-07-23 15:24:07+00:00 | 100.66.0.5 | 159.127.45.148 | cron | | db-586cf7d4b6-kgrgs | 1/1 | Running | 0 | 2019-07-23 15:24:07+00:00 | 100.66.0.8 | 159.127.45.148 | db | | db-migrate-4b1d6ac-757lc | 0/1 | Succeeded | 0 | 2019-07-23 15:24:07+00:00 | 100.66.0.6 | 159.127.45.148 | db-migrate | | ds-cdh-client-b948b4b8b-qvltp | 1/1 | Running | 0 | 2019-07-23 15:24:09+00:00 | 100.66.0.19 | 159.127.45.148 | ds-cdh-client | | ds-operator-84d49b8786-mvssl | 2/2 | Running | 2 | 2019-07-23 15:24:09+00:00 | 100.66.0.13 | 159.127.45.148 | ds-operator | | ds-vfs-7c85df495f-2xbcj | 1/1 | Running | 0 | 2019-07-23 15:24:09+00:00 | 100.66.0.21 | 159.127.45.148 | ds-vfs | | ingress-controller-ff89786db-cbmpj | 0/1 | CrashLoopBackOff | 243 | 2019-07-23 15:24:07+00:00 | 159.127.45.148 | 159.127.45.148 | ingress-controller | | livelog-66f5b7986c-ctzsp | 1/1 | Running | 0 | 2019-07-23 15:24:07+00:00 | 100.66.0.7 | 159.127.45.148 | livelog | | s2i-builder-5b7c868b6d-4lslx | 1/1 | Running | 2 | 2019-07-23 15:24:09+00:00 | 100.66.0.22 | 159.127.45.148 | s2i-builder | | s2i-builder-5b7c868b6d-m8r28 | 1/1 | Running | 2 | 2019-07-23 15:24:10+00:00 | 100.66.0.18 | 159.127.45.148 | s2i-builder | | s2i-builder-5b7c868b6d-t56q2 | 1/1 | Running | 2 | 2019-07-23 15:24:09+00:00 | 100.66.0.23 | 159.127.45.148 | s2i-builder | | s2i-client-77d575bcc8-s98nf | 1/1 | Running | 0 | 2019-07-23 15:24:09+00:00 | 100.66.0.20 | 159.127.45.148 | s2i-client | | s2i-git-server-7855bcbcc5-prmgc | 1/1 | Running | 0 | 2019-07-23 15:24:07+00:00 | 100.66.0.9 | 159.127.45.148 | s2i-git-server | | s2i-queue-76fc7f5f88-jwrwf | 1/1 | Running | 0 | 2019-07-23 15:24:07+00:00 | 100.66.0.3 | 159.127.45.148 | s2i-queue | | s2i-registry-74496d54dc-jkjp4 | 1/1 | Running | 0 | 2019-07-23 15:24:07+00:00 | 100.66.0.15 | 159.127.45.148 | s2i-registry | | s2i-registry-auth-6f6f658947-8dgp9 | 1/1 | Running | 0 | 2019-07-23 15:24:07+00:00 | 100.66.0.11 | 159.127.45.148 | s2i-registry-auth | | s2i-server-5b778bcb8d-n92rk | 1/1 | Running | 2 | 2019-07-23 15:24:08+00:00 | 100.66.0.12 | 159.127.45.148 | s2i-server | | secret-generator-77d7b98444-wwjgt | 1/1 | Running | 0 | 2019-07-23 15:24:08+00:00 | 100.66.0.10 | 159.127.45.148 | secret-generator | | spark-port-forwarder-q6r9t | 1/1 | Running | 0 | 2019-07-23 15:24:09+00:00 | 159.127.45.148 | 159.127.45.148 | spark-port-forwarder | | web-75bbb7d4ff-6ngdl | 1/1 | Running | 0 | 2019-07-23 15:24:08+00:00 | 100.66.0.17 | 159.127.45.148 | web | | web-75bbb7d4ff-g7hf9 | 1/1 | Running | 0 | 2019-07-23 15:24:08+00:00 | 100.66.0.14 | 159.127.45.148 | web | | web-75bbb7d4ff-jtf8b | 1/1 | Running | 0 | 2019-07-23 15:24:08+00:00 | 100.66.0.16 | 159.127.45.148 | web | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Pods not ready in cluster default ['role/ingress-controller']. All required Application services are configured. All required secrets are available. Persistent volumes are ready. Persistent volume claims are ready. Ingresses are ready. Checking web at url: http://cdsw.ams.com Web is not yet up. Cloudera Data Science Workbench is not ready yet ... View more

Did anyone succussfully solved this problem. I am installing a new cloudera cluster using 5.15.1 version and we want to restrict the firewall rules in a range for the nodes to communicate. However, when i run jobs it starts using ports that are not open and hence, fails to run the job.     ... View more

Joey,   That sounds great.  I encourage you to consider going to Cloudera Manager 5.8.x so that you can leverage the new Hive Incremental Replication feature.  Check it out here:   http://www.cloudera.com/documentation/enterprise/latest/topics/cm_bdr_hive_replication.html#concept_tzh_dwq_fw   Cheers,   Ben ... View more