@AK- Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks. ... View more

@GowthamSenthil Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  Thanks. ... View more

thanks for the information, we're running on yarn so I guess the steps mentioned in https://spark.apache.org/docs/3.0.0-preview2/running-on-yarn.html#configuring-the-external-shuffle-service must be executed to configure the external shuffle service on each worker node. Are there any plans to support any other external shuffle service like uber rss or apache uniffle in the future? ... View more

 @DataMike    Yes, you can use the CC APIs as per your requirements [1] [2]   [1] https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cctrl-managing/topics/cctrl-using-rest-api.html [2] https://community.cloudera.com/t5/Customer/Frequently-Used-CRUISE-CONTROL-API-and-important-DOCs/ta-p/324729   From the above articles you can use the following API to rebalance topics/partitions:    curl -k --negotiate -u: -X POST "https://<CC FQDN>:8899/kafkacruisecontrol/rebalance?dryrun=false&rebalance_disk=true"   To avoid high CPU, memory and disk read write you can initiate rebalance process in batches. It will automatically create a batch and rebalance topics.   curl -X POST  "http://$HOSTNAME:8899/kafkacruisecontrol/rebalance?dryrun=true&concurrent_partition_movements_per_broker=10&concurrent_leader_movements=500"   If it will help you then please click on ”Accept as Solution" below this post.   Thank you. ... View more

Thanks. It is very helpful. ... View more

@clouderaskme  The latest CDP 7.1.7 comes with the default Spark 2.4 version.  https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/runtime-release-notes/topics/rt-pvc-runtime-component-versions.html Spark 2.4 supports Python 2.7 and 3.4-3.7. https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/release-guide/topics/cdpdc-os-requirements.html   ... View more

@Bharati Thank you! This worked. However, could you please share which logs had shown that it was trying to copy the system database and information_schema?    ... View more

That worked. Many thanks. ... View more

1. CDH5 works with Spark2 by installing a separate parcel. 2. The document suggests that CDH5 + Spark2 does not support Spark-On-HBase. Spark-On-HBase work with CDH 5 and Spark 1.6 3. With CDH 6.x default Spark version is Spark 2.x+ so as a new feature Spark-On-HBase https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_600_new_features.html#spark_new_features Spark On HBase now runs on top of Apache Spark 2.x. 4. Refer document below to use Spark-on-HBase on CDH6.x: https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_hbase_import.html#concept_asc_ctz_wp     ... View more

Yes, the issue was with java, re-configuring it solved the issue ... View more