Hello @DA-Ka    The data might be skewed in one of the disks because some of the heavily used topics/partitions are in that particular disk.    You may want to profile the data residing (sort the kafka logs.dir) and reassign them to the other disk using the partition reassignment tool kafka-reassign-partition.sh or manually.      du -sh /kafka-logs/*       ... View more

Hello,    Please refer the document to help you with SSL authentication enabled - https://registry-project.readthedocs.io/en/latest/security.html?highlight=ssl   Thanks & Regards, Nandini ... View more

Hello @PabloMO ,   As the Spoiler Error pointed by you,the versions are not matching. You can check it by running "which python"   You can override the below two configs in /opt/cloudera/parcels/CDH-<version>/lib/spark/conf/spark-env.sh and restart pyspark. export PYSPARK_PYTHON=<same version of python> export PYSPARK_DRIVER_PYTHON=<same version of python>    Hope it helps.   Thanks & Regards, Nandini ... View more

Hello @d33play  I checked the documents again and found there is an internal jira CDH-72683 which was tracked for CDH versions to support Kafka Storage Handler.  Kafka Storage Handler is a part of Hive 3.x version but CDH 6.1.1 comes with Apache Hive 2.1.1 Hence it is not supported in CDH versions but is available with CDP. https://docs.cloudera.com/runtime/7.2.0/release-notes/topics/rt-runtime-component-versions.html https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/runtime-release-notes/topics/rt-pvc-runtime-component-versions.html   So, the KafkaStorageHandler package will not work with Hive 2.1.1.   Hope this information helps.   Thanks and Regards, Nandini   ... View more

Summarising all of the above, assuming you need the spark - atlas hook in your system, the solution is as follows.   You need to add the file atlas-application.properties and spark-sql-kafka-0-10_2.11-<version>.jar to the Oozie shared spark  library. The library is located on HDFS at <home>/oozie/share/lib/lib_<date>/spark. The application properties file can be found in various places such as /etc/hive/conf.cloudera.hive/. The jar file can be found in the parcels directory in /opt/cloudera/parcels/CDH/jars/. When you copy the files they need to be owned by oozie.oozie and be world readable. If you make any changes to the shared library you must then restart the Oozie server before jobs can find the new files. ... View more