I had a similar issue starting up pyspark shell, spark 1.6 and turned out that my program is writing log info to /user/spark/applhistorylogs and it does not have sufficient permissions to write to this path on hdfs. Changing permissions to 777 helped. Any idea why this issue popped up all of a sudden. I have been using same environment for last 2 months
... View more
I installed Cloudera 5.5.0 quicksrtart vm and trying to install kafka 1.2.0. according to the link: http://www.cloudera.com/documentation/kafka//1-2-x/topics/kafka_install.html Once I install, distribute CSD, I get the message that KAFKA 2.01-126.96.36.199.p0.5 is not compatible with CDH Version, which should be > 5.2 at the time of activation. I am using CDH 5.5 though on my VM. Below is the output of CDH_version.propertiesa for your reference. [cloudera@quickstart csd]$ cat /usr/lib/hadoop/cloudera/cdh_version.properties # Autogenerated build properties version=2.6.0-cdh5.7.0 git.hash=c00978c67b0d3fe9f3b896b5030741bd40bf541a cloudera.hash=c00978c67b0d3fe9f3b896b5030741bd40bf541a cloudera.cdh.hash=e7465a27c5da4ceee397421b89e924e67bc3cbe1 cloudera.cdh-packaging.hash=8f9a1632ebfb9da946f7d8a3a8cf86efcdccec76 cloudera.base-branch=cdh5-base-2.6.0 cloudera.build-branch=cdh5-2.6.0_5.7.0 cloudera.pkg.version=2.6.0+cdh5.7.0+1280 cloudera.pkg.release=1.cdh5.7.0.p0.92 cloudera.cdh.release=cdh5.7.0 cloudera.build.time=2016.03.23-18:30:29GMT Is there any workaround to fix the compatibility issues. I believe the cloudera manager is getting the version from wrong place. Thanks!
... View more